[ orcid=0000-0002-0416-5676]

Literature Review, Methodology, Software, Data curation, Implementation,
Experimental analysis, Writing - Original draft preparation

1]organization=Department of Computer Science and Engineering, BGC Trust
University Bangladesh, city=Chittagong, postcode=4381, country=Bangladesh

[style=chinese]

URL]https://orcid.org/0000-0003-1740-5517

Conceptualization, Writing - Review & editing and Supervision

2]organization=Centre for Securing Digital Futures, School of Science, Edith
Cowan University, city=Perth, postcode=WA-6027, country=Australia

[cor1]Corresponding author

# An Explainable Transformer-based Model for Phishing Email Detection: A Large
Language Model Approach

Mohammad Amaz Uddin<EMAIL_ADDRESS>[  Iqbal H. Sarker
<EMAIL_ADDRESS>[

###### Abstract

Phishing email is a serious cyber threat that tries to deceive users by
sending false emails with the intention of stealing confidential information
or causing financial harm. Attackers, often posing as trustworthy entities,
exploit technological advancements and sophistication to make detection and
prevention of phishing more challenging. Despite extensive academic research,
phishing detection remains an ongoing and formidable challenge in the
cybersecurity landscape. Large Language Models (LLMs) and Masked Language
Models (MLMs) possess immense potential to offer innovative solutions to
address long-standing challenges. In this research paper, we present an
optimized, fine-tuned transformer-based DistilBERT model designed for the
detection of phishing emails. In the detection process, we work with a
phishing email dataset and utilize the preprocessing techniques to clean and
solve the imbalance class issues. Through our experiments, we found that our
model effectively achieves high accuracy, demonstrating its capability to
perform well. Finally, we demonstrate our fine-tuned model using Explainable-
AI (XAI) techniques such as Local Interpretable Model-Agnostic Explanations
(LIME) and Transformer Interpret to explain how our model makes predictions in
the context of text classification for phishing emails.

###### keywords:

Large Language Model (LLM)Masked Language Model (MLM)PhishingExplainable-AI
(XAI)DistilBERT Transformer

##  1 Introduction

The rapid advancement of internet technology has significantly changed the way
people communicate online while also raising more security issues (Salloum et
al., 2021). One of the prevalent security concerns, the cyberattack termed
“email phishing” poses grave risks such as identity or login credentials
theft, financial repercussions, and harm to an organization’s reputation. In
this current decade, email phishing has changed throughout time to become a
more effective method for carrying out fraudulent actions. As phishers
continuously innovate new techniques, detecting phishing has become
increasingly challenging, posing a formidable task for law enforcement
agencies, governments, and organizations combating this cyber threat. To evade
falling victim to this threat, organizations need to adeptly detect and
prevent these attacks at scale by adopting effective phishing detection
techniques. Leveraging Artificial Intelligence (AI) techniques like Machine
Learning, Deep Learning, and Hybrid Learning which are already utilized for
phishing attack detection, can significantly bolster these efforts (Basit et
al., 2021) (Sarker, 2023). Moreover, because of the rapid advancements in
technology, new methods based on AI are being developed daily in the
cybersecurity field.

Transformer-based models become more popular in the context of text
classification. The transformer is an innovative neural design that uses the
attention mechanism to encode incoming data into powerful characteristics (Han
et al., 2021). Transformer-based models have greatly improved the processing,
comprehension, and interpretation of text data inputs, which has had a
revolutionary effect on the development of phishing classification techniques
(Jamal et al., 2023). Large Language Models like OpenAI’s ChatGPT (Generative
Pre-trained Transformer) and Bard have transformed the landscape of natural
language creation and comprehension, capturing considerable attention from the
general public (Zhao et al., 2023; Yao et al., 2023). The increasing
complexity of LLMs is radically changing artificial intelligence and has the
potential to transform algorithm development and application by understanding
human language through extensive text data patterns and correlations.

Many transformer-based models used in natural language processing (NLP),
including well-known ones like OpenAI’s GPT and Google’s BERT (Bidirectional
Encoder Representations from Transformers), are categorized as Large Language
Models because of their ability to comprehend and produce text that is similar
to that of a human. BERT model, a type of transformer-based neural network
architecture pre-trained on extensive text corpora to grasp contextual word
embeddings (Koroteev, 2021), become increasingly efficient and valuable in
text classification and generation, including tasks like spam and phishing
detection, sentiment analysis (Singh et al., 2021), and more other fields.
BERT is designed to take into account both left and right contexts at every
level to pre-train complex bidirectional representations from unlabeled text
(Devlin et al., 2018). With simply one extra output layer, this method allows
the pre-trained BERT model to be refined, leading to the development of state-
of-the-art models for a variety of applications. In order to model language,
BERT uses a Masked Language Model, in which certain words in a sentence are
randomly replaced with a unique token (usually [MASK]). Next, by examining the
contextual clues offered by nearby words, the model is trained to infer the
original words. BERT makes use of a Transformer architecture technique called
“self-attention”, which allows it to recognize inter-word relationships in
text sequences and assess the relative importance of individual words within a
sentence in order to understand the context. Based on the transformer layers
number and parameters BERT model has different variants such as: BERT-base
(Khadhraoui et al., 2022), ALBERT (Lan et al., 2019), RoBERTa (Liu et al.,
2019), DistilBERT (Sanh et al., 2019), etc.

Despite BERT’s versatility across NLP tasks, comprehending its internal
mechanisms often poses challenges, resembling a “black box” due to its
complexity. Unraveling how the model operates and forms predictions demands
considerable effort (Xu et al., 2019). XAI techniques emerge as a vital
solution, offering significant perceptions of the model’s reasoning and
illuminating the decision-making process (Hoffman et al., 2018; Anan et al.,
2023). Recently, Sarker (Sarker, 2024) has also highlighted the significance
of explainability analysis in various perspectives in the context of
cybersecurity, in which we are interested to explore more with experiments.
XAI techniques can illuminate the words or features deemed most influential by
the BERT model in specific text classification tasks of any nature. In this
research, our main goal is to show transparency in phishing email detection
with explainable AI. The major contribution of the work is as follows:

  * •

We present a transformer-based Phishing email detection model, a fine-tuned
version of DistilBERT.

  * •

Two different XAI techniques LIME and Transformer Interpret are utilized to
explore the interpretability of the proposed fine-tuned model and visualize
the scoring percentage of those techniques on prediction.

  * •

We demonstrate our model performance on both imbalanced and balanced datasets.

The remaining sections of the paper are structured as follows: Section 2
delves into related research works. Section 3 offers a comprehensive
explanation of the proposed model’s framework and methodology. In Section 4,
the experimental outcomes and results are elaborated upon. Section 5 examines
the transparency of the model using XAI. Moreover, within Section 6, there is
an extensive discussion of the results. Lastly, Section 7 presents the
concluding remarks and outlines the future prospects of this work.

##  2 Literature Review

The field of cybersecurity has grown increasingly important and complex in the
modern world. There are several subfields within cybersecurity, including data
security, application security, network security, etc. Cyber-attacks of all
kinds are becoming more and more frequent, and they can destroy or steal
imported data. In addition, in this cybersecurity field, there are an enormous
amount of new and creative techniques being developed daily for identifying
and preventing cyberattacks. For the improvement of phishing email detection
and prevention systems, many researchers have developed a wide range of novel
strategies to counteract phishing emails, utilizing state-of-the-art methods
from deep learning, machine learning, and hybrid models. Phishing is still a
serious cybersecurity concern and one of the main ways hackers breach
networks.

Detecting phishing attempts early on is critical for businesses nowadays. The
security of an organization can be significantly increased by machine
learning, which can identify potential assaults early on (Apruzzese et al.,
2023). In ref. (Yasin and Abuhasan, 2016), a text processing, data mining, and
knowledge discovery approach was combined to create an intelligent phishing
email detection model. They utilized text stemming and WordNet ontology in
pre-processing to enrich the model with synonyms and employed five machine
learning classification algorithms (J48, Naïve Bayes, Support Vector Machine
(SVM), Multi-Layer Perceptron (MLP), and Random Forest (RF)). They have
notably achieved 99.1% accuracy with RF and 98.4% using J48 for an accredited
dataset. The study (Harikrishnan et al., 2018) employed techniques such as SVD
and NMF for feature extraction and dimensionality reduction from legitimate
and phishing email text data. For the numeric representation of phishing
emails, the author integrated TFIDF with the SVD and NMF, and Decision Trees,
Logistic Regression, Random Forest, Naive Bayes, K-Nearest Neighbors (KNN),
AdaBoost, and SVM have been utilized as classifiers. Hamid et al. (2013)
presented a hybrid feature selection technique combining content-based and
behaviour-based approaches to detect phishing emails. Identifying the
behaviour-based features in phishing emails was the main objective of this
research because an attacker cannot disguise these types of features. They
examined the overall process with a phishing dataset that consists of 6923
emails where phishing emails were collected from the Nazario dataset and
legitimate emails from the SpamAssassin dataset and achieved 94% accuracy with
this hybrid feature selection technique. With an emphasis on the importance of
top-ranking features, the study (Zamir et al., 2020) examines feature
selection methods (GR, IG, Relief-F, RFE) on a dataset of 11,055 phishing
sites with 32 attributes. To improve classifier accuracy, it suggests new
attributes that are generated from the strongest and weakest features. A
variety of supervised learning methods with differing feature sets are
compared, such as RF, SVM, bagging, kNN, Neural Network (NN), and j48. The
proposed method using RF showed 97.3% and using stacking achieved 97.4%
accuracy.

In cybersecurity, deep learning has great potential, particularly for
detecting network intrusions, classifying malware, and recognizing phishing
attempts.Alhogail and Alsabih (2021) proposed a deep learning-based phishing
email detection model based on graph convolutional network (GCN) and natural
language processing (NLP). A publicly available balanced phishing dataset has
been utilized where the authors used the email body’s text to extract features
that improved the proposed model’s accuracy. The model showed 98.2% accuracy
with a low false-positive rate of 0.015%. For phishing detection, a new
Intelligent Cuckoo Search (CS) Optimization Algorithm with a Deep Learning-
based Phishing Email Detection and Classification (ICSOA-DLPEC) approach has
been proposed in (Brindha et al., 2023). Data preprocessing (email cleaning,
tokenization, and stop-word elimination) techniques were employed to derive
the important features from the Enron email dataset. The Gated Recurrent Unit
(GRU) model was used with a CS algorithm for the fine-tuning of the parameter
of the proposed model and achieved a high accuracy of 99.72%.Dewis and Viana
(2022) employed Long short-term memory (LSTM) model with NLP to detect both
phishing and spam email detection. They developed a phish responder which is a
Python-based command line solution and used deep learning and NLP for this
classification task. The proposed LSTM model achieved 99% accuracy for the
text-based dataset. Fang et al. (2019) introduced THEMIS, a phishing email
detection model built on an improved recurrent convolutional neural networks
(RCNN) model which was improved by using the Bidirectional Long Short-Term
Memory (Bi-LSTM). 99.84% accuracy is demonstrated by the suggested THEMIS
model on datasets that are not balanced where the positive rate (FPR) is
0.043%. Other deep learning algorithms, such the reinforcement learning,
Recurrent Neural Network (RNN), and Deep Belief Network (DBN), are also in use
(Zhang and Li, 2017; Bahnsen et al., 2017; Smadi et al., 2018).

Moreover, transformer-based models are also making a good impact in the
cybersecurity field.Thapa et al. (2023) concentrated on two specific models:
RNN and BERT to test how well these models could identify phishing emails.
They looked at how well these models performed when the email data was
distributed unevenly across several organizations and when added more data. In
this experiment, the BERT model’s test accuracy was 96.1%. In addition,
another study (Atawneh and Aljehani, 2023) focused on different deep learning
architectures (CNNs, LSTMs, RNNs, BERT) to identify the emails as legitimate
or phishing by analyzing different features of emails. The proposed model
using BERT and LSTM, reaching an impressive accuracy of 99.61% on publicly
available 4 different datasets, outperforming other methods. Nowadays, other
variants of the BERT model such as DistilBERT, RoBERTa, and TinyBER (Jiao et
al., 2019) are performing well in the phishing and spam detection research
sector (Jamal et al., 2023; Lee et al., 2020; Songailaitė et al., 2023). The
authors (Jamal et al., 2023) presented an Improved Phishing and Spam Detection
Model (IPSDM) which consists of two BERT variants such as the fine-tuned
DistilBERT and RoBERTA model and showed good accuracy in this research scope.
Lee et al. Lee et al. (2020) proposed Context-Aware Tiny Bert (CatBERT) which
exhibits faster processing and robustness to adversarial attacks than other
transformer models, especially those that involve intentional keyword
substitutions with synonyms or typos. The detection rate of this proposed
model was 87% which was far better than other compared models. Furthermore,
researchers also used fine-tuned BERT-based models for phishing URL detection
(Wang et al., 2023; Maneriker et al., 2021).

Overall, this literature survey shows the existing works although showing good
performance by using different datasets and models but most of the research
didn’t work model’s interpretability or explainability. “How does a model
work?” or “Which things or materials contribute to detecting phishing?” those
types of questions can arise in their working procedure. In this work, to
answer those types of questions and solve the existing drawbacks in the
literature, we have worked with explainable AI using not only one technique
but also used two different techniques with different methodologies. In the
literature, although some researchers did full experiments for phishing email
detection but they utilized spam datasets as phishing datasets. In
cybersecurity, we know that two different types of emails exist: spam and
phishing, and not all spam emails are considered to be phishing. For this
reason, we only focused on the phishing email dataset.

##  3 Methodology

In this paper, we have tried to explore the Transformer-based self-attention
mechanism BERT model with the explainable AI. Initially, we collected a
dataset and prepared it with data cleaning and balancing techniques. After
preparing the data we have focused on optimizing and fine-tuning the model.
For this phishing email detection purpose, we have selected the DistilBERT
model and pre-trained with the prepared data. The DistilBERT model
demonstrates good performance in both balanced and imbalanced scenarios of
datasets when identifying phishing emails from texts. At the end of the
experiment, we employed two XAI techniques Explanations LIME and Transformer
Interpret to explore our fine-tuned model and described how it is working and
performing with good accuracy in this text classification. The overall
methodology of the proposed method is depicted in Fig. 1.

Figure 1: The proposed overall methodology.

###  3.1 Data Collection and Processing

For this experiment, a publicly available dataset has been collected from
kaggle. The dataset describes different kinds of email text bodies that can be
used to identify phishing emails using in-depth text analysis. This dataset
consists of “Safe Email” and “Phishing Email”. Fig. 2 shows the data overview.
The dataset has 11,322 Safe Emails and 7,328 Phishing Emails which shows that
it is an imbalanced dataset. Along with this, this dataset showed that null
values exist in rows. As null values can’t be processed that’s why those rows
are removed. By using the resampling technique minor class “Phishing Email” is
oversampled. This resampling technique mainly generates copies of other
instances and adds those copies to the dataset. It is essential to address
uneven data in order to develop equitable and precise predictive models.
Balancing data ensures algorithms work well across all categories by reducing
biases toward the majority class. This prevents skewed outputs and makes it
possible to create a more accurate and equitable model for underrepresented
classes. After oversampling, the dataset has 11,322 safe emails and 11,322
phishing emails in equal proportion. This data balancing also helps to reduce
the risk of oversampling. Fig. 3 illustrates the imbalance and balanced data
before and after sampling.

Figure 2: The sample data overview

(a) Imbalanced data

(b) Balanced data

Figure 3: Imbalanced and balanced data.

###  3.2 Dataset splitting

For this experiment, we have split the dataset into 70-30% where 70% data is
training data and 30% is testing data. In 70% training set total of 15,850
data exist and the testing set consists of 6,794 data. Initially, the training
set was utilized for the train the model, and the testing test was used to
find out the effectiveness of the model on unseen data identification and
prediction is phishing or not.

###  3.3 Model Selection

In this experiment, we have employed the transformer-based BERT variant
architecture DistilBERT which is a lightweight, fast, and economical model.
This is developed for language processing tasks and also compresses the
original BERT model and makes it faster. Knowledge distillation was utilized
in the pre-training phase of this pre-trained model, which allowed it to be
effectively reduced in size by 40% while maintaining 97% of its language
understanding skills and delivering a 60% speed increase (Sanh et al., 2019).
This small variant has twelve attention heads, 768 hidden layers, and six
transformer layers (Adoma et al., 2020). Whereas DistilBERT contains about 66
million parameters, the original BERT model has 110M (million) parameters. By
lowering the number of parameters, DistilBERT maintains excellent performance
while achieving a smaller size and faster inference speed.

####  3.3.1 Model Optimization

To improve its handling of terms that are not in its lexicon and to capture
meaningful sub-word representations, BERT uses a Word-Piece tokenizer, which
splits words into smaller sub-words. The preprocessed phishing dataset is
tokenized using the tokenizer of the DistilBERT transformer. In this tokenized
portion, the maximum sequence length of input data has been used 512. Other
crucial components of tokenization are padding and truncation. The major
purpose of these procedures is to ensure that each tokenized input sequence
has a constant length. This is important because it creates consistent
embeddings and efficient processing in batches.

Pre-trained weights acquired throughout the pre-training phase are used to
initialize the DistilBERT models. Selecting an optimal batch size is paramount
in deep learning, as it significantly influences both training and testing
accuracies, along with overall runtime (Lin, 2022). The careful selection of
batch size determines the efficiency of the neural network training process,
striking a balance between computational resources and model performance. The
batch sizes for training and testing data are optimized at 32 and 64,
respectively. Each epoch’s training data is assigned at random to ensure the
model sees a diversity of unseen data samples and prevents overfitting. This
shuffling method increases the model’s ability to generalize.

During training, an optimizer is essential in adjusting the parameters of a
neural network. The main goal of this is to minimize the error or loss
function of the model for better performance. For this research work, AdamW
(Adam Weight Decay) (Loshchilov and Hutter, 2017) optimizer has been used to
get better performance and avoid overfitting. This optimizer is a modification
version of Adam which is widely adopted for training deep neural networks
(Zhuang et al., 2022). By separating weight decay from the learning rate,
AdamW enhances its implementation. It improves regularization and training
dynamics by separating weight decay from gradient updates, in contrast to
Adam. Learning rate is a crucial hyperparameter governing step size during
training for optimizing models. The convergence and overall performance of the
system are dependent on various aspects such as the model architecture,
optimization techniques, and task domain. For efficient fine-tuning and
fruitful model training, this parameter must be adjusted. Selecting a high
learning rate can introduce instability, resulting in poor performance on
unseen data, while a low learning rate may extend the training process,
requiring more epochs and increasing computational costs. In this research, a
learning rate, 2e-5 is used in the training process for the transformer-based
DistilBERT model.

The overall discussion about model optimization indicates the parameter
settings for the introduced best model for this specific experiment. The
hyperparameter settings for the suggested model are shown in Table 1. We
worked with every possible value of the parameters by using an iterative
process to find the ideal value. Following that, these improved hyperparameter
settings are used to train the suggested model.

Table 1: Parameter Settings for Phishing Email Detection Task. Parameter Name | Worked With | Optimal Value
---|---|---
Training Batch size | 8, 16, 32 | 32
Testing Batch size | 8, 16, 32, 64 | 64
Optimizer | AdamW | AdamW
Learning rate | 1e-5, 2e-5, 3e-5 | 2e-5
Number of epochs | 3, 4, 5, 6, 7 | 6

####  3.3.2 Fine Tuning

A pre-trained model is fine-tuned using task-specific data and backpropagation
to alter parameters on certain tasks and datasets. The primary objective of
this is to improve the model’s performance across various tasks and datasets
while minimizing the loss function. In this experiment,
DistilBERTForSequenceClassification has been utilized to adopt the pre-trained
model. DataLoader is used to configure the train_loader and test_loader by
setting prepared data, batch size, and shuffling command where train_loader is
configured with shuffling and test_loader is configured without shuffling.
After that, we fine-tuned this DistilBERT model by adjusting and adopting some
necessary steps which are shown in Fig. 4.

Figure 4: The overall process of fine-tuning.

In this overall fine-tuning process, following the optimizer’s setup, the
model is fine-tuned by feeding it the input parameters (input_ids,
attention_mask, and label) during the forward pass so that predictions can be
made. After the forward pass, gradients of the loss for the parameters
(weights and biases) are calculated in the backward pass. These gradients show
how changing the parameters would affect the loss. Lastly, an update is made
to the model parameters. The optimizer is mainly responsible for changing the
neural network’s parameters in accordance with the calculated gradients. It
adjusts the weights and biases in an attempt to reduce the loss. The model
gains the capacity to fine-tune its parameters throughout several epochs,
which improves its predictive capability on the training set. The ultimate
goal is to identify underlying patterns in the training set and use those
patterns to generalize well to new data. Algorithm 1 shows the overall fine-
tuning process in training.

Algorithm 1 Fine-Tuning the DistilBERT model

1:DistilBERT model, training data loader, optimizer, and number of epochs

2:Trained fine-tuned model

3:$\text{epochs}\Leftarrow n$

4:$\text{optimizer}\Leftarrow\text{Optimizer}(\text{model.parameters()},\text{lr}\Leftarrow
X)$

5:for epoch in range(epochs) do

6: model.train()

7: for batch in train_loader do

8: $\text{input\\_ids}\Leftarrow\text{batch}[0]$

9: $\text{attention\\_mask}\Leftarrow\text{batch}[1]$

10: $\text{labels}\Leftarrow\text{batch}[2]$

11: $\text{gradients}\Leftarrow\text{0}$

12:
$\text{forwardpass}(\text{input\\_ids},\text{attention\\_mask}\Leftarrow\text{attention\\_mask},\text{labels}\Leftarrow\text{labels})$

13: loss_calculation()

14: backwardpass()

15: update_parameters()

16: end for

17:end for

####  3.3.3 Model Explainability

The ability of a model to explain its decisions is crucial since it provides
insight into the process. Verifying the accuracy of judgments and reducing
errors are made feasible by understanding how models work. Explainable AI has
been a popular area with a growing group that has developed efficient
techniques to understand deep neural network predictions among the more
complex machine learning models (Holzinger et al., 2022). This is important
because sophisticated models frequently act as opaque “black boxes" making it
difficult for people to comprehend the logic behind particular decisions. In
this research work, we have worked with model explanability to solve the black
box problem. To understand the model prediction two explainability models LIME
and Transformers Interpret have been utilized.

Generally, this experiment for phishing email detection is a text
classification task that LimeTextExplainer used to explain the black box
DistilBERT model. The purpose of the LIME method is to faithfully explain any
classifier or regressor’s predictions (Ribeiro et al., 2016). For improved
interpretability, LIME provides an understandable depiction of the decision-
making process, providing light on the “why" behind the predictions or
suggestions made by these models. In order to achieve model explainability,
LIME first perturbs the email content by making modest adjustments,
rearranging words, or adding new words. After that, the black-box transformer-
based model is applied to the altered email content to generate predictions.
From these impacted instances, LIME derives features. This local model should
roughly represent the behaviour of the black-box model surrounding the chosen
instance. Next, explanations for the black-box model’s prediction for the
original data are produced using the local model. This explanation usually
highlights the words or key elements most important during the model’s
decision-making.

A Transformers Interpret tool was also utilized to explain the transformer-
based model for this classification work. The class explainer process was
carried out using this tool using a model and tokenizer. With the use of this
class explanation, word attributions are computed. Positive attribution
numbers signify a word’s positive contribution to the projected class, while
negative numbers signify a word’s negative contribution. The terms that
demonstrate a strong contribution to decision-making are also highlighted in
this explanation. The overall process of model explainability is shown in Fig.
5. Algorithm 2 shows the explanation process of the model prediction.

Figure 5: The overall process of model explainability Algorithm 2
Explainability Analysis

1:Trained model, tokenizer, and text

2:Explanation of the model prediction

3:class_names $\Leftarrow$ [X, Y, …N]

4:function Explain_Prediction(model, tokenizer, text)

5: LIME_Explanation(model, text)

6: Transformer_Interpret(model, tokenizer, text)

7:end function

8:function LIME_Explanation(model, text)

9: explainer $\Leftarrow$ LimeTextExplainer(class_names $\Leftarrow
class\\_names$)

10: explanation $\Leftarrow$ explainer.explain_instance(text, model,
num_features $\Leftarrow n$, top_labels $\Leftarrow 1$)

11: exp_class $\Leftarrow$ explanation.available_labels()

12: exp $\Leftarrow$ explanation.as_list(label$\Leftarrow$ exp_class)

13: explanation_visualize()

14:end function

15:function Transformer_Interpret(model, tokenizer, text)

16: cls_explainer $\Leftarrow$ SequenceClassificationExplainer(model,
tokenizer)

17: word_attributions $\Leftarrow$ cls_explainer(text)

18: cls_explainer_visualize()

19:end function

##  4 Results

The fine-tuned transformer-based model was trained and tested with both
unbalanced and balanced datasets. To assess the performance of the model
different measurements such as confusion matrix, precision, recall, F1-score,
and overall accuracy have been calculated. Overall summarize the performance,
a model needs to calculate the confusion matrix first. It is a kind of table
that provides a detailed breakdown of the actual and predicted classification
for specific task domain data. The confusion matrix of the DistilBERT model
for both imbalanced and balanced datasets is shown in Table 2. True Positive
(TP), True Negative (TN), False Positive (FP), and False Negative (FN) are the
main components of the confusion matrix and as well as are needed for the
calculation of Precision, recall, and F1-score (Uddin et al., 2022). The
precision, recall, and F1-score of the DistilBERT model for both imbalanced
and balanced datasets are shown in Table 3 and Table 4.

Table 2: Confusion matrix of DistilBERT for both imbalanced and balanced dataset | Imbalanced | Balanced
---|---|---
Class Name | Safe Email | Phishing Email | Safe Email | Phishing Email
Safe Email | 3235 | 116 | 3281 | 98
Phishing Email | 24 | 2216 | 5 | 3410
Table 3: The precision, recall, and f1-score for imbalanced dataset. Class | Precision | Recall | F1-score | Support
---|---|---|---|---
Safe Email | 0.99 | 0.97 | 0.98 | 3351
Phishing Email | 0.95 | 0.99 | 0.97 | 2240
Table 4: The precision, recall, and f1-score for balanced dataset. Class | Precision | Recall | F1-score | Support
---|---|---|---|---
Safe Email | 1.00 | 0.97 | 0.98 | 3379
Phishing Email | 0.97 | 1.00 | 0.99 | 3415

Precision: Precision which expresses the ratio of true positives to all
predicted positives, indicates how accurate positive predictions are and
emphasizes the model’s capacity to identify positive samples.

Precision $\displaystyle=\frac{TP}{TP+FP}$ (1)

Recall: It calculates the ratio of true positives to all predicted positives
to assess a model’s accuracy in identifying positive samples. It denotes the
sensitivity of the model, capturing the actual positive rate in the class
observations.

Recall $\displaystyle=\frac{TP}{TP+FN}$ (2)

F1-score: A balanced statistical metric that assesses a model’s
efficaciousness in predicting positive instances while accounting for
imbalanced datasets is the F1-score, which is derived from the harmonic mean
of accuracy and recall. It offers a thorough perspective that balances recall
and precision, impacting the final score determined by the model’s projections
of the positive rate and actual positive cases.

F1-score
$\displaystyle=2\times\frac{\text{Precision}\times\text{Recall}}{\text{Precision}+\text{Recall}}$
(3)

The frequency of valid predictions in a model is quantified by accuracy, which
also represents the percentage of correctly predicted samples out of all the
samples the model generates.

$\text{Accuracy}=\frac{\text{TN}+\text{TP}}{\text{TN}+\text{TP}+\text{FN}+\text{FP}}$
(4)

The accuracy of a model can differ based on the various parameters, data
balanced or not. Initially in this experiment, an imbalanced dataset was
utilized to assess the model’s performance in both training and testing. This
dataset is highly imbalanced based on the safe email and phishing email data
where the number of data of safe email is quite higher than phishing email. To
train the model with 70% imbalanced data we have used 6 epochs with a batch
size of 32. After 6 epochs the model showed 98.90% training accuracy for this
imbalanced dataset with 0.0201 training loss. On the other hand, we get a
testing accuracy 97.50 with testing loss 0.0900 with batch size 64. Fig. 6
shows a graph of accuracy and loss with 6 epochs for an imbalanced dataset.

Figure 6: Training vs. Testing Accuracy and Loss on Imbalanced Data

Moreover, we also conducted this experiment with a resampled and balanced
dataset. After preparing the balanced dataset again used 70% of the data for
training purposes. Using this portion of data, we worked on model optimization
and fine-tuning and trained the model. This time with balanced data the fine-
tuned model showed 99.07% training accuracy with 0.0218 training loss. After
that, we tested this trained model with 30% unseen balanced data and got
98.48% testing accuracy with 0.0447 testing loss. This performance showed that
the performance of the model with a balanced dataset is quite better than an
imbalanced dataset. Fig. 7 shows a graph of accuracy and loss with 6 epochs
for a balanced dataset. The performance comparison of the model on both
balanced and imbalanced dataset shows in Table 5.

Figure 7: Training vs. Testing Accuracy and Loss on Imbalanced Data. Table 5: The Performance comparison of fine-tuned DistilBERT model on Imbalanced and Balanced dataset. Dataset Mode |  Training Loss |  Training Accuracy |  Testing Loss |  Testing Accuracy
---|---|---|---|---
Imbalanced |  0.0201 |  98.90 |  0.0900 |  97.50
Balanced |  0.0218 |  99.07 |  0.0447 |  98.48

Finally, we have compared our Phishing Email Detection study with some
previous studies in the same field in order to assess its quality. This
comparison study is summarized in Table 6, which evaluates our work from
several angles. These include the models used for machine and deep learning,
the datasets used, and whether or not the study offers human interpretation
and decision explainability for the model.

Table 6: Explainability of Different Studies on Phishing Email Detection Reference |  Datasets | Models |  Explainability Analysis
---|---|---|---
2023(Jamal et al., 2023) |  Email Spam Detection Dataset, PhishingEmailData | DistilBERT, RoBERTA |  No
2021(Alhogail and Alsabih, 2021) |  Fraud Dataset | GCN |  No
2019(Fang et al., 2019) |  IWSPA-AP | THEMIS |  No
2023(Thapa et al., 2023) |  IWSPA-AP, Nazario, Enron, CSIRO, Phisbowl | THEMIS, BERT |  No
2023(Atawneh and Aljehani, 2023) |  Phishing Email Dataset, Benign Email Dataset | BERT-LSTM |  No
This Study |  Phishing Email | Fine-tuned DistilBERT |  Yes

The comparison showed that our research work provides transparency in this
phishing email detection task. However, some listed research work showed
higher accuracy than our proposed model. This happened because they used
different datasets in their experiment. In this study, we only focused on the
phishing email dataset but other existing works were also done this work with
the spam dataset. Furthermore, to the best of our knowledge, none of the
papers listed in Table 6 offer a thorough human explanation of the
classifier’s performance or clarify how the model functions in phishing email
detection.

##  5 Explainability Analysis

LIME and Transformer Interpret both showed how the fine-tuned DistilBERT model
arrives at its prediction on phishing email detection by approximating the
model’s behaviour on every word of the email text. Lime and Transformer
Interpret provided accurate results by answering the “Why” model predicting
the selected text is a “Safe Email” or “Phishing Email”. Based on the
explainability, those explainable techniques generate word attributions by
providing positive and negative numbers on each word. Those numbers showed
which word was more specifically reasonable for the predicted class. By using
those numbers the explainable model visualizes the whole text by highlighting
the words that scored positive and as well as negative on the prediction.

For this explainability analysis in LIME explainer, we initialized the number
of features (words) 15 for the dimensionality reduction. If the number of
words of a selected test is very large that time the computational time will
increase as well as it increases the dimensionality which also can be a reason
why the systems crashed. Limiting this number of features makes the
explanation process computationally more feasible and provides a clearer and
more concise picture of which words in the case of text are influential in the
model’s prediction for a particular instance. Fig. 8 shows the model
explainability using LIME and Table 7 and Table 8 show the positive and
negative weights associated with words that help users understand phishing
email detection.

(a) Explanation for the first text

(b) Explanation for the second text

Figure 8: LIME Explanation for two Phishing Email Texts Table 7: The positive and negative weights using LIME for first text Positive Numbers | Negative Numbers
---|---
Word | Value | Word | Value
‘low’ | 1.760212017947662 | ‘neonate’ | -0.1267345497528774
‘price’ | 1.5916451492409904 | ‘3107’ | -0.013956063314091143
‘software’ | 1.2586059555064724 |  |
‘setupmefree’ | 0.9966181271429526 |  |
‘com’ | 0.3984979685547945 |  |
‘http’ | 0.1447738056917774 |  |
‘a’ | 0.06584106490791478 |  |
Table 8: The positive and negative weights using LIME for second text Positive Numbers | Negative Numbers
---|---
Word | Value | Word | Value
‘megapixel’ | 0.45124215111245936 | ‘a’ | -0.09469945665856638
‘get’ | 0.4369880334209157 |  |
‘free’ | 0.4206519368628723 |  |
‘digital’ | 0.3721259726068145 |  |
‘camera’ | 0.2388125335048898 |  |
‘canon’ | 0.19498844421129996 |  |
‘6’ | 0.04953986530836751 |  |
‘3’ | 0.024635744735869915 |  |

Along with LIME, we also examined the model interpretability with similar text
by using Transformer Interpret. During this interpretation, this Transformer
Interpret uses the tokenizer to split the large words into sub-words with the
‘##’ prefix. This happens when an unusual or non-vocabulary word is
encountered by the model, which breaks it down into smaller sub-words to
enable efficient processing. Fig. 9 shows Transformer Interpretation for a
phishing email and Table 9 and Table 10 show the word importance, with
positive and negative numbers associated with each word for two phishing
emails.

(a) Explanation for first text

(b) Explanation for second text

Figure 9: Transformer Interpret Explanation for two Phishing Email Text Table 9: The positive and negative numbers using Transformer Interpret for first Phishing Email Text Positive Numbers | Negative Numbers
---|---
Word | Value | Word | Value
‘low’ | 0.7698059423920272 | ‘.’ | -0.09537831853568463
‘price’ | 0.4465021887746993 | ‘setup’ | -0.037898186884516454
‘software’ | 0.10363795047092349 | ‘?’ | -0.04837875144898304
‘http’ | 0.05008396373925551 | ‘a’ | -0.212114259929307
‘:’ | 0.06724251662885174 | ’=’ | -0.13427331689723981
‘/’ | 0.018869178109329452 | ‘310’ | -0.06162690120821821
‘neon’ | 0.07499325538696087 | ‘##7’ | -0.013360917838605864
‘##ate’ | 0.09833698869154901 | ‘.’ | -0.1269640647507237
‘me’ | 0.09196861346171359 |  |
‘free’ | 0.16949987141093556 |  |
‘com’ | 0.20340796716052334 |  |
‘/’ | 0.006729002810807768 |  |
Table 10: The positive and negative numbers using Transformer Interpret for second Phishing Email Text Positive Numbers | Negative Numbers
---|---
Word | Value | Word | Value
‘get’ | 0.49733595483460524 | ‘a’ | -0.2688505206057748
‘free’ | 0.4645088664478948 | ‘6’ | -0.09532782298942542
‘mega’ | 0.18191195439136695 | ‘.’ | -0.08192496249808812
‘3’ | 0.014327378740528552 | ‘##pi’ | -0.11609630028309895
‘##x’ | 0.5855798135703232 | ‘!’ | -0.18838964048006548
‘digital’ | 0.021290989158880363 |  |
‘canon’ | 0.13188078942699152 |  |
‘camera’ | 0.05155301993306188 |  |
‘>’ | 0.03909299350598931 |  |

In Fig. 9 the highlighted text with green color shows the word importance to
classify the analyzed text as Phishing. On the other hand, the red portions
showed those words or sub-words that worked against the classified class. As
we worked with a binary classification task here “True Label" = 1 means
"Phishing Email" and when “True Label" = 0 means “Safe Email”.

According to the analysis, both explainable approaches explain how the
suggested model predicts "Phishing Email" for each of the two tested input
texts, but they differ in how they define explanation. LIME mostly uses small
textual modifications to evaluate the impact of the model’s prediction on the
entire text, while Transformer Interpret employs an attention mechanism to
examine each segment of the input sequence. Furthermore, in the LIME and
Transformer Interpret explanation processes, the resulting quantity of tokens,
words, or sub-words differs for the two phishing email contents. The positive
words are mainly responsible for the predicting class. More specifically, when
those words are present in a text that time model predicts a class. A
comprehensive difference between the two explainable AI techniques is shown in
Fig. 10 and Fig. 11. The number of scoring percentages for those email texts
is not the same for LIME and Transformer Interpret. This is only because the
working procedures of those techniques are not the same.

(a) LIME Scoring

(b) Transformer Interpret Scoring

Figure 10: LIME and Transformer Interpret Comparison on first text

(a) LIME Scoring

(b) Transformer Interpret Scoring

Figure 11: LIME and Transformer Interpret Comparison on second text

##  6 Discussion

The results of both an imbalanced and balanced dataset depict an enhancement
in the performance of the proposed transformer-based fine-tuned DistilBERT
model. The fine-tuned version of the DistilBERT model showed 98.90% training
and 97.50% testing accuracy on an imbalanced dataset whereas this model showed
99.07% training and 98.48% testing accuracy on the balanced dataset. Although
these results show that the model’s performance on a balanced dataset is quite
better than an imbalanced dataset, the difference is not so high,
approximately 1%. This demonstrated that the fine-tuned version performs well
in an imbalanced dataset. Additionally, the performance shows that the
optimized model performed better than other previous studies.

Besides, this research work also answered the “How was the model performed?"
and “Why was it performed?" using XAI, motivated from recently published book
by Sarker (Sarker, 2024). Two well-known Explainable and interpretable
techniques are utilized for the model understanding. It is more difficult to
understand how DistilBERT makes its predictions because, as we know, it is a
complex model. These AI techniques aided in gaining an understanding of the
model’s decision-making process. Both LIME and Transformer Interpret
highlighted the areas of the text that are more responsible for the
classification of the phishing of safe email. This highlights areas worked
with words or sub-words, making the decision more understandable for humans,
and users can also comprehend it easily. Mainly, those techniques calculated
each word or sub-words numeric value or weight where some words indicate
positive value and some negative. Words with calculated higher positive values
are more important for classifying the predicted class, while those with more
negative values do not support the predicted class. This experimental analysis
showed the transparency of the model in this phishing email classification
task.

##  7 Conclusion

This study has investigated the detection of phishing emails by utilizing a
Large Language Model. This also has shown how effective it is to use state-of-
the-art natural language processing methods to improve the precision and
effectiveness of phishing email identification. The overall experiment was
done with both a balanced and imbalanced dataset. We have obtained promising
results in differentiating between phishing and valid emails by utilizing
DistilBERT’s contextual language understanding technique. Mainly, to achieve
this good performance the model is optimized by changing some important
parameters such as batch size, optimization function, learning rate, etc, and
also fine-tuning has been done. This fine-tuned version of DistilBERT obtained
98.48% testing accuracy in this phishing email classification task.
Furthermore, LIME and Transformer Interpret techniques have been employed to
see the explanation of the model and how it makes decisions. The decision-
making process is now more visible and understandable for users because of the
usage of explainable AI, which promotes comprehension and trust. The overall
model explainability describes the robustness of the proposed model.

The field of Large Language Models has experienced quick and dynamic
development due to the substantial investment made by both businesses and
consumers. New open-source models are being produced almost every day,
demonstrating the ongoing development and innovation in this field. In the
future, we plan to expand the scope of our research by experimenting with a
wider variety of models and datasets. Furthermore, we intend to investigate
the use of different variants of BERT-based models for the identification of
different language phishing emails. Moreover, deep analysis of the model will
be conducted to ensure its robustness by enhancing the explainable AI
techniques.

## Data Availability Statement

The dataset that is used in this study is available at Kaggle (Online Link):
https://www.kaggle.com/datasets/subhajournal/phishingemails.

## References

  * Salloum et al. (2021) S. Salloum, T. Gaber, S. Vadera, K. Shaalan,  Phishing email detection using natural language processing techniques: a literature survey,  Procedia Computer Science 189 (2021) 19–28.
  * Basit et al. (2021) A. Basit, M. Zafar, X. Liu, A. R. Javed, Z. Jalil, K. Kifayat,  A comprehensive survey of ai-enabled phishing attacks detection techniques,  Telecommunication Systems 76 (2021) 139–154.
  * Sarker (2023) I. H. Sarker,  Machine learning for intelligent data analysis and automation in cybersecurity: current and future prospects,  Annals of Data Science 10 (2023) 1473–1498.
  * Han et al. (2021) K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, Y. Wang,  Transformer in transformer,  Advances in Neural Information Processing Systems 34 (2021) 15908–15919.
  * Jamal et al. (2023) S. Jamal, H. Wimmer, I. Sarker,  An improved transformer-based model for detecting phishing, spam, and ham: A large language model approach,  arXiv preprint arXiv:2311.04913 (2023).
  * Zhao et al. (2023) W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al.,  A survey of large language models,  arXiv preprint arXiv:2303.18223 (2023).
  * Yao et al. (2023) Y. Yao, J. Duan, K. Xu, Y. Cai, E. Sun, Y. Zhang,  A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,  arXiv preprint arXiv:2312.02003 (2023).
  * Koroteev (2021) M. Koroteev,  Bert: a review of applications in natural language processing and understanding,  arXiv preprint arXiv:2103.11943 (2021).
  * Singh et al. (2021) M. Singh, A. K. Jakhar, S. Pandey,  Sentiment analysis on the impact of coronavirus in social life using the bert model,  Social Network Analysis and Mining 11 (2021) 33.
  * Devlin et al. (2018) J. Devlin, M.-W. Chang, K. Lee, K. Toutanova,  Bert: Pre-training of deep bidirectional transformers for language understanding,  arXiv preprint arXiv:1810.04805 (2018).
  * Khadhraoui et al. (2022) M. Khadhraoui, H. Bellaaj, M. B. Ammar, H. Hamam, M. Jmaiel,  Survey of bert-base models for scientific text classification: Covid-19 case study,  Applied Sciences 12 (2022) 2891.
  * Lan et al. (2019) Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, R. Soricut,  Albert: A lite bert for self-supervised learning of language representations,  arXiv preprint arXiv:1909.11942 (2019).
  * Liu et al. (2019) Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov,  Roberta: A robustly optimized bert pretraining approach,  arXiv preprint arXiv:1907.11692 (2019).
  * Sanh et al. (2019) V. Sanh, L. Debut, J. Chaumond, T. Wolf,  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,  arXiv preprint arXiv:1910.01108 (2019).
  * Xu et al. (2019) F. Xu, H. Uszkoreit, Y. Du, W. Fan, D. Zhao, J. Zhu,  Explainable ai: A brief survey on history, research areas, approaches and challenges,  in: Natural Language Processing and Chinese Computing: 8th CCF International Conference, NLPCC 2019, Dunhuang, China, October 9–14, 2019, Proceedings, Part II 8, Springer, 2019, pp. 563–574.
  * Hoffman et al. (2018) R. R. Hoffman, S. T. Mueller, G. Klein, J. Litman,  Metrics for explainable ai: Challenges and prospects,  arXiv preprint arXiv:1812.04608 (2018).
  * Anan et al. (2023) R. Anan, T. S. Apon, Z. T. Hossain, E. A. Modhu, S. Mondal, M. G. R. Alam,  Interpretable bangla sarcasm detection using bert and explainable ai,  in: 2023 IEEE 13th Annual Computing and Communication Workshop and Conference (CCWC), IEEE, 2023, pp. 1272–1278.
  * Sarker (2024) I. H. Sarker, AI-driven cybersecurity and threat intelligence: cyber automation, intelligent decision-making and explainability, Springer, 2024.
  * Apruzzese et al. (2023) G. Apruzzese, P. Laskov, E. Montes de Oca, W. Mallouli, L. Brdalo Rapa, A. V. Grammatopoulos, F. Di Franco,  The role of machine learning in cybersecurity,  Digital Threats: Research and Practice 4 (2023) 1–38.
  * Yasin and Abuhasan (2016) A. Yasin, A. Abuhasan,  An intelligent classification model for phishing email detection,  arXiv preprint arXiv:1608.02196 (2016).
  * Harikrishnan et al. (2018) N. Harikrishnan, R. Vinayakumar, K. Soman,  A machine learning approach towards phishing email detection,  in: Proceedings of the Anti-Phishing Pilot at ACM International Workshop on Security and Privacy Analytics (IWSPA AP), volume 2013, 2018, pp. 455–468.
  * Hamid et al. (2013) I. R. A. Hamid, J. Abawajy, T. Kim,  Using feature selection and classification scheme for automating phishing email detection (2013).
  * Zamir et al. (2020) A. Zamir, H. U. Khan, T. Iqbal, N. Yousaf, F. Aslam, A. Anjum, M. Hamdani,  Phishing web site detection using diverse machine learning algorithms,  The Electronic Library 38 (2020) 65–80.
  * Alhogail and Alsabih (2021) A. Alhogail, A. Alsabih,  Applying machine learning and natural language processing to detect phishing email,  Computers & Security 110 (2021) 102414.
  * Brindha et al. (2023) R. Brindha, S. Nandagopal, H. Azath, V. Sathana, G. P. Joshi, S. W. Kim,  Intelligent deep learning based cybersecurity phishing email detection and classification.,  Computers, Materials & Continua 74 (2023).
  * Dewis and Viana (2022) M. Dewis, T. Viana,  Phish responder: A hybrid machine learning approach to detect phishing and spam emails,  Applied System Innovation 5 (2022) 73.
  * Fang et al. (2019) Y. Fang, C. Zhang, C. Huang, L. Liu, Y. Yang,  Phishing email detection using improved rcnn model with multilevel vectors and attention mechanism,  IEEE Access 7 (2019) 56329–56340.
  * Zhang and Li (2017) J. Zhang, X. Li,  Phishing detection method based on borderline-smote deep belief network,  in: Security, Privacy, and Anonymity in Computation, Communication, and Storage: SpaCCS 2017 International Workshops, Guangzhou, China, December 12-15, 2017, Proceedings 10, Springer, 2017, pp. 45–53.
  * Bahnsen et al. (2017) A. C. Bahnsen, E. C. Bohorquez, S. Villegas, J. Vargas, F. A. González,  Classifying phishing urls using recurrent neural networks,  in: 2017 APWG symposium on electronic crime research (eCrime), IEEE, 2017, pp. 1–8.
  * Smadi et al. (2018) S. Smadi, N. Aslam, L. Zhang,  Detection of online phishing email using dynamic evolving neural network based on reinforcement learning,  Decision Support Systems 107 (2018) 88–102.
  * Thapa et al. (2023) C. Thapa, J. W. Tang, A. Abuadbba, Y. Gao, S. Camtepe, S. Nepal, M. Almashor, Y. Zheng,  Evaluation of federated learning in phishing email detection,  Sensors 23 (2023) 4346.
  * Atawneh and Aljehani (2023) S. Atawneh, H. Aljehani,  Phishing email detection model using deep learning,  Electronics 12 (2023) 4261.
  * Jiao et al. (2019) X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, Q. Liu,  Tinybert: Distilling bert for natural language understanding,  arXiv preprint arXiv:1909.10351 (2019).
  * Lee et al. (2020) Y. Lee, J. Saxe, R. Harang,  Catbert: Context-aware tiny bert for detecting social engineering emails,  arXiv preprint arXiv:2010.03484 (2020).
  * Songailaitė et al. (2023) M. Songailaitė, E. Kankevičiūtė, B. Zhyhun, J. Mandravickaitė,  Bert-based models for phishing detection (2023).
  * Wang et al. (2023) Y. Wang, W. Zhu, H. Xu, Z. Qin, K. Ren, W. Ma,  A large-scale pretrained deep model for phishing url detection,  in: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2023, pp. 1–5.
  * Maneriker et al. (2021) P. Maneriker, J. W. Stokes, E. G. Lazo, D. Carutasu, F. Tajaddodianfar, A. Gururajan,  Urltran: Improving phishing url detection using transformers,  in: MILCOM 2021-2021 IEEE Military Communications Conference (MILCOM), IEEE, 2021, pp. 197–204.
  * Adoma et al. (2020) A. F. Adoma, N.-M. Henry, W. Chen,  Comparative analyses of bert, roberta, distilbert, and xlnet for text-based emotion recognition,  in: 2020 17th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP), IEEE, 2020, pp. 117–121.
  * Lin (2022) R. Lin,  Analysis on the selection of the appropriate batch size in cnn neural network,  in: 2022 International Conference on Machine Learning and Knowledge Engineering (MLKE), IEEE, 2022, pp. 106–109.
  * Loshchilov and Hutter (2017) I. Loshchilov, F. Hutter,  Decoupled weight decay regularization,  arXiv preprint arXiv:1711.05101 (2017).
  * Zhuang et al. (2022) Z. Zhuang, M. Liu, A. Cutkosky, F. Orabona,  Understanding adamw through proximal methods and scale-freeness,  arXiv preprint arXiv:2202.00089 (2022).
  * Holzinger et al. (2022) A. Holzinger, A. Saranti, C. Molnar, P. Biecek, W. Samek,  Explainable ai methods-a brief overview,  in: International workshop on extending explainable AI beyond deep models and classifiers, Springer, 2022, pp. 13–38.
  * Ribeiro et al. (2016) M. T. Ribeiro, S. Singh, C. Guestrin,  " why should i trust you?" explaining the predictions of any classifier,  in: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 2016, pp. 1135–1144.
  * Uddin et al. (2022) M. A. Uddin, K. T. Shahriar, M. M. Haque, I. H. Sarker,  Cyber-attack detection through ensemble-based machine learning classifier,  in: International Conference on Machine Intelligence and Emerging Technologies, Springer, 2022, pp. 386–396.