# Adaptive Deep Learning for Efficient Visual Pose Estimation aboard Ultra-
low-power Nano-drones
††thanks: This publication is part of the project PNRR-NGEU which has received
funding from the MUR – DM 118/2023.

Beatrice Alessandra Motetti1, Luca Crupi2, Mustafa Omer Mohammed Elamin
Elshaigi1, Matteo Risso1,
Daniele Jahier Pagliari1, Daniele Palossi2, Alessio Burrello1  1 Politecnico
di Torino, Turin, 10129, Italy
2 Dalle Molle Institute for Artificial Intelligence, USI and SUPSI, Lugano,
6962, Switzerland  Emails<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Sub-$10\text{\,}\mathrm{cm}$ diameter nano-drones are gaining momentum thanks
to their applicability in scenarios prevented to bigger flying drones, such as
in narrow environments and close to humans. However, their tiny form factor
also brings their major drawback: ultra-constrained memory and processors for
the onboard execution of their perception pipelines. Therefore, lightweight
deep learning-based approaches are becoming increasingly popular, stressing
how computational efficiency and energy-saving are paramount as they can make
the difference between a fully working closed-loop system and a failing one.
In this work, to maximize the exploitation of the ultra-limited resources
aboard nano-drones, we present a novel adaptive deep learning-based mechanism
for the efficient execution of a vision-based human pose estimation task. We
leverage two State-of-the-Art (SoA) convolutional neural networks (CNNs) with
different regression performance vs. computational costs trade-offs. By
combining these CNNs with three novel adaptation strategies based on the
output’s temporal consistency and on auxiliary tasks to swap the CNN being
executed proactively, we present six different systems. On a real-world
dataset and the actual nano-drone hardware, our best-performing system,
compared to executing only the bigger and most accurate SoA model, shows 28%
latency reduction while keeping the same mean absolute error (MAE), 3% MAE
reduction while being iso-latency, and the absolute peak performance, i.e., 6%
better than SoA model.

###### Index Terms:

Nano-drones, Adaptive Inference, TinyML

©2024 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

##  I Introduction

Nano-sized Unmanned Aerial Vehicles (UAVs) have emerged as a groundbreaking
technology thanks to their under-$10\text{\,}\mathrm{cm}$ size and less than
$40\text{\,}\mathrm{g}$ weight. They have opened new frontiers for
applications in GPS-denied environments, confined spaces, and close to humans
[1]. However, their miniaturization comes at the price of simplified equipped
sensory, mechanical, and computational subsystems. The power available for
computing, typically less than 100 milliwatts, is very limited compared to
standard-sized commercial drones equipped with powerful CPUs or even mobile
GPUs. Consequently, nano-UAVs typically mount Microcontroller unit (MCU)-class
processors. This poses a significant challenge when executing resource-
intensive perception, planning, and control algorithms. At the same time,
offloading these tasks to a remote server through a continuous stream of raw
sensor data is restrictive given the instability of the wireless connection
and its limited range and does not guarantee a predictable response latency
[2].

Therefore, to enable the onboard intelligence on nano-UAVs, lightweight
Convolutional Neural Networks (CNNs) offer a possible solution to address
these challenges in perception tasks. They have been shown to work effectively
even with minimal sensor data, accommodating ultra-low-power cameras with
limited resolution and dynamic range [3, 4]. Furthermore, they offer
predictable and fixed computational and memory requirements during inference,
enabling real-time operation without wireless coverage for offloading.

So far, the State-of-the-Art (SoA) autonomous navigation of nano-UAVs is
characterized by deploying one static CNN [4, 5], acting as a high-level
perception module. If, on the one hand, this approach offers a viable
solution, on the other hand, it limits the level of accuracy and computational
efficiency of the onboard perception. For some deployment scenarios, this
design choice means addressing a simple task with a too-complex CNN – wasting
computational resources and, therefore, energy – while in other cases, a
static CNN might underperform due to a too-complex environment.

Recently, adaptive machine learning has gained traction as a method to
dynamically control the trade-off between accuracy and complexity in neural
network-based applications [6]. To cope with variable-complexity inputs,
adaptive approaches build ensembles of models that are conditionally executed
based on environmental conditions: in the simplest embodiment, a more
lightweight, less memory demanding, and faster model is employed when a task-
specific policy function deems the processed datum to be relatively easy,
while a larger, more powerful model is used for more challenging inferences.

This work uses this paradigm to demonstrate the potential for optimizing a
vision-based human pose estimation task onboard a nano-UAV: the fundamental
algorithmic building block for “people monitoring” and “follow-me”
applications. To the best of our knowledge, our work proposes the first
employment of an adaptive system capable of adjusting its complexity based on
the mission’s environment. Our detailed contribution is as follows:

  * •

We propose two distinct adaptive ensembles, which combine three models from
[5]. The two reported less accurate networks are used as small models in our
ensembles, and the most accurate network as big model in both ensembles.
Further, we propose three novel adaptation policies based on the output’s
temporal consistency of pose estimation and on auxiliary tasks explicitly
tailored for visual human pose estimation aboard nano-UAVs.

  * •

We demonstrate that our adaptive algorithms, i.e., the ensembles combined with
our proposed policies, outperform the SoA results on two datasets for the same
robotic task.

  * •

We demonstrate that, compared to SoA static CNNs, our approach generates a
broader solution space with numerous intermediate points that provide a good
trade-off between accuracy and complexity compared to static models. Compared
to the most accurate SoA network, we reduce the Mean Absolute Error (MAE) on
the estimated pose by 3.15% while maintaining a constant latency. When we
consider iso-MAE performance, we instead reduce the latency of execution by
28.03%.

##  II Background & Related Work

###  II-A Human Pose Estimation aboard Nano-Drones

Our work is in the context of drone navigation tasks aimed at following a
human subject, maintaining a constant distance [4]. The input data are grey-
scale images captured by an onboard camera, while the outputs are poses then
converted into set-points for the low-level control. More specifically, we
focus on the perception sub-task, which consists of a relative pose estimation
between a human subject and a nano-UAV, approached using a CNN model. While
SoA computer vision techniques [7, 8, 9] have achieved remarkable accuracy on
this problem, they remain beyond the practical capabilities of nano-drones due
to their high computational demands ($\sim 10^{10}$ Multiply and ACcumulate -
MAC - operations) [9]. In contrast to these algorithms, which estimate
complete skeletal structures [8] or dense 3D mesh representations of
individuals [9], CNNs deployed on nano-drones focus their predictions on a
minimal set of elements: the head position in 3D space ($x,y,z$) and a
rotation angle relative to the gravity z-axis ($\phi$).

However, deploying CNNs on nano-UAVs still presents substantial challenges due
to the limited hardware resources, as mentioned in Sec. I. Traditional
solutions relying on off-the-shelf MCUs, such as those in [10, 11], can only
accommodate simple neural architectures to achieve real-time performance. For
example, the network in [11] performs a modest $27\text{\,}\mathrm{kMAC}$ per
frame at a rate of $100\text{\,}\mathrm{Hz}$. Consequently, these approaches
are primarily suitable for processing low-dimensional input signals and ill-
suited for the computational demands of camera image processing. Thus far, the
PULP-Frontnet CNN and its optimized variants [4, 5] stand as one of the few
instances where real-time visual-based human pose estimation tasks have been
successfully executed aboard a nano-UAV. These methods leverage general-
purpose ultra-low-power multi-core System-on-Chip (SoC) architectures,
tailored software deployment pipelines, and fine-tuned CNN architectures to
carry out the task efficiently. PULP-Frontnet has a complexity of
$14.7\text{\,}\mathrm{MMAC}$ and achieves an inference rate of
$45\text{\,}\mathrm{Hz}$ with a power consumption of
$92.2\text{\,}\mathrm{mW}$ when deployed on a Crazyflie 2.1 nano-drone [4].
Subsequently, in [5], Neural Architecture Search (NAS) was employed to further
statically optimize this CNN, obtaining a latency reduction of 13.2% [5]
without accuracy degradation.

All advancements in executing CNN-based vision tasks on nano-drones have
focused on static (data-independent) optimizations. This approach misses an
important opportunity since, in typical drone navigation tasks, the
environment can often change, e.g., from indoor to outdoor, from empty spaces
to crowded ones, etc, calling for a perception method that can adapt to
changes in the input data distribution. This adaptability can lead to a faster
inference for “easy” input data, which provides a higher average throughput
and, consequently, a faster response from the control part.

###  II-B Adaptive Machine Learning

The challenge of optimizing Deep Learning (DL) models for execution on ultra-
low-power edge nodes, trading off slight reductions in accuracy for
substantial gains in latency, energy efficiency, or memory usage, has been the
subject of extensive research recently [12, 13]. One significant
categorization of DL optimization techniques distinguishes them as static and
dynamic [14]. The former involves compressing a model before deployment,
either during the training phase or post-training. Static methods include
quantization, pruning, and NAS [13], now commonly used to optimize neural
networks. A critical limitation of these approaches is their inability to
adapt the inference complexity at runtime when environmental conditions
evolve.

Dynamic (or “adaptive”) inference techniques, including the approaches
presented in this paper, aim to address this limitation. An adaptive
inference’s complexity is reduced when the input data is “simple”, and a lower
amount of computation suffices to ensure high accuracy (e.g., for a vision
task, the relevant object is well-centered in the frame, not blurred, and
against a uniform background). Conversely, more compute budget is allocated
for “challenging” inputs (e.g., a moving or partially obstructed object) [6,
15, 16, 17]. The easiest way to implement an adaptive system is by
appropriately combining the outputs of two independent static models [6]: a
smaller, less complex model for easy-to-process inputs and a larger, more
accurate one for the others. Other alternatives also exist, such as using only
parts of the channels of a complex model to obtain the lightweight one [15,
16], or early-exiting after a subset of layers [17].

Regardless of the specific scheme, a crucial component for all adaptive DL
systems is a discrimination policy, which determines which model to activate
for a given input data [6]. An effective policy should provide an acceptable
loss of accuracy compared to the most complex model while resulting in a
substantial computational saving. Notably, while policies for classification
tasks have been extensively explored, the same is not true for regression
tasks. To the best of our knowledge, we are the first to propose two novel
policies that consider the temporal correlation between images captured by a
camera and an auxiliary task to gauge the input’s complexity. In Sec. III, we
delve into a detailed analysis of these policies.

##  III Materials & Methods

###  III-A Static Neural Networks

Inspired by [6], we build our adaptive system from an ensemble of two
independent “static” CNNs of different complexity and accuracy. Namely, we
consider two ensembles by combining three SoA models proposed in [5], whose
MAE, number of parameters, and NMACs are reported in Tab. I. F1 and F2 are two
versions of a convolutional network based on PULP-Frontnet composed by 7
convolutional layers, the first with a $5\times 5$ filter and the others with
a $3\times 3$ filter. The difference between the two networks is in the number
of filters (or channels) of each layer. M1.0 is a reduced version of a
classical MobileNet v1.0, pruned by a SoA NAS algorithm [13] to minimize the
number of filters. All networks have been trained on the open source dataset
of [5]. For our experiments, we combine these three CNNs in pairs to create
two different ensembles, fixing the big model to be M1.0 and changing the
small model between the other two.

TABLE I: Static models metrics extracted from [5]. Network | MAE | Params | MAC
---|---|---|---
| $x$ | $y$ | $z$ | ${\phi}$ | sum |  |
F1 | 0.27 | 0.27 | 0.28 | 0.52 | 1.34 | 14.8 k | 4.51 M
F2 | 0.21 | 0.18 | 0.24 | 0.46 | 1.10 | 44.5 k | 7.09 M
M1.0 | 0.19 | 0.14 | 0.23 | 0.48 | 1.04 | 46.8 k | 11.42 M

###  III-B Adaptive Inference for Visual Pose Estimation

Adaptive inference for regression problems must be formulated differently from
the case of classification. Most adaptive policies rely on the class
probabilities estimated by a softmax classifier function to compute
“confidence” scores [14, 6, 15, 16, 17]. This cannot be directly extended to a
regressor, which produces a pointwise estimate of continuous variables (in our
case, $x$, $y$, $z$, and $\phi$) without associated confidence. Given this
limitation, we introduce ad-hoc policies to manage inference adaptively based
on our specific problem’s characteristics. Specifically, we introduce two
novel policies, Output-based Partitioning and Auxiliary Task-based
Partitioning, to realize adaptive inference for visual pose estimation on
nano-drones.

Note that an ideal policy should execute the big model (M1.0) if and only if
it provides a lower prediction error than the small model on the currently
considered input. At the same time, the policy itself should have negligible
computational complexity compared to the execution of the two models to avoid
nullifying the gains.

####  III-B1 Output-based Partitioning

Figure 1: OP policy: the small model is always executed to compute a first set
of outputs, that are possibly corrected by the big model if they differ
significantly from the prediction at the previous time stamp.

The Output-based Partitioning (OP) policy is based on assessing changes in
predictions across consecutive time steps. As illustrated in Fig. 1, at each
new time stamp $t$, we first run the small model and monitor variations,
compared to time $t-1$, in the predicted, min-max scaled (thus dimensionless)
$x$, $y$, $z$, and $\phi$ values. Specifically, we compute:

$O_{\text{sum}}=x+y+z+\phi,\ \ OP_{t}=O_{\text{sum},t}-O_{\text{sum},t-1}$ (1)

A higher $OP_{t}$ score indicates that the relative position between human and
drone is changing rapidly, necessitating the drone to change its location.
This is associated with more challenging predictions due to the moving camera
and the subject’s distance from the center of the image. Conversely, a lower
$OP_{t}$ signifies a slowly moving or stationary drone.

Based on these considerations, we build an adaptive system that works as
follows: when $OP_{t}\leq th_{OP}$, the predictions of the small model are
used “as is”. Vice versa, the big network is executed when $OP_{t}>th_{OP}$,
and the final outputs are computed as the average between the small and big
model predictions, based on the observation that using both models
concurrently, as an ensemble, leads to enhanced accuracy. Notably, $th_{OP}$
is a tunable threshold that can be set at runtime to manipulate the prediction
error vs. complexity trade-off (lower $th$ implies more big model executions,
and vice versa).

Figure 2: Auxiliary task-based partitioning. (a) A first auxiliary CNN is
executed to localize the head, and one of the two policies is applied. (b)
Based on the policy outcome, one of the two models of the ensemble is
executed.

The critical limitation of this scheme is that the small model is always
executed since its outputs are needed to compute $OP_{t}$. Numerically, the
total cost is:

$C=C_{small}+N_{big}(th_{OP})\cdot C_{big}$ (2)

where $C_{small}$/$C_{big}$ are the NMACs or latency of the small/big models,
respectively, and $N_{big}(th_{OP})=\sum_{t}\mathbbm{1}(OP_{t}>th_{OP})$, with
$\mathbbm{1}(\cdot)$ being the indicator function, is the number of times the
big model is called.

####  III-B2 Auxiliary Task-based Partitioning

To avoid the constant execution of the small model, we propose an alternative
policy that leverages an auxiliary classification task executed before the
pose estimation. As shown in Fig. 2, we train a small additional CNN to locate
the human head in one of the grid cells superimposed to the input image. We
test grids of dimensions $2\times 2$, $3\times 3$, and $8\times 6$.

We build the auxiliary CNN as a strongly reduced version of PULP-Frontnet,
with additional stride-2 pooling layers to rapidly shrink the activation
tensors. We start from 4 convolution+pooling blocks with 8, 16, 32, and 64
filters, respectively, and a final fully connected layer for just 656 kMACs.
We then prune unimportant filters from this network using the mask-based
method from [13].

Formulating head localization as a multi-class classification allows us to
derive two adaptive inference policies. The first uses the SoA score margin
method for confidence estimation (Aux-SM), whereas the second is based on
domain knowledge and is called Head Localization-Class (Aux-HLC).

Aux-SM assumes that if the auxiliary CNN’s confidence is low in classifying
the head position, then the original pose regression task will be harder for a
given input (e.g., because the image is blurry). We estimate the auxiliary
task confidence with the score margin:

$SM_{t}=\max(Q_{i,j,t})-\mathrm{second\\_max}(Q_{i,j,t})$ (3)

where $Q_{i,j,t}$ represents the probabilities assigned to the $(i,j)$ grid
cell for frame $t$. Higher confidence values indicate that the prediction is
more reliable; therefore, the input image is easier, and pose estimation can
be done with the small model. In this case, we define a tunable threshold
$th_{SM}$ and invoke the big model if and only if $SM_{t}\leq th_{SM}$.

Figure 3: $8\times$6 grid division of the input images. In each quadrant, the
difference between the MAE of F1 and M1.0 is reported. A green square marks
the quadrant to which the head of the person belongs.

Aux-HLC assumes that pose estimation is harder when the head is in specific
locations of the image (i.e., borders/corners vs center). To create a policy
from this hypothesis, we rely on a validation set and create an error map,
which indicates how much the big model outperforms the small one in each grid
cell. Namely, each error map location contains
$E_{i,j}=MAE_{small,i,j}-MAE_{big,i,j}$, that is, the average MAE difference
computed on all validation samples for which the ground truth head location is
in cell $(i,j)$. The policy invokes the big model if and only if
$E_{\hat{i},\hat{j}}>th_{HLC}$, where $(\hat{i},\hat{j})$ is the grid cell
predicted by the auxiliary CNN at time $t$. Fig. 3 shows the error map for
models F1 and M1.0. As shown, the difference increases at the edges and even
more at the corners.

For both Aux policies, the total cost is:

$C=C_{Aux}+(1-N_{big}(th))\cdot C_{small}+N_{big}(th)\cdot C_{big}$ (4)

where $th\in\\{th_{SM}$, $th_{HLC}\\}$ and depends on the policy, and
$C_{Aux}$ is the cost of the auxiliary CNN. Given that the latter is lower
than $C_{small}$, it is evident that these policies incur a lower total cost
compared to the OP policy when the big model is invoked frequently. However,
with the auxiliary policies, the small and big model outputs are not ensembled
for hard inputs, possibly leading to a higher error.

###  III-C Target Platform

The target robotic platform is the Crazyflie 2.1, a nano-drone weighting only
$27\text{\,}\mathrm{g}$ produced by Bitcraze. The nano-drone mounts an AI-deck
board featuring a camera and the GAP8 SoC by GreenWaves Technologies [18]. The
camera acquires grayscale images up to 320x320 pixels, while the SoC extends
the computational capabilities offered by the onboard STM32 processor for
control purposes. GAP8 is a parallel ultra-low power platform [19] with
hardware support for integer-only arithmetic. It features a single-core fabric
controller (FC) and an 8-core cluster (CL). The FC orchestrates transfers
between memories and offloads computationally intense tasks to the CL. The SoC
also includes a $64\text{\,}\mathrm{kB}$ L1 memory shared between the CL cores
and a $512\text{\,}\mathrm{kB}$ L2 memory within the FC. The off-chip memories
consist of an 8 MB DRAM and a 64 MB FLASH. The transfers between DRAM and L2
are operated autonomously through a programmable micro-DMA. Transfers between
L2 and L1 are performed by a DMA that is in the CL domain.

To deploy CNNs on this platform, we exploit two tools: PLiNIO [13] and DORY
[20]. PLiNIO is a multi-optimization library that we leverage for shrinking
the auxiliary CNN channels, as well as for applying Quantization-Aware
Training (QAT) to all models to obtain integer-only implementations at 8-bit
precision deployable on GAP8. The DORY compiler automatically converts the
trained integer models from Python to C code, leveraging an optimized kernel
library. It applies hardware-aware tiling and schedules asynchronous memory
transfers between DRAM and L2, or L2 and L1, to orchestrate the execution of
the entire network.

All CNN inferences are executed on GAP8 at a frequency of 170 MHz. Predictions
are transferred via UART to the STM32 MCU, which hosts the other parts of the
closed-control loop system. The control loop involves the same four tasks of
[4]: (i) the CNN model pose estimation, the most computationally intensive
task, (ii) a Kalman filter to smooth the sequences of poses, (iii) the
velocity controller, and (iv) the low-level control for motor actuation and
stabilization. Note that this paper focuses solely on the perceptual task,
although it is expected that reducing its latency and error could also enhance
the control loop’s performance.

##  IV Experimental Results

###  IV-A Setup

For our experiments, we employed the dataset introduced in [5], which served
as the benchmark for evaluating the performance of the reference static
models. It comprises a total of 30.3k images, divided into training,
validation, and test sets in the proportions of 70%, 20%, and 10%,
respectively. We used the validation set to select the most accurate pruned
auxiliary CNN and build the error map for Aux-HLC, whereas all results are
reported on the test set. To show the generalization of our method, we also
test on a second dataset introduced in [5]. Similarly to the SoA one, this
dataset is collected with a real nano-drone in a different laboratory
environment and with different subjects. This dataset comprises 45k images
divided into 72% for training, 18% for validation, and 10% for testing. We
will henceforth refer to these two datasets as the “Known Dataset” and the
“Unseen Dataset.”

Our comparison baselines are the three static CNNs ($F^{1}$, $F^{2}$, and
$M^{1.0}$) since they constitute the SoA for this task. In the following
sections, we refer to the adaptive solution that employs $F^{1}$ as the small
model as D1 and the one that utilizes $F^{2}$ as D2.

###  IV-B Auxiliary Task Exploration

Figure 4: Auxiliary task-based policies comparison on the Known dataset.

Fig. 4 presents a comprehensive comparison of the auxiliary task policies,
considering both CNN pairs and different grid dimensions. We did not consider
finer grids because doing so would compromise the network’s ability to predict
the correct quadrant, resulting in nearly random outputs. The different
policies are compared in terms of total MAE on the four regressed variables
versus the average number of clock cycles per inference on GAP8, over the
Known dataset’s test set. Different points for the same policy refer to
different settings of the respective tunable thresholds ($th_{SM}$ and
$th_{HLC}$).

While all policies perform similarly on D1, the best results are obtained with
Aux-HLC, using an $8\times 6$ grid. For D2, the top-2 policies are Aux-HLC
with an $8\times 6$ grid and Aux-SM with a $2\times 2$ grid, with Aux-HLC
achieving optimal performance in lower MAE regions and Aux-SM performing
better in the low cycles regime. For instance, with the Aux-HLC policy, we can
obtain a MAE of 1.05, which is only 0.57% higher than the error of the big
model, while simultaneously reducing the inference cycles by 26.07%. In the
following sections, we will only consider these two optimal combinations of
type and grid size for the Aux policies results.

###  IV-C Output vs. Auxiliary Task Policies

Figure 5: OP and Aux policies comparison on the Known dataset.

In this section, we compare Aux and OP policies on the Known dataset. As a
baseline for our analysis, we consider a Random policy, that is, a “trivial”
dynamic model in which a zero-cost policy randomly selects between the
execution of the small or the big model. Fig. 5 reports the comparison results
for the two adaptive systems, revealing two distinct trends.

For D1, the most effective policy is Aux-HLC. Interestingly, the OP policy
performs worse than the random one in this case. This can be ascribed to the
low accuracy of the small model, which causes a misalignment between the
$OP_{t}$ score and the actual need to invoke the big model. The decision to
employ the big model becomes then somewhat random. This, together with the
high cost incurred by the OP policy when the big model is invoked frequently
(both models executed for most frames), yields a higher latency without
accuracy benefits.

Conversely, for D2, OP significantly outperforms the other policies, resulting
in the attainment of new SoA results. At iso-MAE with the static big model, we
observe a substantial reduction of 28.03% in inference cycles, whereas at iso-
latency, we reduce the MAE by 3.15%. Finally, our approach yields the best
known MAE (0.98) for this task, surpassing the current state of the art by
6.13%.

###  IV-D Crazyflie Deployment

TABLE II: Deployment of different dynamic models and policies on the Crazyflie
2.1.

Models | Method | MAE | Latency | % Big | Energy | Memory
---|---|---|---|---|---|---
F1 | Static | 1.34 | 7.06 ms | 0 | 0.57 mJ | 153 kB
F2 | Static | 1.10 | 8.82 ms | 0 | 0.71 mJ | 183 kB
M1.0 | Static | 1.04 | 21.76 ms | 100 | 1.92 mJ | 235 kB
D1 | Random | 1.19 | 14.41 ms | 50.0 | 1.25 mJ | 250 kB
D1 | Aux-HLC 8x6 | 1.19 | 13.24 ms | 39.1 | 1.14 mJ | 289 kB
D2 | Random | 1.04 | 21.76 ms | 100 | 1.92 mJ | 280 kB
D2 | OP | 1.04 | 15.66 ms | 31.4 | 1.32 mJ | 280 kB

In Tab. II, we present a detailed breakdown of some of the Pareto solutions
from Fig. 5 deployed on the Crazyflie 2.1. We select the $th_{*}$ value that
maximizes the latency benefit from adaptive inference for each ensemble,
comparing the corresponding solutions with the Random policy at iso-MAE.

For the D1 ensemble, the most substantial gain is achieved by the Aux-HLC
policy, invoking the big model for 39.1% of the predictions. In comparison to
the Random Policy, we achieve reductions in latency and energy consumption of
8.1% and 8.8%, respectively. Furthermore, compared to exclusively running the
big model, at the cost of a slightly higher total MAE (+0.15), we reduce the
latency and energy consumption by 39.1% and 40.6%.

For the D2 ensemble, the OP policy significantly outperforms the Random
policy. The most interesting gains emerge when the adaptive approach is
compared to the static big model. Indeed, running the big model only 31.4% of
the time with our proposed adaptive system is sufficient to maintain the same
MAE while concurrently achieving reductions in latency and energy consumption
of 28.03% and 31.25%, respectively.

Clearly, all adaptive systems incur a memory overhead since 2 or 3 (in the
case of Aux policies) CNNs are deployed on the nano-drone instead of one.
However, as shown in Tab. II, even the largest adaptive ensemble easily fits
the 512 kB L2 memory of GAP8. Note that the memory results in the table are
computed as the sum of the weights of all the deployed networks and of the
biggest activation buffer required to run inference, making the total memory
for D1 and D2 less than the sum of the composing static models.

###  IV-E Unseen Dataset Results

Figure 6: Comparative analysis of the proposed adaptive policies on the Unseen
dataset.

Finally, Fig. 6 shows the generality of our policies by presenting results on
the Unseen dataset. The same conclusions drawn from our analysis on the Known
Dataset remain valid. For the D1 ensemble, the best-performing algorithm is
Aux-HLC, which achieves the maximum latency reduction (9.2%) compared to the
Random Policy baseline at a MAE of 1.33. In contrast, for the D2 ensemble, the
OP continues to be superior. Two significant results it achieves are: (i) the
best overall MAE of 1.22, improving the SoA by 4.9%, and (ii) a 6.49% latency
reduction at iso-MAE with the big model.

##  V Conclusions

We have presented a practical use case of adaptive inference for perception
tasks aboard nano-drones, focusing on visual human pose estimation. We have
derived two adaptive systems from three SoA compact CNNs and proposed and
compared different optimal adaptive policies in different working conditions.
These policies have reduced the previous SoA error on the total regressive
loss for human pose estimation on nano-drones obtained in [5]. On the other
hand, fixing the desired MAE to 1.04 (the best result from [5]) and
considering the same hardware platform, we achieved a lower latency (15.66 ms)
within a power envelope of $90\text{\,}\mathrm{mW}$. Additionally, as shown in
Fig. 5 and 6, adaptive inference allows to obtain a rich set of intermediate
Pareto-optimal solutions in the latency vs. MAE space, selectable at runtime
by changing the value of a single tunable threshold. Future works will include
the exploration of more advanced adaptive inference techniques.

## References

  * [1] D. Palossi _et al._ , “An open source and open hardware deep learning-powered visual navigation engine for autonomous nano-UAVs,” in _International Conference on DCOSS_ , 2019.
  * [2] B. Varghese _et al._ , “Challenges and opportunities in edge computing,” in _IEEE International Conference on Smart Cloud_ , 2016.
  * [3] L. Lamberti _et al._ , “Tiny-pulp-dronets: Squeezing neural networks for faster and lighter inference on multi-tasking autonomous nano-drones,” in _IEEE International Conference on AICAS_ , 2022.
  * [4] D. Palossi _et al._ , “Fully onboard ai-powered human-drone pose estimation on ultralow-power autonomous flying nano-uavs,” _IEEE IOTJ_ , 2022\.
  * [5] E. Cereda _et al._ , “Deep neural network architecture search for accurate visual pose estimation aboard nano-uavs,” in _IEEE ICRA_ , 2023.
  * [6] E. Park _et al._ , “Big/little deep neural network for ultra low power inference,” in _International Conference on CODES+ISSS_ , 2015.
  * [7] X. Sun _et al._ , “Integral human pose regression,” in _IEEE ECCV_ , 2018\.
  * [8] D. C. Luvizon _et al._ , “2d/3d pose estimation and action recognition using multitask deep learning,” in _IEEE CVPR_ , 2018.
  * [9] R. A. Güler _et al._ , “Densepose: Dense human pose estimation in the wild,” in _IEEE CVPR_ , 2018.
  * [10] G. Shi _et al._ , “Neural-swarm: Decentralized close-proximity multirotor control using learned interactions,” in _IEEE ICRA_ , 2020.
  * [11] W. Zhao _et al._ , “Learning-based bias correction for time difference of arrival ultra-wideband localization of resource-constrained mobile robots,” _IEEE Robotics and Automation Letters_ , 2021.
  * [12] V. Sze _et al._ , “Efficient Processing of Deep Neural Networks: A Tutorial and Survey,” _Proceedings of the IEEE_ , 2017.
  * [13] D. J. Pagliari _et al._ , “PLiNIO: A user-friendly library of gradient-based methods for complexity-aware dnn optimization,” in _Forum on Specification & Design Languages (FDL)_, 2023.
  * [14] F. Daghero _et al._ , “Energy-efficient deep learning inference on edge devices,” in _Hardware Accelerator Systems for Artificial Intelligence and Machine Learning_. Elsevier, 2021.
  * [15] H. Tann _et al._ , “Runtime configurable deep neural networks for energy-accuracy trade-off,” in _IEEE CODES_ , 2016.
  * [16] F. Daghero _et al._ , “Human activity recognition on microcontrollers with quantized and adaptive deep neural networks,” _ACM Trans. Embed. Comput. Syst._ , 2022.
  * [17] S. Teerapittayanon _et al._ , “BranchyNet: Fast inference via early exiting from deep neural networks,” in _ICPR_ , 2016.
  * [18] E. Flamand _et al._ , “Gap-8: A risc-v soc for ai at the edge of the iot,” in _IEEE Int. Conf. on ASAP_ , 2018.
  * [19] F. Conti _et al._ , “An iot endpoint system-on-chip for secure and energy-efficient near-sensor analytics,” _IEEE TCAS-I_ , 2017.
  * [20] A. Burrello _et al._ , “Dory: Automatic end-to-end deployment of real-world dnns on low-cost iot mcus,” _IEEE Trans. Comput._ , 2021.


# Advancing DDoS Attack Detection: A Synergistic Approach Using Deep Residual
Neural Networks and Synthetic Oversampling

Ali Alfatemi1, Mohamed Rahouti1, Ruhul Amin1, Sarah ALJamal1, Kaiqi Xiong2,
Yufeng Xin3

###### Abstract

Distributed Denial of Service (DDoS) attacks pose a significant threat to the
stability and reliability of online systems. Effective and early detection of
such attacks is pivotal for safeguarding the integrity of networks. In this
work, we introduce an enhanced approach for DDoS attack detection by
leveraging the capabilities of Deep Residual Neural Networks (ResNets) coupled
with synthetic oversampling techniques. Because of the inherent class
imbalance in many cyber-security datasets, conventional methods often struggle
with false negatives, misclassifying subtle DDoS patterns as benign. By
applying the Synthetic Minority Over-sampling Technique (SMOTE) to the CICIDS
dataset, we balance the representation of benign and malicious data points,
enabling the model to better discern intricate patterns indicative of an
attack. Our deep residual network, tailored for this specific task, further
refines the detection process. Experimental results on a real-world dataset
demonstrate that our approach achieves an accuracy of 99.98%, significantly
outperforming traditional methods. This work underscores the potential of
combining advanced data augmentation techniques with deep learning models to
bolster cyber-security defenses.

## Introduction

The digital transformation has significantly transformed various aspects of
our lives, yet it has also ushered in numerous challenges. Among these,
Distributed Denial of Service (DDoS) attacks have emerged as one of the most
pernicious, casting a shadow on the aspirations of a connected future (Rahouti
et al. 2022). These attacks, characterized by an overwhelming surge of traffic
to a target server or network, effectively render services unavailable to
genuine users. Recent statistics illuminate the gravity of this situation: a
staggering 7.9 million DDoS attacks were recorded in just the first half of
2023 (Doe 2023). This surge not only incapacitates services but also erodes
the Quality of Service (QoS) that legitimate users are accustomed to.

Amidst this backdrop, the increasing penetration of modern networks, including
Vehicular and Internet of Things (IoT) networks, into our daily lives becomes
evident. These systems, underpinned by distributed edge clouds, have been
architected to efficiently cater to latency-sensitive services for a sprawling
IoT domain (Madi et al. 2021). However, the anticipated proliferation of IoT
devices, projected to soar to approximately 26.4 billion connections by 2026
(Ericsson 2021), augments the potential attack surface. The innate
vulnerabilities of many IoT devices, often bereft of comprehensive security
protocols, offer a ripe landscape for cyber adversaries (Cvitić et al. 2021b,
a). These devices, when manipulated, can be marshaled to orchestrate sweeping
DDoS attacks on globally distributed edge servers (Madi et al. 2021; He et al.
2021; Wang et al. 2018), posing challenges not only to individual users but
also to the broader digital ecosystem.

Over the years, various machine learning frameworks have been championed for
DDoS attack detection. Traditional stalwarts such as K-Nearest Neighbor (KNN)
(Vu, Choi, and Choi 2008), Support Vector Machine (SVM) (Cheng et al. 2009),
Random Forest (RF) (Wang, Zheng, and Li 2017), and Naive Bayes (NB) (Fadlil,
Riadi, and Aji 2017) have been extensively explored. These models, by virtue
of their design, classify patterns in network traffic to discern anomalies.
Nevertheless, their efficacy tends to wane when confronted with evolving and
intricate DDoS attack paradigms.

In pursuit of more robust solutions, recent research endeavors have gravitated
towards deep learning. The innate capability of deep learning architectures to
understand and represent complex data relationships offers a promising avenue.
Convolutional Neural Networks (CNNs), with their astute ability to parse raw
data patterns, are being actively researched for DDoS detection (Hasan, Hasan,
and Sattar 2018). Deep Autoencoders, focusing on the creation of condensed
data representations, hold potential in gleaning subtle, malicious traffic
patterns (Krishnan, Duttagupta, and Achuthan 2019). Furthermore, Artificial
Neural Networks (ANNs) (Ahanger 2017; Alzahrani and Hong 2018) utilize
nonlinear activations to delineate intricate relationships between varied data
inputs, offering nuanced insights into potential threats.

The comparative prowess of deep learning methodologies over their traditional
counterparts is underscored by recent empirical studies. For instance, a CNN
model demonstrated an accuracy rate of 92% on the NSL-KDD dataset, trouncing
both SVM, which clocked 87%, and KNN, which registered 80% (Hasan, Hasan, and
Sattar 2018). Deep Autoencoders, in another study, achieved an accuracy of
99.6% on the CIC-IDS2017 dataset, underscoring their potential for DDoS
detection (Krishnan, Duttagupta, and Achuthan 2019). However, the
cybersecurity landscape is ever-evolving, with attackers perpetually
innovating their strategies. This necessitates a continuous research impetus
to stay abreast and ensure robust defense mechanisms.

In the intricate tapestry of cybersecurity challenges, DDoS attack detection
stands out as an area demanding innovative solutions. This paper embarks on a
comprehensive journey to unravel a pioneering approach to DDoS attack
detection. At the heart of our proposed methodology lies a harmonious fusion
of the Synthetic Minority Over-sampling Technique (SMOTE) and the advanced
capabilities of Deep Residual Neural Networks (ResNets). This blend is
meticulously designed to craft a detection mechanism that encapsulates
robustness, efficiency, and adaptability, ensuring resilience against the
ever-evolving cyber adversarial tactics. Through the course of this paper, we
strive to provide an in-depth examination of this approach, highlighting its
transformative potential in the realm of DDoS attack detection.

The contributions of this paper are summarized as follows:

  1. 1.

Novel detection fusion: We present a unique integration of SMOTE with Deep
ResNets, offering a fresh perspective to DDoS attack detection mechanisms in
the literature.

  2. 2.

Adaptability: Our proposed model is not just a static solution; it’s designed
with the foresight to adapt to dynamic cyber threats, ensuring its relevance
and efficacy in the face of evolving attack strategies.

  3. 3.

Comprehensive evaluation: Through rigorous experiments on benchmark datasets,
we demonstrate the superiority of our approach over existing methods,
substantiating our claims with empirical evidence.

  4. 4.

Practical implications: Beyond theoretical contributions, this paper
underscores the real-world applicability of the proposed method, emphasizing
its potential to bolster cybersecurity defenses in diverse operational
environments.

The subsequent sections of this article are methodically laid out to provide
readers with a comprehensive understanding of the topic. Initially, Related
Work Section offers an overview of the current state-of-the-art findings in
the field. This is followed by Research Background Section Research
Background, which presents the theoretical background of the proposed
approach. Next, Methodology Section delves into the implementation specifics
of the proposed methodologies and techniques that have been employed for the
research problem presented in this work. Moving forward, Results and
Discussion Section presents the pivotal evaluation results, complemented with
in-depth discussions to provide clarity and context. Last, Conclusion Section
gives a concluding summary and encapsulates the essence of this work.

## Related Work

DDoS attacks remain one of the most formidable challenges in network security
(Rahouti et al. 2021; Owusu et al. 2023). Traditional detection methods often
struggle with the constantly evolving nature of these attacks, which are
becoming more sophisticated and distributed (Zhao et al. 2023). The rise of
Deep Learning, particularly Deep ResNets, offers a promising avenue to tackle
this issue (Ghillani 2022). ResNets, introduced by He et al. (He et al. 2016),
have the unique capability of training very deep networks by introducing skip
connections, which address the vanishing gradient problem, thereby enhancing
performance (Shaikh and Gupta 2022).

One major challenge in training deep neural networks for DDoS detection is the
imbalance in datasets (Zhu, Liu, and Zhu 2022). Real-world network traffic
data often contains a significantly larger number of benign samples compared
to malicious ones. This can result in models that are biased towards benign
classifications. To counteract this data unbalanced issue, researchers have
turned to synthetic oversampling techniques. These methods artificially
generate new samples by mimicking the characteristics of the minority class,
aiming to balance the class distribution (Wei et al. 2022). Such techniques,
when combined with ResNets, have been shown to significantly improve the
accuracy and recall rates for detecting DDoS attacks (Hussain et al. 2020).

Recent advancements in synthetic oversampling have considered the unique
structures and patterns of network traffic data. Techniques like the adapted
SMOTE for network traffic have been researched to better fit the
characteristics of DDoS traffic (Xu et al. 2023). Furthermore, the integration
of oversampling within the training pipeline of ResNets, rather than as a pre-
processing step, has been a recent innovation (Dablain, Krawczyk, and Chawla
2022), allowing the network to learn more discriminative features.

In brief, the integration of synthetic oversampling techniques with Deep
ResNets represents a cutting-edge approach to enhance DDoS attack detection.
By addressing dataset imbalance directly within the deep learning framework,
our proposed methodology holds promise in creating more robust and accurate
DDoS detection models in the face of evolving threats.

## Research Background

This section presents the theoretical background and abstraction of the
proposed methodology.

Let $D=\\{(x_{1},y_{1}),(x_{2},y_{2}),\dots,(x_{n},y_{n})\\}$ be the dataset
where

  * •

$x_{i}$ is the feature vector of the i-th sample.

  * •

$y_{i}$ is the corresponding label where $y_{i}=1$ denotes a DDoS attack and
$y_{i}=0$ denotes benign traffic.

### Synthetic Minority Over-sampling Technique (SMOTE)

SMOTE generates synthetic samples in the feature space. For each minority
sample $x$,

  1. 1.

choose one of the k nearest neighbors $x_{nn}$ from the minority class,

  2. 2.

compute the difference between the feature vectors $x$ and $x_{nn}$:
$\text{diff}=x_{nn}-x$,

  3. 3.

multiply the difference by a random number between 0 and 1:
$gap=\text{diff}\times\text{random}(0,1)$, and

  4. 4.

add the gap to the original sample to create the synthetic sample:
$x_{syn}=x+gap$.

Mathematically, for each $x_{i}$ where $y_{i}=1$:

$x_{syn}=x_{i}+\text{random}(0,1)\times(x_{nn}-x_{i})$

### Deep Residual Network

Given that ResNet structures leverage skip connections, the forward
propagation for a basic block can be represented as:

$F(x)=\mathcal{F}(x,W_{i})+x$

where

  * •

$F(x)$ is the output of the block.

  * •

$\mathcal{F}(x,W_{i})$ represents the residual mapping (could be multiple
convolutional layers).

  * •

$W_{i}$ are the weights of the block.

  * •

$x$ is the input to the block.

The overall ResNet model can be described in terms of two phases, training
phase and detection phase. Training phase: decomposed of two key steps as
follows:

  * •

Apply SMOTE on $D$ to get a balanced dataset $D_{\text{balanced}}$.

  * •

Train the ResNet model on $D_{\text{balanced}}$ to minimize the loss function
$L$, which could be binary cross-entropy for a binary classification task:

$L=-\frac{1}{n}\sum_{i=1}^{n}[y_{i}\log(\hat{y_{i}})+(1-y_{i})\log(1-\hat{y_{i}})]$

, where $\hat{y_{i}}$ is the model’s prediction for the i-th sample.

Detection phase: For a new traffic sample $x$, if the output of the network
$\hat{y}>\text{threshold}$, it is classified as a DDoS attack, otherwise
benign.

### Attention-Augmented Residual Network

To capture the interdependencies between different features in the network
traffic, we use an attention mechanism. For a layer’s output $Z$:

$\displaystyle A$ $\displaystyle=\text{Softmax}(W_{a}Z+b_{a})$ $\displaystyle
Z^{\prime}$ $\displaystyle=A\odot Z$

where

  * •

$W_{a}$ and $b_{a}$ are the attention’s weight and bias.

  * •

$A$ is the attention weights after softmax normalization.

  * •

$\odot$ represents element-wise multiplication.

  * •

$Z^{\prime}$ is the attention-augmented output.

This mechanism can be integrated into several blocks of the ResNet, helping
the network to focus more on significant features.

#### Dual-Phase Training

##### Phase 1– Pre-training on Original Data:

First, we train our attention-augmented ResNet on the original dataset $D$.
This objective function in this phase is the same binary cross-entropy:

$L_{1}=-\frac{1}{n}\sum_{i=1}^{n}[y_{i}\log(\hat{y_{i}})+(1-y_{i})\log(1-\hat{y_{i}})]$

##### Phase 2– Refinement with Augmented Data:

Post pre-training, the model is further trained on $D_{\text{balanced}}$,
which is the SMOTE-augmented dataset. However, we introduce a regularization
term to the loss function, penalizing large deviations from the initial
predictions (from Phase 1). Let $\hat{y_{i}}^{(1)}$ be the prediction from
Phase 1 and $\hat{y_{i}}^{(2)}$ be the prediction in Phase 2 for sample
$x_{i}$:

$\displaystyle L_{2}=$
$\displaystyle-\frac{1}{n}\sum_{i=1}^{n}[y_{i}\log(\hat{y_{i}}^{(2)})+(1-y_{i})\log(1-\hat{y_{i}}^{(2)})]$
$\displaystyle+\lambda\sum_{i=1}^{n}(\hat{y_{i}}^{(1)}-\hat{y_{i}}^{(2)})^{2},$

where $\lambda$ is a regularization hyperparameter. The enhanced model
consists of the attention-augmented ResNet trained with the dual-phase process
described above.

The implemented methodology is depicted in Figure 1. This figure shows the
schematic representation of the data processing and analysis pipeline for DDoS
attack detection. The process begins with data preprocessing, followed by
synthetic oversampling using SMOTE to address class imbalance. The balanced
data is then fed into a deep residual neural network for intricate pattern
recognition, leading to the detection of DDoS attacks. Details of the
implemented methodology are discussed next.

Table 1: Flow features and statistics in the CICIDS dataset. Feature |  Info
---|---
Decision tuple |  ID, src/dest IP, src/dst port, protocol
Time |  Timestamp, duration
Fwd pkts |  Total, len (total, max, min, std, min)
Bwd pkts |  Total, len (total, max, min, std, min)
IAT |  Mean, std, max, min
Fwd IAT |  Total, mean, std, max, min
Bwd IAT |  Total, mean, std, max, min
Fwd flags |  Push, URG
Flags |  Bwd (Push, URG), Count
Pkts len/size |  Pkts (min, std, max, mean, var), size (avg)
Pkt loss |  Down/up ratio
Flags count |  FIN/SYN/RST/PSH/ACK/URG/ CWE/ECE
Fwd pkt header |  Len, avg (Seg size, bytes/bulk, bulk rate)
Bwd pkt header |  Len, avg (Seg size, bytes/bulk, bulk rate)
Subflow |  Fwd/bwd (pkts, Bytes)
Init win Bytes |  Fwd, Bwd
Active/idle |  Mean, max, std, min
Other labels |  Inbound, Similar HTTP

## Methodology

### Dataset Overview

In this work, we have adopted the CIC dataset (Sharafaldin, Lashkari, and
Ghorbani 2018), which was meticulously assembled with the assistance of the
CICFlowMeter extraction tool, as indicated by (for Cybersecurity CIC).
Spanning several days, the dataset provides a wealth of network flow
information, with each entry enriched by 83 unique attributes. The entirety of
this dataset encompasses an impressive 225,745 entries.

A detailed breakdown of these attributes can be found in Table 1. Notable
features to highlight include flag_rst (denoting the Reset flag present in the
TCP header), pk_len_std (which represents the standard deviation in packet
lengths), fwd_subflow_bytes_mean (which measures the average byte size of
forward-directed subflows), flow_duration (capturing the total flow lifespan),
bwd_pkt_len_mean (giving the average byte size for packets directed backward),
and bwd_pkt_len_tot (indicating the aggregate length of packets moving in the
opposite direction). This comprehensive dataset offers valuable insights into
network behaviors and patterns, making it indispensable for our analysis.

### Data Preprocessing

#### Initial Data Handling:

Upon the initial inspection of the dataset, it was observed that certain rows
contained NaN (Not a Number) values. These values can often be indicative of
missing or corrupt data, and can thus skew the results if not addressed. To
maintain the integrity of our analysis, such rows were judiciously removed.

    data = data.dropna()

#### Transformation of Categorical Data:

Real-world datasets often comprise a mix of numerical and categorical data.
While numerical data can be directly fed into models, categorical data often
requires transformation. In our dataset, the labels ‘BENIGN’ and ‘DDoS’ were
categorical. To facilitate their processing by our deep learning model, they
were encoded into numerical values (0 and 1) using Label Encoding, a technique
that assigns each unique category a unique integer.

    L’ = LabelEncoder(L)

#### Training and Testing Data Splitting:

To objectively evaluate the performance of our model, it’s standard practice
to split the data into training and test sets. The training set, comprising
80% of the data, is used to train the model. The remaining 20% is reserved as
the test set, which is used to evaluate the model’s performance on unseen
data. This partitioning ensures a robust evaluation of the model’s
generalization capabilities.

$(X_{tr},X_{t},y_{tr},y_{t})=split(X,y,t_{size}=0.2)$

#### Feature Normalization and Scaling:

In datasets with multiple features, each feature can have a different range of
values. Some features might have values in the range of thousands while others
might lie between 0 and 1. Such disparities can disproportionately influence
the model’s learning process. To ensure each feature contributes equally,
feature scaling, specifically Standard Scaling, was applied (Alfatemi et al.
2022). Each feature value $x$ was transformed using:

$x^{\prime}=\frac{x-\mu}{\sigma}$ (1)

where, $\mu$ is the mean and $\sigma$ is the standard deviation.

In addition, any infinite values detected were replaced with the mean of their
respective columns, ensuring a consistent dataset.

### Data Augmentation using SMOTE

One of the challenges in machine learning, especially in classification tasks,
is dealing with imbalanced datasets. In real-world scenarios, it’s common for
one class to have significantly more samples than the other. This imbalance
can lead to models that are biased towards the majority class.

To combat this, we employ SMOTE. Unlike simple oversampling, which just
duplicates samples, SMOTE generates synthetic samples in the feature space.

Given a sample $x_{i}$ from the minority class, SMOTE selects $k$ of its
nearest neighbors. Let $x_{zi}$ be one of these neighbors. The synthetic
sample $x_{new}$ is then created as:

$x_{new}=x_{i}+\lambda\times(x_{zi}-x_{i})$ (2)

where $\lambda$ is a random value between 0 and 1. This method creates samples
that are coherent with the underlying data distribution.

    Initialize SMOTE
    Resample X_train and y_train using SMOTE

### Deep Residual Networks

Deep Neural Networks (DNNs) have displayed remarkable capabilities in learning
complex representations. However, as the network grows deeper, training
becomes challenging due to issues like vanishing and exploding gradients (He
et al. 2016).

ResNets introduce a novel architectural change to combat this problem. Instead
of trying to learn an underlying function $H(x)$, ResNets aim to learn the
residual function $F(x)=H(x)-x$. This is operationalized through skip (or
residual) connections that bypass one or more layers.

Mathematically, the output $y$ of a Residual Block for an input $x$ is:

$y=F(x,\\{W_{i}\\})+x$ (3)

where $F$ is the residual mapping to be learned and $\\{W_{i}\\}$ are the
weights of the block. This formulation ensures that even in cases where $F(x)$
is close to zero, the network can still perform identity mapping, ensuring
easier optimization.

#### Residual Blocks:

Each Residual Block in our network comprises two convolutional layers with
batch normalization and ReLU activations. The core idea is to have the input
$x$ bypass these layers and be directly added to their output. If there is a
dimension mismatch, a 1x1 convolutional layer adjusts the input dimensions
before the addition.

Figure 1: Schematic representation of the data processing and analysis
pipeline for DDoS attack detection. The detection workflow starts with initial
data preparation. This is succeeded by synthetic oversampling via SMOTE to
rectify class disproportion. Subsequently, a deep residual neural network
processes the equilibrated data to recognize complex patterns, culminating in
the identification of DDoS intrusions.

### Model Training and Evaluation

The detection of Distributed Denial of Service (DDoS) attacks is a critical
facet of ensuring the robustness and reliability of online systems. The
combination of deep learning and data augmentation techniques offers a
promising avenue to enhance the accuracy and reliability of such detection
mechanisms.

#### Loss Function:

In the domain of binary classification, especially with imbalanced datasets
prevalent in cyber-security, the Dice Loss, also known as the Sørensen–Dice
coefficient, emerges as an effective metric. This loss function aims to
maximize the overlap between the predicted and true labels, making it highly
suitable for our application.

Given our predicted probabilities $p$ and the true labels $y$, the Dice
coefficient $D$ can be defined as:

$D(p,y)=\frac{2\times\lvert p\cap y\rvert+\epsilon}{\lvert p\rvert+\lvert
y\rvert+\epsilon}$ (4)

Where: $\lvert p\cap y\rvert$ denotes the cardinality (size) of the
intersection between the predicted and true labels. $\epsilon$ is a small
constant added to prevent division by zero.

The Dice Loss $L$ is then the complement of the Dice coefficient:

$L(p,y)=1-D(p,y)$ (5)

The pseudo code for the Dice Loss is as Algorithm 1 :

Algorithm 1 Loss Function

1:procedure DiceLoss(inputs, targets)

2: $\text{inputs}\leftarrow\text{sigmoid}(\text{inputs})$

3: $\text{inputs}\leftarrow\text{flatten}(\text{inputs})$

4: $\text{targets}\leftarrow\text{flatten}(\text{targets})$

5:
$\text{intersection}\leftarrow\text{sum}(\text{inputs}\times\text{targets})$

6: $D(p,y)=\frac{2\times\lvert p\cap y\rvert+\epsilon}{\lvert p\rvert+\lvert
y\rvert+\epsilon}$

7: return $1-\text{dice}$

Minimizing the Dice loss during training helps in maximizing the overlap
between predicted and true labels, enhancing the model’s detection capability.

#### Optimizer (Adagrad):

The choice of optimizer can significantly influence the convergence and
overall performance of deep learning models. Adagrad is an adaptive learning
rate optimizer that is suitable for sparse data, making it a potential
candidate for cyber-security datasets.

Given the gradient $g$ at time step $t$ for a parameter $w$, the update rule
is:

$w_{t+1}=w_{t}-\frac{\eta}{\sqrt{G_{t,ii}+\epsilon}}g_{t}$ (6)

where $\eta$ is the initial learning rate, $G_{t}$ is a diagonal matrix
storing past squared gradients, and $\epsilon$ is a smoothing term to prevent
division by zero.

#### Evaluation Metrics:

After training, the model’s performance on unseen data is evaluated using
various metrics:
Precision: Measures the ratio of correctly predicted positive observations to
the total predicted positives.

$P=\frac{T}{T+F}$ (7)

where $P$ is precision, $T$ is true positive, and $F$ is false positive.
Recall: Measures the ratio of correctly predicted positive observations to all
observations in actual class.

$R=\frac{T}{T+N}$ (8)

where $R$ is recall, $T$ is true positive, and $N$ is false negative.
F1-Score: Harmonic mean of Precision and Recall, provides a balance between
them.

$F1-Score=2\times\frac{P\times R}{P+R}$ (9)

where $P$ is precision, and $R$ is recall.

ROC-AUC Score: Represents the model’s ability to distinguish between the
positive and negative classes.

## Results and Discussion

The crux of this research lies in effectively detecting DDoS events, a
persistent menace in the realm of cyber-security. By integrating the
capabilities of deep learning, specifically Deep ResNets, with the data
balancing prowess of the SMOTE, we aimed to architect a solution that
addresses the inherent challenges of the detection task.

The performance evaluation metrics provide insightful measurements regarding
the efficacy of the deep residual neural network model specifically designed
for DDoS attack detection. Notably, this model has been meticulously trained
on a dataset that has been balanced utilizing the SMOTE technique. Such
meticulous training results in the model showcasing commendable precision,
recall, F1-score, and ROC-AUC score as sown in Table 2. These metrics not only
vouch for the model’s robustness but also underscore its capacity to adeptly
identify DDoS attacks in authentic, real-world situations.

Table 2: Performance evaluation metrics of the deep residual neural network model for DDoS attack detection. The model, trained on a dataset balanced using SMOTE, demonstrates superior precision, recall, F1-score, and ROC-AUC score, highlighting its effectiveness in accurately detecting DDoS attacks in real-world scenarios. Metric | Value (%)
---|---
Accuracy | 99.98
Precision | 99.98
Recall | 99.96
F1-Score | 99.97
ROC-AUC | 1.00

### Precision and Recall: Contextual Understanding

Precision, the positive predictive value, quantifies the number of correct
positive identifications against the total number of positive identifications.
A score of 99.98% in our experiments suggests that the model is adept at
minimizing false positives, which in the context of DDoS attacks, translates
to not mislabeling benign traffic as malicious. On the flip side, recall, or
sensitivity, measures the model’s capability to identify all potential
threats. A value of 99.96% implies that the model missed a minuscule fraction
of actual attacks, making it a reliable tool in real-world scenarios.

### F1-Score and ROC-AUC Score: A Balanced Perspective

The F1-Score, a harmonic mean of precision and recall, provides a singular
metric that encapsulates the model’s balanced performance. At 99.98%, it
underscores the model’s prowess in consistently maintaining high precision and
recall. The ROC-AUC score, while being a testament to the model’s
discriminative power, also indicates its resilience to varying decision
thresholds. A perfect score of 1.00 exemplifies the model’s robustness and its
adaptability to diverse operational environments.

### Accuracy: The Overarching Metric

While precision, recall, and the F1-score offer detailed insights, accuracy
remains the most intuitive metric for overall performance. An astounding
accuracy of 99.98% is indicative of the model’s holistic capability to
correctly classify both benign and malicious traffic.

These exceptional metrics do not just stem from the architecture of the deep
residual networks or the application of SMOTE. It’s the confluence of data
preprocessing, effective feature encoding, architectural nuances of the neural
network, and the strategic oversampling of under-represented data that
collectively contribute to this performance. The results set a benchmark for
DDoS detection, but they also pave the way for future research. As cyber
threats evolve, so must our detection mechanisms. The adaptability and
performance of this model make it a formidable tool against DDoS attacks, but
there remains an ever-present need to iterate, refine, and adapt to the ever-
changing landscape of cyber-security.

### Performance Evolution Across Training Epochs

In our pursuit to understand the training dynamics of our proposed model, we
paid particular attention to its performance evolution over the course of 50
epochs. Delving into the details, the training accuracy of the model exhibited
an intriguing pattern, as depicted in the accompanying graph.

Figure 2: Training Accuracy over Epochs

In the initial phases of training, specifically the first ten epochs, there
was a pronounced and swift escalation in accuracy. This suggests that the
model was quick to grasp the underlying patterns in the data, perhaps
signifying the appropriateness of the chosen architecture or the efficiency of
the initial weights.

As shown in Figure 2, it is evident that post this swift rise, the accuracy
began to plateau. From roughly the 10th epoch to the culmination at the 50th
epoch, the growth rate in accuracy was more modest. Such behavior often points
to the model reaching a state of relative optimization, where further gains
become incremental. However, it is reassuring to note that the accuracy didn’t
display significant downward fluctuations, indicating the absence of
overfitting.

Figure 3: Training Losses over Epochs

Parallel to this, the model’s training loss, as portrayed in Figure 3,
showcased a complementary trend. A sharp decline in the loss was observable in
the early epochs, aligning with the period where accuracy saw rapid growth. As
epochs progressed, this decline in loss started tapering off, mirroring the
trend observed in the accuracy graph. Such a pattern in loss reduction is
consistent with the model’s learning process, wherein it initially corrects
major errors and then refines its parameters to address the subtler
intricacies of the data.

In comparison with prior works, the model’s behavior underscores its
resilience and adaptability. Traditional techniques, while effective to a
degree, have not showcased such pronounced improvements in initial epochs.
This not only elevates the standing of our proposed model but also hints at
its potential to be a game-changer in its domain. Nonetheless, it is essential
to approach these findings with a degree of caution. While the presented
trends are promising, further tests, especially on unseen datasets, would
provide a more comprehensive understanding of the model’s real-world
applicability.

### Effectiveness for Detection of DDoS Attacks

DDoS attacks remain a prominent threat in the digital era, with attackers
continually evolving their tactics and intensifying their onslaught. Given the
scale and potential devastation of such attacks, any delay or inaccuracy in
detection can lead to catastrophic outcomes, be it financial loss, damage to
reputation, or operational disruptions.

In our endeavor, the coupling of Deep ResNets with the SMOTE has demonstrated
a marked advancement in detection capabilities. There are several facets to
its effectiveness:

  * •

Timely detection: By harnessing the power of deep learning, our model can
rapidly process vast amounts of network traffic, ensuring that threats are
identified in real-time. Such promptness is critical in preempting and
mitigating the effects of a DDoS attack.

  * •

Reduced false positives: The precision score of 99.98% indicates the model’s
aptitude in minimizing erroneous detections. This is pivotal, as false
positives can lead to unnecessary panic, wasted resources, and can divert
attention from actual threats.

  * •

Recognition of subtle patterns: DDoS attacks, especially low-volume ones, can
often masquerade as legitimate traffic. The nuanced architecture of our neural
network, combined with a balanced dataset courtesy of SMOTE, ensures that even
these covert attack patterns are discerned.

  * •

Adaptability: The cyber threat landscape is in a state of perpetual evolution.
Our model’s architecture, combined with its training methodology, equips it
with a certain degree of adaptability. This is crucial for ensuring that the
system remains effective even as new DDoS tactics emerge.

In essence, the effectiveness of this work in detecting DDoS attacks not only
lies in its superior metrics but also in its operational advantages. In the
real-world, where every second counts during a cyber onslaught, our approach
offers a reliable, timely, and adaptable shield against DDoS threats.

## Conclusion

In this study, we tackled the challenging problem of DDoS attack detection, a
prevalent threat in today’s interconnected cyber landscape. The central
contribution of our work is the integration of synthetic oversampling
techniques with Deep ResNets, yielding a robust and highly accurate detection
system. Our approach addresses the class imbalance inherent in many cyber-
security datasets, a factor that often hampers the performance of traditional
detection algorithms. By leveraging the Synthetic Minority Over-sampling
Technique (SMOTE), we effectively balanced the dataset, enabling the model to
capture nuanced attack patterns often overlooked by conventional methods.

The deep residual network we employed further bolstered the detection process,
adapting to the intricacies of the data and offering a significant performance
boost. Experimental results confirmed the efficacy of our approach, achieving
an impressive accuracy rate of 99.98% on a real-world dataset.

Our findings underscore the importance of considering dataset imbalances in
cyber-security tasks and highlight the potential of deep learning models in
this domain. Future work could explore the integration of additional data
augmentation techniques, delve deeper into network architectures tailored for
cyber-security, and expand testing to diverse datasets to further validate the
robustness and generalizability of our approach.

## References

  * Ahanger (2017) Ahanger, T. A. 2017.  An effective approach of detecting DDoS using artificial neural networks.  In _2017 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET)_ , 707–711. IEEE.
  * Alfatemi et al. (2022) Alfatemi, A.; Peng, H.; Rong, W.; Zhang, B.; and Cai, H. 2022.  Patient subgrouping with distinct survival rates via integration of multiomics data on a Grassmann manifold.  _BMC Medical Informatics and Decision Making_ , 22(1): 1–9.
  * Alzahrani and Hong (2018) Alzahrani, S.; and Hong, L. 2018.  Detection of distributed denial of service (DDoS) attacks using artificial intelligence on cloud.  In _2018 IEEE World Congress on Services (SERVICES)_ , 35–36. IEEE.
  * Cheng et al. (2009) Cheng, J.; Yin, J.; Liu, Y.; Cai, Z.; and Wu, C. 2009.  DDoS attack detection using IP address feature interaction.  In _2009 International Conference on Intelligent Networking and Collaborative Systems_ , 113–118. IEEE.
  * Cvitić et al. (2021a) Cvitić, I.; Perakovic, D.; Gupta, B. B.; and Choo, K.-K. R. 2021a.  Boosting-based DDoS detection in internet of things systems.  _IEEE Internet of Things Journal_ , 9(3): 2109–2123.
  * Cvitić et al. (2021b) Cvitić, I.; Peraković, D.; Periša, M.; and Botica, M. 2021b.  Novel approach for detection of IoT generated DDoS traffic.  _Wireless Networks_ , 27(3): 1573–1586.
  * Dablain, Krawczyk, and Chawla (2022) Dablain, D.; Krawczyk, B.; and Chawla, N. V. 2022.  DeepSMOTE: Fusing deep learning and SMOTE for imbalanced data.  _IEEE Transactions on Neural Networks and Learning Systems_.
  * Doe (2023) Doe, J. 2023.  Top 6 DDoS Lessons for 1H 2023.  Accessed: 2023-10-20.
  * Ericsson (2021) Ericsson. 2021.  Ericsson Mobility Report - June 2021.  Accessed: 2023-10-20.
  * Fadlil, Riadi, and Aji (2017) Fadlil, A.; Riadi, I.; and Aji, S. 2017.  Review of detection DDOS attack detection using naive bayes classifier for network forensics.  _Bulletin of Electrical Engineering and Informatics_ , 6(2): 140–148.
  * for Cybersecurity (CIC) for Cybersecurity (CIC), C. I. 2017.  CICFlowMeter: Network Traffic Flow Generator Tool.  https://www.unb.ca/cic/datasets/ids-2017.html.
  * Ghillani (2022) Ghillani, D. 2022.  Deep learning and artificial intelligence framework to improve the cyber security.  _Authorea Preprints_.
  * Hasan, Hasan, and Sattar (2018) Hasan, M. Z.; Hasan, K. Z.; and Sattar, A. 2018.  Burst header packet flood detection in optical burst switching network using deep learning model.  _Procedia computer science_ , 143: 970–977.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * He et al. (2021) He, Q.; Wang, C.; Cui, G.; Li, B.; Zhou, R.; Zhou, Q.; Xiang, Y.; Jin, H.; and Yang, Y. 2021.  A game-theoretical approach for mitigating edge DDoS attack.  _IEEE Transactions on Dependable and Secure Computing_ , 19(4): 2333–2348.
  * Hussain et al. (2020) Hussain, F.; Abbas, S. G.; Husnain, M.; Fayyaz, U. U.; Shahzad, F.; and Shah, G. A. 2020.  IoT DoS and DDoS attack detection using ResNet.  In _2020 IEEE 23rd International Multitopic Conference (INMIC)_ , 1–6. IEEE.
  * Krishnan, Duttagupta, and Achuthan (2019) Krishnan, P.; Duttagupta, S.; and Achuthan, K. 2019.  VARMAN: Multi-plane security framework for software defined networks.  _Computer Communications_ , 148: 215–239.
  * Madi et al. (2021) Madi, T.; Alameddine, H. A.; Pourzandi, M.; and Boukhtouta, A. 2021.  NFV security survey in 5G networks: A three-dimensional threat taxonomy.  _Computer Networks_ , 197: 108288.
  * Owusu et al. (2023) Owusu, E.; Rahouti, M.; Hsu, D. F.; Xiong, K.; and Xin, Y. 2023.  Enhancing ML-Based DoS Attack Detection Through Combinatorial Fusion Analysis.  In _2023 IEEE Conference on Communications and Network Security (CNS)_ , 1–6. IEEE.
  * Rahouti et al. (2021) Rahouti, M.; Xiong, K.; Ghani, N.; and Shaikh, F. 2021.  SYNGuard: Dynamic threshold-based SYN flood attack detection and mitigation in software-defined networks.  _IET Networks_ , 10(2): 76–87.
  * Rahouti et al. (2022) Rahouti, M.; Xiong, K.; Xin, Y.; Jagatheesaperumal, S. K.; Ayyash, M.; and Shaheed, M. 2022.  SDN security review: Threat taxonomy, implications, and open challenges.  _IEEE Access_ , 10: 45820–45854.
  * Shaikh and Gupta (2022) Shaikh, A.; and Gupta, P. 2022.  Real-time intrusion detection based on residual learning through ResNet algorithm.  _International Journal of System Assurance Engineering and Management_ , 1–15.
  * Sharafaldin, Lashkari, and Ghorbani (2018) Sharafaldin, I.; Lashkari, A. H.; and Ghorbani, A. A. 2018.  Toward generating a new intrusion detection dataset and intrusion traffic characterization.  _ICISSp_ , 1: 108–116.
  * Vu, Choi, and Choi (2008) Vu, N. H.; Choi, Y.; and Choi, M. 2008.  DDoS attack detection using K-Nearest Neighbor classifier method.  In _Proceedings of the IASTED International Conference on Telehealth/Assistive Technologies_ , 248–253.
  * Wang et al. (2018) Wang, A.; Chang, W.; Chen, S.; and Mohaisen, A. 2018.  A data-driven study of DDoS attacks and their dynamics.  _IEEE Transactions on Dependable and Secure Computing_ , 17(3): 648–661.
  * Wang, Zheng, and Li (2017) Wang, C.; Zheng, J.; and Li, X. 2017.  Research on DDoS attacks detection based on RDF-SVM.  In _2017 10th International Conference on Intelligent Computation Technology and Automation (ICICTA)_ , 161–165. IEEE.
  * Wei et al. (2022) Wei, G.; Mu, W.; Song, Y.; and Dou, J. 2022.  An improved and random synthetic minority oversampling technique for imbalanced data.  _Knowledge-Based Systems_ , 248: 108839.
  * Xu et al. (2023) Xu, H.; Sun, Z.; Cao, Y.; and Bilal, H. 2023.  A data-driven approach for intrusion and anomaly detection using automated machine learning for the Internet of Things.  _Soft Computing_ , 1–13.
  * Zhao et al. (2023) Zhao, S.; Santana, L.; Owusu, E.; Rahouti, M.; Xiong, K.; and Xin, Y. 2023.  Enhancing ML-Based DoS Attack Detection with Feature Engineering: IEEE CNS 23 Poster.  In _2023 IEEE Conference on Communications and Network Security (CNS)_ , 1–2. IEEE.
  * Zhu, Liu, and Zhu (2022) Zhu, T.; Liu, X.; and Zhu, E. 2022.  Oversampling with reliably expanding minority class regions for imbalanced data learning.  _IEEE Transactions on Knowledge and Data Engineering_.