# DroneVis: Versatile Computer Vision Library for Drones

Ahmed Heakl<EMAIL_ADDRESS>0009-0009-8712-1457 Egypt-Japan
University of Science and TechnologyAlexandriaEgypt21934 ,  Fatma Youssef
<EMAIL_ADDRESS>Egypt-Japan University of Science and
TechnologyAlexandriaEgypt21934 ,  Victor Parque<EMAIL_ADDRESS>Waseda
UniversityDepartment of Modern Mechanical EngineeringTokyoJapan169-8050 and
Walid Gomaa<EMAIL_ADDRESS>Egypt-Japan University of Science and
Technology and Alexandria UniversityAlexandriaEgypt21934

###### Abstract.

This paper introduces DroneVis, a novel library designed to automate computer
vision algorithms on Parrot drones. DroneVis offers a versatile set of
features and provides a diverse range of computer vision tasks along with a
variety of models to choose from. Implemented in Python, the library adheres
to high-quality code standards, facilitating effortless customization and
feature expansion according to user requirements. In addition, comprehensive
documentation is provided, encompassing usage guidelines and illustrative use
cases. Our documentation, code, and examples are available in
https://github.com/ahmedheakl/drone-vis.

Unmanned Aerial Vehicles (UAVs), Drones, Computer Vision, User interface
systems and human-computer interaction.

††ccs: Computer systems organization Real-time system specification††ccs:
Computing methodologies Computer vision problems††ccs: Applied computing
Surveillance mechanisms

##  1\. Introduction

Drones, also referred to as Unmanned Aerial Vehicles (UAVs), have emerged as
invaluable assets in various fields, including agriculture, environmental
monitoring, disaster response, and surveillance (Mohsan et al., 2023; Madhavan
et al., 2018; Barbedo, 2019). Their true potential lies in their capacity to
comprehend their surroundings and make intelligent decisions. This potential
is realized through the integration of computer vision algorithms into drones,
enabling them to execute a diverse range of tasks based on the live camera
feed they capture (Arafat et al., 2023; Mohsan et al., 2023; Akbari et al.,
2021).

Previous research has focused on enabling object detection and tracking on
drones, particularly in surveillance applications. For instance, person
detection and tracking is implemented in (Rohan et al., 2019) using a single-
shot detector (SSD) on Parrot drones, achieving a remarkable frame rate of 58
frames per second (Rohan et al., 2019). Another framework, known as Deep Drone
(Han et al., 2016), incorporated object detection using Faster R-CNN at 1.6
frames per second and detection using the Kernelized Correlation Filter (KFC)
algorithm at 70 frames per second (Henriques et al., 2012, 2014).

While object detection and tracking are crucial functionalities for drones,
they represent only a fraction of the essential tasks that these devices can
perform. Tasks such as crowd counting for rescue and security operations,
action recognition for surveillance applications, depth estimation to enhance
spatial awareness, and face detection and pose estimation for surveillance
purposes hold equal significance. Unfortunately, prior work predominantly
concentrated on implementing specific models for object detection and
tracking, neglecting the need for a diverse range of models and tasks tailored
to users’ specific requirements (Rohan et al., 2019; Han et al., 2016).

Recognizing these gaps, we introduce DroneVis, a versatile library designed to
empower researchers and practitioners in the field of drone-based computer
vision. DroneVis facilitates the execution of computer vision tasks,
especially on Parrot AR2 drones (Parrot, 2012) depicted in Fig. 1. This
unmanned aerial vehicle is highly user-friendly, versatile, and cost-
effective, making it a prime choice for numerous applications. Featuring an
integrated camera with 720p high-definition live video streaming and
recording, this drone can capture high-quality visual data for a wide range of
applications. Furthermore, its built-in Wi-Fi module allows seamless
connectivity to laptops through the drone’s Wi-Fi network, offering an
impressive communication range of up to 50 meters.

Figure 1. Parrot AR2 drone.

DroneVis offers a selection of models for each task, providing users with the
flexibility to customize their approach according to their preferences. To
ensure robustness, DroneVis includes an extensive set of test cases.
Additionally, we provide comprehensive documentation that outlines the
library’s features, installation instructions, and an overview of the various
tasks and models it encompasses.

Our main contributions can be summarized in the following points:

  * •

Introducing DroneVis, a versatile library for automating computer vision tasks
such as object detection, tracking, segmentation, crowd counting and depth
estimation on Parrot drones.

  * •

Offering comprehensive documentation for the library and practical usage
examples enabling straightforward implementation.

  * •

Rigorous testing, ensuring the library’s reliability with an impressive 83%
test coverage.

  * •

User-friendly control options catering to users without extensive programming
knowledge.

  * •

Development of a gesture recognition model for intuitive remote drone control.

  * •

Provision of default models for various computer vision tasks, with the
flexibility to customize and change these default models based on user
preference.

  * •

Development of superior face detection, especially suitable for detecting
distant faces in drone applications.

The rest of the paper is organized as follows. Section 2 elaborates on the
features offered by the DroneVis library. Section 3 discusses drone control
and the available user interfaces. Section 4 provides a comprehensive
exploration of the array of tasks that DroneVis facilitates, along with the
diverse models tailored for each task. In Section 5, we conduct a comparative
analysis with other relevant software solutions. Section 6 presents an
integration example between the drone and computer vision models in autonomous
navigation using DroneVis. Finally, Section 7 serves as the conclusion,
shedding light on potential avenues for future research and development.

##  2\. Library Features

In this section, we elaborate on the key features of DroneVis and its
contribution to the advancement of computer vision-enabled drone applications.

###  2.1. Integrated State-of-the-Art Computer Vision Algorithms

DroneVis serves as a robust repository of state-of-the-art computer vision
models, thereby offering users a versatile toolkit for a wide range of tasks.
These models encompass renowned architectures, such as Faster R-CNN, YOLOv8,
BlazeFace, just to name a few. With capabilities spanning from object
detection/recognition, action recognition and crowd counting to face
detection, depth estimation, and pose estimation, the library empowers users
to effortlessly select and employ the most suitable model for their specific
application requirements.

###  2.2. Documentation

DroneVis stands out in documentation with two key features:

  * •

Contributor-Friendly Codebase: Meticulously documented code empowers users to
easily customize and extend the library, supporting tasks from hyperparameter
tuning to creating new classes.

  * •

Comprehensive User Resources: Extensive documentation on
GitHub111https://github.com/ahmedheakl/drone-vis and
ReadTheDocs222https://drone-vis.readthedocs.io/en/latest/index.html includes
guides on model adjustments and use-case scenarios, fostering collaboration
and contribution within the user community.

###  2.3. Demo for Testing

DroneVis features a dedicated demo simulation to validate the library’s
robustness and performance. This application is designed to facilitate
rigorous testing and evaluation of the library’s real-time data handling,
inference accuracy, and detection capabilities. The library achieves a minimum
frames-per-second (fps) rate of 4.5 on an Intel Core 8 CPU, demonstrating its
capability to deliver real-time results.

###  2.4. Test Coverage

DroneVis boasts a high code coverage and employs type hints throughout its
implementation to ensure the integrity and reliability of the library’s
codebase. This quality assurance measure reinforces code robustness and
reliability, with code coverage and quality metrics consistently exceeding 80%
with pytest framework (Krekel et al., 2004).

###  2.5. Wide Variety of Drone Control

DroneVis extends its utility beyond computer vision algorithms by providing
comprehensive control over the drone’s operations. Users can seamlessly
initiate critical actions such as takeoff, landing, reset, emergency response,
and calibration. Furthermore, the library equips users with the capability to
maneuver the drone along various spatial dimensions, enabling movement in
directions encompassing right, left, up, down, forward, and backward.

###  2.6. High Quality Implementation

DroneVis adheres to PEP8 coding standards, ensuring a structured and coherent
codebase. This commitment enhances readability, simplifies comprehension, and
fosters a collaborative development environment, empowering users for
confident contributions and library expansion.

###  2.7. Multiple User Interfaces

Recognizing the diverse preferences of users, DroneVis presents a variety of
user interfaces to cater to different usage scenarios. Users can seamlessly
interact with the library through a graphical user interface (GUI), a command-
line interface (CLI), or an innovative gesture-based interface. The latter
enables users to control the drone’s actions using intuitive hand gestures,
showcasing the library’s commitment to enhancing user experiences.
Additionally, for those who prefer more traditional methods, DroneVis also
supports control of the drone via keyboard and joysticks.

###  2.8. User Friendly API

DroneVis provides a user-friendly and intuitive API, specifically designed to
speed up the process of developing customized computer vision applications on
drones. The API’s simplicity empowers users to easily integrate their
algorithms with the library’s framework. As a demonstration, consider the
following code snippet, which showcases the ease of utilizing the
YOLOv8Detection model for object detection within DroneVis:

⬇

"""Example Usage: Object Detection with YOLOv8"""

from dronevis.models import YOLOv8Detection

model = YOLOv8Detection() # create mode instance

model.load_model() # load pre-trained model

model.detect_webcam() # start detection on webcam

##  3\. Drone Control and User Interface

In this section, the available user interfaces and alternative control
mechanisms will be discussed in detail, highlighting their features and use
cases. Additionally, we explore how the library facilitates connectivity
between the laptop and the drone to enable these control mechanisms.

###  3.1. Graphical User Interface (GUI)

The library is built-in with out-of-the-box GUI to ease the use of the
proposed computer vision control and gives the user real-time, full access
over their drone. The GUI is built with Tkinter for its ease of use, fast
response, and rendering. Moreover, it is a cross-platform library allowing
smoother integration with almost all platforms such as Linux, Windows, and
MAC. To commence, the user should only initiate the following command within
the terminal:

$ dronevis-gui
This action will result in the GUI window opening, as depicted in Fig. 2.

Figure 2. DroneVis GUI.

The GUI depicted in Fig. 2 facilitates comprehensive drone management,
allowing users to control drone movement through both basic, specialized, and
reset control sections. It also provides the capability to execute various
computer vision tasks on the drone’s camera feed using the vision control
section.

###  3.2. Command Line Interface (CLI)

The library also incorporates CLI, a resource-efficient user interface that is
favoured for devices with limited computing power. CLI efficiently manages
processing and memory resources, ensuring effective communication with the
drone. CLI is designed mainly for systems that lack a graphical user interface
(GUI) window, making it well-suited for remote services and situations where
GUI interactions are not feasible. Unlike GUI, CLI relies solely on text-based
commands for controlling drone movement. It doesn’t include graphical
features, which are common in GUI, especially in the context of vision tasks.
To start the CLI, the user should simply write the following command from the
terminal:

$ dronevis
Executing this command will bring up the CLI Interface shown in Fig. 3.

Figure 3. DroneVis CLI.

###  3.3. Keyboard and Joystick Control

In addition to the aforementioned user interfaces, DroneVis supports drone
control via a keyboard and joystick. This control method is suitable for users
who are familiar with traditional remote control devices and offers a tactile
and responsive way to operate the drone. Keyboard and joystick control is
valuable for tasks requiring precise manoeuvring.

###  3.4. Hand Gesture Drone Control

While user interfaces provide comprehensive access to drone functions,
including movement control and various computer vision tasks through the
drone’s camera, they necessitate proximity to a computer device. In cases
where simple drone movement control is sufficient, hand gesture control can
offer enhanced intuitiveness and interactivity compared to traditional user
interfaces. The DroneVis library enables straightforward drone control using
hand gestures, as illustrated in Fig. 4. To achieve this capability, the
Mediapipe hands model (Zhang et al., 2020) is employed to extract the
keypoints of each hand, yielding 3D coordinates for 21 keypoints. Six distinct
gestures are utilized for drone control, as showcased in Fig. 4. A small
dataset comprising 328 images encompassing these six gestures is collected for
training a gesture classifier model, as depicted in Fig. 5. The model’s input
consists of the 63-dimensional keypoints extracted by Mediapipe ($21\times
3$), representing the $x$, $y$, and $z$ coordinates of the $21$ keypoints. The
classifier network incorporates a single fully connected layer with 50 units,
followed by a leaky ReLU activation, and an output layer with 6 units
corresponding to the six gestures, followed by a softmax activation. Layer
count and sizes are empirically determined for optimal performance, ensuring
real-time feasibility for drone control. The dataset is partitioned into 80%
training, 10% validation, and 10% testing sets, with label stratification to
account for minor class imbalances. An exemplar output classification is
demonstrated in Fig. 4, showcasing the associated class label.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 4. Hand gestures for drone control. Figure 5. Gesture classifier
network.

###  3.5. Drone Connectivity

This section focuses on creating a robust connection between the laptop and
the drone, facilitating seamless communication during the drone’s flight.

Upon linking to the drone’s Wi-Fi network, separate sockets are initiated for
each of the drone’s dedicated ports, each designed for a distinct purpose.
Notably, one socket focuses on video streaming, while another handles control
commands, and yet another gathers navigation data. These sockets are
instrumental for transmitting requests and retrieving data, maintaining a
seamless line of communication during the drone’s flight. To ensure real-time
and responsive control, DroneVis implements separate threads for each
connection type—command, video, and navigation data. Each thread aligns with a
designated port, enhancing the efficiency of drone communication.
Comprehensive details regarding drone connections can be explored in the drone
connection section on the ReadtheDocs platform333https://drone-
vis.readthedocs.io/en/latest/connection/drone_connection.html.

##  4\. Tasks and Models

The DroneVis library offers a comprehensive collection of models designed for
diverse computer vision tasks. In this section, we provide an in-depth
analysis of each task, elucidating the specific nature and requirements of the
task at hand. Furthermore, we present an overview of the neural network models
available within the library for each task, where we utilize the available
pre-trained weights rather than training the models from scratch, emphasizing
their respective strengths and capabilities. Additionally, we evaluate and
recommend the most suitable model for each task based on its accuracy and
computational efficiency, considering factors such as running time.

To test the capabilities of the DroneVis library, we conducted a series of
experiments utilizing a drone equipped with a high-quality camera of 720p. The
drone was flown within the confines of the Egypt-Japan University of Science
and Technology campus, capturing real-world footage that served as the basis
for numerous computer vision tasks. This real-world data allowed us to
evaluate the performance and applicability of the library in practical
scenarios.

For certain tasks, such as object detection, tracking, pose estimation,
segmentation, and depth estimation, we relied on our drone footage as the
primary source of input data. The high-resolution imagery obtained from the
drone’s camera was instrumental in these tasks, providing rich visual data for
analysis. The data collection process adhered to ethical guidelines and
included verbal consent from the individuals involved. Subjects were informed
about the research objectives and the intended use of the aerial footage, and
their voluntary verbal consent was obtained. The study was conducted in
compliance with the research ethics of Egypt-Japan University of Science and
Technology. We also respect the terms and conditions of the data usage
agreement with the university and the participating individuals.

However, there were scenarios in which we encountered limitations. Action
recognition, for instance, presented challenges as the well-known action
recognition dataset include a wide range of complex actions, that could not be
feasibly performed within the confines of the university campus. In such
cases, we turned to test videos from well-known datasets to assess the
library’s performance. Additionally, for lane and road detection, we faced
regulatory restrictions that prohibited drone flight outside the university
campus. Consequently, we could not rely on our drone footage for these tasks,
and instead, we utilized test videos from established datasets to measure the
models’ performance.

###  4.1. Object Detection

Object detection is a fundamental task in computer vision, particularly in the
context of Parrot drones. It involves the automatic identification and
localization of multiple objects within images or video streams. By employing
advanced algorithms and techniques, object detection enables Parrot drones to
navigate and avoid obstacles autonomously. The significance of object
detection lies in its wide range of applications, such as aerial surveillance,
search and rescue operations, agriculture monitoring, and autonomous
navigation in complex environments.

Several popular object detection models include Faster R-CNN, SSD, and YOLO.
Faster R-CNN and SSD are anchor-based models that rely on predefined anchor
boxes of varying scales and aspect ratios to identify objects within images.
These models perform regression to align the anchor boxes with ground truth
bounding boxes, achieving precise localization. In contrast, YOLO (You Only
Look Once) operates differently. It divides the image into a grid and predicts
bounding boxes and class probabilities for each grid cell.

In this paper, we delve into the details of the above-mentioned models, and
explore their effectiveness in automating computer vision algorithms on Parrot
drones. Fig. 6(d) illustrates the output of these different object detection
models.

(a) YOLOv8

(b) YOLOv5

(c) Faster RCNN

(d) SSD

Figure 6.  Output of object detection using various models.

####  4.1.1. Faster R-CNN Model

The Faster Regional Proposal CNN (Faster R-CNN)(Ren et al., 2015) improves
upon R-CNN(Girshick, 2015) and Fast R-CNN (Girshick et al., 2014) by
integrating region proposal generation within the network architecture. Unlike
traditional methods using selective search (Uijlings et al., 2013), Faster
R-CNN introduces a Region Proposal Network (RPN) that shares convolutional
layers with the object detection network. The RPN efficiently generates region
proposals by sliding a small network window over the feature map,
simultaneously predicting objectness scores and refining bounding box
coordinates. This architectural enhancement allows end-to-end training and
faster inference by sharing convolutional layers between the RPN and the
object detection network.

####  4.1.2. Single Shot MultiBox Detector (SSD) Model

SSD (Liu et al., 2016) innovates by utilizing a single network to predict
object bounding boxes and class probabilities directly from feature maps at
multiple scales. Through default bounding boxes of various aspect ratios and
scales at each feature map location, SSD efficiently detects objects of
different sizes. The network integrates initial convolutional layers for low-
level feature extraction, auxiliary layers tailored for object detection, and
convolutional predictors designed to forecast detections. Default boxes with
various aspect ratios contribute to multi-scale feature maps, enhancing the
network’s object detection capabilities. In contrast to Faster R-CNN, SSD
simplifies the pipeline and reduces computational costs by performing object
detection in a single pass, eliminating the need for a separate region
proposal network.

####  4.1.3. You Only Look Once (YOLO) Model

YOLO (Redmon et al., 2016) is the most popular and efficient model in computer
vision. Introduced in 2015 to be trained end-to-end, it aimed at real-time
object detection and classification. The model family belongs to one-stage
object detection models that process an entire image in a single forward pass
of a CNN. Unlike two-stage detection models such as R-CNN and its variants –
first propose regions of interest and then, classify these regions- YOLO
processes the entire image in a single pass, making it exceedingly faster. In
this work, we chose to incorporate YOLOv5 (Jocher, 2020) and YOLOv8 (Jocher et
al., 2023) for the sake of comparison because of their reliable detection and
speed.

YOLOv5 (Jocher, 2020) builds upon the YOLOv3 head network and introduces the
EfficientDet backbone network, resulting in significant improvements in
detection speed and accuracy. It incorporates dynamic anchor assignment to
better fit object sizes, improved data augmentation for challenging
conditions, and a modified non-maximum suppression algorithm for more
efficient and accurate detection. With its flexible and Pythonic structure,
YOLOv5 became the world’s state-of-the-art repository for object detection in
2020.

YOLOv8 (Jocher et al., 2023) is the last model in the YOLO series (at the time
of developing our work), surpassing all of them in both accuracy and speed.
YOLOv8 introduced minor changes, e.g., removal/addition of some CNN layers and
changing the kernel sizes), yet the major change was anchor-free detections.
YOLOv8 predicts the center of an object directly instead of the offset from a
known anchor box. It is more flexible as it does not require the manual
specification of anchor boxes, which can be difficult to choose and can lead
to sub-optimal results in previous models of YOLO. In addition, YOLOv8
introduced multiple models for solving other common tasks in computer vision –
Instance Segmentation, Image Classification, and Object Tracking.

Table 1 provides a comparative analysis of the performance of the previously
mentioned object detection models, evaluating their speed in frames per second
(fps) and accuracy in average precision (AP), which measures how well the
model correctly detects and localizes instances. It is evident that, in terms
of speed, YOLO v8 outperforms all other models, achieving an impressive frame
rate of 80 frames per second (fps), while the slowest model, Faster R-CNN,
lags significantly behind at a mere 15 fps. Additionally, YOLO v8 exhibits
exceptional detection capabilities, with an average precision of 53.9%.

###  4.2. Object Tracking

Object tracking is the process of locating a certain object or multiple
objects in a video stream overtime while keeping a unique identity to each
object. Object tracking has numerous applications such as video surveillance.
Multi-object detection composes of detecting the objects in each frame,
localizing them, and associating similar objects in different frames. It can
be seen that the accuracy of the tracking mainly depends on the detection
model used. In this work, we employ YOLOv8 (Jocher et al., 2023), offering a
tracking mode through two cutting-edge algorithms: BoT-SORT (Aharon et al.,
2022) and ByteTrack (Zhang et al., 2022).

ByteTrack stands out from other tracking algorithms as it utilizes all
detected bounding boxes, even the ones with low scores, enabling recovery of
objects with low score detection. It leverages the Kalman filter for
predicting new box locations and associates bounding boxes with predicted
tracklets, prioritizing high score boxes. BoT-SORT builds upon ByteTrack and
improves accuracy by employing a more precise Kalman filter and incorporating
camera motion compensation, effectively reducing box identity switches.
Experiments on the MOT17 benchmark yield a Multi-object Tracking Accuracy
(MOTA) of 80.6 for both ByteTrack and BoT-SORT algorithms. Fig. 7 shows the
output of BoT-SORT tracker (default tracker for YOLO v8) in the first and
twentieth frames.

(a) First frame.

(b) Twentieth frame.

Figure 7.  Output of tracking using YOLOv8.

###  4.3. Pose Estimation

Pose estimation aims to detect the positions of specific points, known as
keypoints, within an image. These keypoints often correspond to significant
features, such as joints, landmarks, or distinctive elements of an object. The
positions of these keypoints are typically represented using 2D or 3D
coordinates. The outcome of a pose estimation model consists of the keypoints
on the object within the image, often accompanied by confidence scores for
each point. Pose estimation is an ideal solution when the objective is to
recognize particular parts of an object within a scene and understand their
spatial relationships with respect to each other. For this task, we offer two
state-of-the-art models in pose estimation: YOLOv8-pose (Jocher et al., 2023)
and MediaPipe pose estimation (Bazarevsky et al., 2020). An example of the
output of YOLOv8 pose model and MediaPipe model are shown in Fig. 8.

####  4.3.1. YOLOv8-Pose

Another feature provided by YOLOv8 is pose estimation. For this task, YOLOv8
was pretrained on the COCO dataset (Lin et al., 2014). The model outputs 17 2D
keypoints with a mAP of $50.4$. The mAP metric quantifies how effectively the
model’s predicted masks align with the ground truth masks for each object
instance within the image. The model runs in real-time and accurately
estimates the pose even in crowd scenes.

####  4.3.2. MediaPipe Pose Estimation

MediaPipe Pose Estimation is based on the Blazepose architecture (Bazarevsky
et al., 2020). Unlike YOLOv8-Pose, MediaPipe provides 33 3D keypoints in real-
time. These keypoints are a superset of the 17 keypoints provided by YOLOv8
(COCO dataset keypoints), and they also include keypoints for the face, hands,
and feet (found in BlazeFace (Bazarevsky et al., 2019) and BlazePalm (Zhang et
al., 2020)). The pipeline of this pose estimation involves first detecting a
person in the image using a face detector and then predicting the keypoints,
assuming that the face is always visible. MediaPipe Pose Estimation is mainly
designed for fitness applications for a single person or a few people in the
scene so it suffers in detecting multiple persons in an image as shown in Fig.
11(b).

(a) Yolov8 Pose.

(b) Mediapipe Pose.

Figure 8.  Output of pose estimation.

###  4.4. Instance Segmentation

Instance segmentation goes beyond object detection by not only identifying
individual objects in an image but also accurately segmenting them from the
surrounding background. The result of an instance segmentation model is a
collection of masks or contours outlining each object in the image,
accompanied by class labels and confidence scores for each object. Instance
segmentation is particularly valuable when detailed information about object
boundaries and shapes is necessary, in addition to their spatial locations
within the image (Yao et al., 2020). For this task, DroneVis offers two
models: YOLOv8 segmentation and MediaPipe segmentation. The results produced
by these two models are visually depicted in Fig. 9(b).

(a) Yolov8 Segmentation.

(b) Mediapipe Segmentation.

Figure 9.  Instance segmentation output.

####  4.4.1. YOLOv8 Segmentation

Within YOLOv8, a dedicated segmentation module is available, trained on the
COCO dataset (Lin et al., 2014). This segmentation module demonstrates a
noteworthy mean average precision (mAP) of 44.6. Moreover, it operates at an
impressive processing speed of 67 frames per second. The results of YOLOv8
segmentation are visually presented in Fig. 9(a), showcasing the module’s
proficiency in accurately segmenting all subjects within the image.

####  4.4.2. MediaPipe Segmentation

MediaPipe, in addition to its primary function of pose estimation 4.3, offers
the capability of subject segmentation. By enabling the enable_segmentation
parameter within the pose model (Google, 2023), MediaPipe can detect and
segment a single subject while predicting their pose, as shown in Fig. 9(b).
However, it faces limitations when detecting and segmenting multiple subjects
within the same image, as observed in pose estimation. MediaPipe’s design,
tailored for single individual detection, presents challenges in simultaneous
detection and segmentation of multiple subjects. In terms of accuracy and
processing speed, MediaPipe achieves a segmentation accuracy rate of 96.21%
and processes at 21 frames per second. These results were obtained using their
proprietary ’Selfie Dataset,’ which is not publicly available (Google, 2023).

###  4.5. Face Detection

Face detection focuses on the identification and localization of human faces
within images or video frames. By utilizing advanced algorithms and machine
learning models, face detection can analyze pixel patterns in an image to
detect facial features like eyes, nose, and mouth, enabling the determination
of the presence of a human face. The importance of face detection becomes
particularly evident in various applications, especially in the context of
drones, particularly in surveillance scenarios (Ding and Tao, 2016).

Within DroneVis, we provide diverse face detection models, each tailored to
specific needs. Some of these models excel in identifying faces close to the
camera, such as the MediaPipe Face model. Meanwhile, other models exhibit
reliable performance in scenarios featuring faces at varying distances from
the camera. This group includes the Haar Face Detector, Hog Face Detector,
Dlib CNN Face Detector, and OpenCV DNN Face Detector. Furthermore, we offer a
robust face detection model designed to handle distant faces, making it
particularly suitable for drone applications where subjects are typically
situated at a considerable distance from the drone. This model is YOLOv8 fine-
tuned for face detection, ensuring high accuracy even in challenging
scenarios. The outputs of the diverse face detection models are depicted in
Fig. 10(d). In the first row, the results pertain to faces in close proximity
to the camera, an instance where all models exhibit robust performance.
Meanwhile, the second row showcases the outcomes for faces positioned at a
considerable distance from the camera, featuring the outputs of the Haar Face
Detector Model and the fine-tuned YOLOv8 Model. Notably, the remaining models
encountered limitations in detecting faces at such distances and were
consequently omitted from the figure. The subsequent sections provide a
comprehensive exploration of the aforementioned models’ characteristics and
capabilities.

(a) MediaPipe face model output for close faces.

(b) Other models output for close faces.

(c) Haar face detector output for far faces.

(d) Fine-tuned YOLOv8 for far faces.

Figure 10.  Output of various face detection models for near and distant
faces.

####  4.5.1. MediaPipe Face

MediaPipe Face Classifier, an advanced deep learning-based face detection
approach by Google researchers (Bazarevsky et al., 2019). Using a multi-task
learning framework, it optimizes face detection and classification tasks,
combining CNNs and RNNs for feature extraction and face categorization. The
method excels in diverse poses and lighting conditions, achieving 99.63%
accuracy with a low false positive rate of 0.17 on the ”Labeled Faces in the
Wild” (LFW) dataset (Huang et al., 2007). Computationally efficient, suitable
for deployment on resource-constrained devices like smartphones or drones, the
method primarily focuses on close-up faces and may struggle with distant
faces, limiting its applicability in scenarios such as drone-based
applications

####  4.5.2. Haar Face Detector

The Haar classifier is based on the Haar-like features proposed by Viola and
Jones (Viola and Jones, 2001). These features are simple rectangular filters
that capture specific patterns in an image, such as edges, lines, and corners.
The classifier works by applying a series of these Haar-like features to sub-
regions of an image and evaluating the response at each stage to determine the
likelihood of a face being present. The Haar classifier consists of a cascade
of weak classifiers, which are trained using a variant of the AdaBoost
algorithm. Evaluated on “Labeled Faces in the Wild” (LFW) dataset (Huang et
al., 2007), the classifier has demonstrated impressive results, achieving
accuracies of over 95% with low false-positive rates.

####  4.5.3. HOG Face Detector

The Histogram of Oriented Gradients (HOG) (Dalal and Triggs, 2005) classifier
is a popular face detection algorithm in computer vision. It captures gradient
orientation distributions in image cells to represent features and uses a
sliding window approach for detection. Compared to Haar classifiers, HOG is
more robust to lighting and pose variations, detects faces of varying sizes,
and offers computational efficiency for real-time applications. It achieved an
accuracy of 95.6% with a false positive rate of 0.13 on the ”Labeled Faces in
the Wild” (LFW) dataset. However, HOG requires ample training data and careful
parameter selection. Despite its limitations, HOG remains widely used and
effective in computer vision.

####  4.5.4. Dlib CNN Face Detector

The Convolutional Neural Network (CNN) Face Detector in Dlib is a deep
learning-based approach for detecting faces in images. Unlike traditional
computer vision techniques that rely on hand-crafted features, CNNs learn to
extract relevant features from the input data, which makes them more effective
in dealing with complex tasks like face detection. It achieved an accuracy of
97.6% on the LFW dataset. Additionally, the model is robust to variations in
lighting, pose, and expression, making it suitable for real-world
applications.

####  4.5.5. OpenCV DNN Face Detector

The OpenCV library includes a deep neural network (DNN) based face detector,
introduced in version 3.3. This detector utilizes the ResNet-10 architecture
as its backbone and employs a single-shot multibox detector approach. It
combines feature extraction using Haar-like features with classification using
a CNN to identify faces in images. The model achieved an impressive accuracy
of 99.60% on the LFW dataset. However, one potential drawback is its higher
computational cost compared to the Haar cascade classifier, making it less
suitable for resource-constrained devices like smartphones or embedded
systems.

####  4.5.6. Fine-tuned YOLOv8 for Face Detection

The previously mentioned face detection models demonstrated excellent
performance for close-range faces. However, their effectiveness notably
declined when confronted with distant faces. In response to this challenge, we
have introduced an alternative face detection model, a fine-tuned version of
YOLOv8 specifically optimized for face detection. We harnessed the renowned
capabilities in object detection of YOLOv8 by tailoring the model to face
detection through fine-tuning, where every layer of the YOLOv8 (nano version)
model was made trainable. This process utilized a dedicated face detection
dataset (Elmenshawii, 2023), which includes distant faces, making it
especially more suitable for drone applications than the LFW dataset, which
contains near faces only. The dataset encompasses approximately 16.7k images,
with 80% allocated for training and 20% for testing. The model underwent
training for 28 epochs, employing a meticulously chosen learning rate of
0.000713, determined through rigorous experimentation to optimize performance.
The test results highlight the model’s performance, achieving a precision of
0.89522, a recall of 0.80549, a mean Average Precision (mAP) of 0.88056, and
an impressive operating speed of 55 frames per second (fps).

In Fig. 10(d), it is evident that our fine-tuned YOLOv8 model excelled in
detecting all faces positioned at a considerable distance from the camera. In
contrast, the Haar face detector model managed to detect only two out of seven
faces in the figure, while the other face detection models struggled to detect
faces at such a significant distance from the camera.

###  4.6. Crowd Counting

Crowd counting involves estimating the number of people in crowded scenes like
urban areas, stadiums, and public events. Crowd counting is essential with
drones for monitoring gatherings, managing crowd flow, and assessing affected
populations for disaster response in case of emergency.

Various approaches exist in crowd counting: detection-based, regression-based,
and density-based regression counting. Detection-based methods involve people
detection before counting, facing challenges in crowded scenes. Regression-
based methods directly estimate the count but can be sensitive to variations
and clutter. Density-based methods analyze overall crowd density, offering
simplicity and efficiency but might lose accuracy in complex scenes. For low
error rates and real-time performance, we adopt a cascaded convolutional
neural network (Sindagi and Patel, 2017). This leverages multi-task learning
for crowd density map estimation and crowd count classification into distinct
groups. Crowd count classification captures scene-level information, while
density estimation focuses on local density, enhancing accuracy. Fig. 11(b)
illustrates an example of the model output.

(a) Input.

(b) Output.

Figure 11.  Input and Output of Crowd counting.

###  4.7. Action Recognition

Action recognition involves the analysis of video sequences to identify and
categorize specific human actions or activities. Action recognition with
drones holds significant importance in various fields due to its potential to
enhance safety and decision-making. By accurately identifying and tracking
human actions or movements from an aerial perspective, drones can play a
crucial role in security and surveillance operations, enabling rapid response
to potential threats or emergencies. Additionally, drones equipped with action
recognition capabilities in sports and entertainment can capture dynamic and
engaging footage, enhancing the viewer experience. For this task, we offer
three different video recognition models that achieve state-of-the-art results
on well-known video recognition datasets. An example of the output of any of
the action recognition models is shown in Fig. 12.

####  4.7.1. Time Space Transformer (TimeSformer)

Instead of applying joint space-time attention, which is time-consuming, a
divided space and time attention is proposed in (Bertasius et al., 2021). In
this approach, each patch in an image is first used to compute temporal
attention, with all patches having the same spatial index. The resulting
encoding is then used for computing spatial attention with patches having the
same temporal index. This approach surpasses the usage of spatial attention
only or joint space-time attention in terms of accuracy. In addition, the
inference cost of this approach is minimal compared to other well-known
approaches relying on 3D convolution, such as SlowFast (Feichtenhofer et al.,
2019).

####  4.7.2. Video Vision Transformer (ViViT)

Four different architectures of video transformers are proposed in (Arnab et
al., 2021) based on four different factorization techniques of spatiotemporal
features. In the first variation (Spatiotemporal attention), the transformer
encoder accepts different spatiotemporal portions of the video, performing
joint space-time attention without factorization. In the second variation
(Factorized encoder), a spatial transformer is applied first for all frames in
a video, and then a temporal transformer deals with the learned encoding of
each spatial transformer. In the third variation (Factorized self-attention),
a similar approach to the first variation is applied but with factorization
into spatial attention and temporal attention. In the final variation
(Factorized dot-product attention), the factorization is applied to the multi-
head dot-product attention operation. The first variation has the highest
inference cost and achieves the best results on the Kinetics-400 dataset (Kay
et al., 2017). On the other hand, the second variation has the minimal
inference cost compared to the other variations, with a slight decrease in
accuracy compared to the first variation.

####  4.7.3. Video Masked Autoencoders (VideoMAE)

A tube masking strategy is proposed in (Tong et al., 2022) to force the
network to learn important features in a video, which is achieved by masking a
large portion of cubes in the video - cube embedding is used to represent one
token - with negligible motion, reducing the computation cost and enhancing
the model performance. The few remaining unmasked tokens are used with the
vanilla Vision Transformer (ViT) to capture the spatiotemporal relation
between them. This approach achieved state-of-the-art results on the
Kinetics-400 dataset (Kay et al., 2017) with 87.4% accuracy, compared to 80.7%
and 84.9% accuracy of TimeSformer and ViViT, respectively. In terms of
inference cost, VideoMAE requires 7397 GFLOPS, while TimeSformer and ViViT
require 8353 and 3981 GFLOPS, respectively (Tong et al., 2022).

Figure 12. Action Recognition output. Source: UCF101 dataset (Soomro et al.,
2012).

###  4.8. Depth Estimation

Monocular depth estimation means deducing the distance or depth information of
objects within a scene from a single 2D image or video frame. This process
facilitates comprehension of the spatial arrangement of the environment and
the relative separations between objects. Crucial for drones in autonomous
navigation, obstacle avoidance, and scene understanding, monocular depth
estimation extracts cues from the drone’s camera feed; the drone gains the
ability to perceive its surroundings, gauge the elevation of obstacles or
landmarks, and make informed decisions to ensure secure and efficient flight.

The method in (Kim et al., 2022) is employed, featuring an encoder with
transformers for a large receptive field and a decoder with skip connections
and selective feature fusion using attention maps. This preserves structural
details and generates a detailed feature map. The approach was evaluated on
KITTI dataset (Geiger et al., 2013), achieving 0.908 $\delta_{1}$,
representing the percentage of pixels with a relative depth error below a
threshold of 1.25. Fig. 13 illustrates an example output, where darker regions
denote nearer objects and brighter regions represent farther objects.

Figure 13. Depth estimation output.

###  4.9. Road Segmentation and Lane Detection

Drones equipped with road segmentation and lane detection capabilities
significantly enhance their surrounding awareness, enabling their application
in diverse scenarios such as search and rescue operations, traffic management,
and surveillance tasks. In this context, we employed YOLOP (Wu et al., 2022),
a panoptic driving perception network designed to simultaneously address
multiple tasks, including object detection, deliverable area segmentation, and
lane detection.

The architecture of YOLOP comprises a shared encoder, leveraging the
CSPDarknet backbone (Wang et al., 2021), which is augmented by Spatial Pyramid
Pooling (SSP) (He et al., 2015) and Feature Pyramid Network (Lin et al., 2017)
modules. These additions facilitate the generation of multi-scale features
with varying semantic levels. The encoder’s output is then directed to a
decoder, which includes three separate heads dedicated to each of the three
tasks. The model is trained end-to-end for the three tasks simultaneously.

The performance evaluation of the YOLOP model was conducted using the BDD100k
dataset (Yu et al., 2018), yielding commendable results across the board.
Notably, the model achieved a road segmentation accuracy of 91.5% mean
Intersection over Union (mIOU) and a lane detection accuracy of 70.50%, with a
remarkably real-time processing speed of 17 frames per second (fps). Figures
14(a), and 14(b) show the output of the driveable area segmentation head and
the lane detection head simultaneously.

(a) Driveable area segmentation output.

(b) Lane detection output

Figure 14. Output of road segmentation and lane detection. Source: Kaggle
KITTI Road Segmentation Dataset available at
https://www.kaggle.com/datasets/sakshaymahna/kittiroadsegmentation.

###  4.10. Our Recommendation

This section presents a comparative analysis of previously discussed models,
offering a recommended model for each task based on a balance between model
performance and speed. Initially, these recommended models are pre-configured
as the default choices within the DroneVis library. However, users retain the
flexibility to tailor these defaults according to their individual
preferences.

Table 1 summarizes the aforementioned tasks and associated models. The table
provides insights into their individual performance metrics, frames per second
(FPS), the platform they are compatible with, and the size of their pre-
trained weights. These evaluations were conducted on a laptop equipped with an
Intel Core 8 CPU. The dfault models for each task in DroneVis are highlighted
in bold. In this context, our attention is directed toward tasks that offer
multiple model options. We establish the criteria guiding our recommendations
for these specific tasks:

  * •

For object detection, we advocate the use of the YOLOv8 model, as it
demonstrates exceptional speed and accuracy, achieving 80 FPS and 53.9 AP.

  * •

In the domain of pose estimation and instance segmentation, YOLOv8 is also our
top recommendation, primarily due to its superior speed compared to MediaPipe.
Additionally, YOLOv8 performs effectively when dealing with multiple
individuals, in contrast to MediaPipe, which is limited to providing pose
estimations and segmentation for a single subject.

  * •

Regarding action recognition, we suggest the adoption of VideoMAE. This choice
is based on the model’s more compact weight size when compared to other action
recognition models. Moreover, VideoMAE stands out with its exceptional
accuracy while maintaining a competitive processing speed.

  * •

For face detection, we suggest the adoption of the fine-tuned YOLOv8 model
because of its reliable performance with distant faces compared to other face
detection models. In addition, it has the highest processing speed.

Table 1. Performance Comparison of Various Models in the Computer Vision Tasks
Implemented in Drone-Vis. Default models are in bold font.

Task | Models | Framework | FPS | Evaluation Score | Weights Size | Quantized | Dataset
---|---|---|---|---|---|---|---
Object Detection | Faster R-CNN | Pytorch | 15 | 42.7 AP | 77 MB |  | COCO
| SSD | Pytorch | 24 | 46.5 AP | 14 MB |  | COCO
| YOLOv5 | Pytorch | 20 | 50.7 AP | 186MB (xlarge) |  | COCO
| YOLOv8 | Pytorch | 80 | 53.9 AP | 50MB (medium) | ✔ | COCO
Object Tracking | YOLOv8 | Pytorch | 53 | 77.8 (MOTA) | 24MB (small) | ✔ | COCO
Instance Segmentation | YOLOv8 | Pytorch | 46 | 44.6 AP | 24MB (small) | ✔ | COCO
| MediaPipe | TF Lite | 21 | 96.21% | 249 KB | ✔ | Selfie Dataset
Pose Estimation | YOLOv8 | Pytorch | 67 | 50.4 AP | 7MB (nano) | ✔ | MPII
| MediaPipe | TF Lite | 30 | 53.8 mAP | 6MB (lite) | ✔ | MPII
Face Detection | MediaPipe | TF Lite | 200 | 99.6% | 225KB | ✔ | LFW
| Haar | OpenCV | 15 | 95% | 900KB |  | LFW
| HOG | Dlib | 35 | 95.5% | 127KB |  | LFW
| DLib CNN | Dlib | 4 | 97.6% | 713KB |  | LFW
| OpenCV DNN | OpenCV | 45 | 99.6% | 5.4MB |  | LFW
| Fine-tuned YOLOv8 | Pytorch | 55 | 85.79% | 36.6 MB | ✔ | Face Detection Dataset
Crowd Counting | Cascaded MTL | Pytorch | 11 | 101 MAE | 10MB |  | UCY
Action Recognition (16 frames) | TimeSformer | Pytorch | 4 | 77.9% | 487MB |  | ActivityNet
| ViViT | Pytorch | 5 | 84.9% | 356MB |  | YouTube-VOS
| VideoMAE | Pytorch | 4 | 87.4% | 346MB |  | UCF101
Depth Estimation | GLPDepth | Pytorch | 4 | 0.908 $\delta_{1}$ | 245MB |  | KITTI
Road Segmentation | YOLOP | Pytorch | 17 | 75 mAP | 36MB (onnx) |  | BDD100k

##  5\. Comparison to Related Software

There is limited prior research related to the application of computer vision
algorithms on Parrot drones. The work in (Rohan et al., 2019) focuses on
implementing person detection and tracking on Parrot drones, achieving a frame
rate of 58 frames per second. Similarly, the Deep Drone project by Han et al.
(Han et al., 2016) also addresses these tasks, employing Faster R-CNN for
detection and the KFC algorithm for tracking at frame rates of 1.6 and 70
frames per second, respectively. However, it’s worth noting that neither of
these previous works provides a library for their research, and their source
code is not publicly accessible (Rohan et al., 2019; Han et al., 2016). This
lack of accessibility restricts the ability of other researchers and
developers to contribute to or build upon their findings.

In contrast to these previous studies, the MMCV library (MMCV Contributors,
2018) serves as a valuable point of reference for comparison. MMCV is designed
for a wide range of computer vision tasks, including image and video
processing, data visualization, various convolutional neural network
architectures, and customized activation functions. Additionally, MMCV offers
support for multiple tasks such as classification, detection, segmentation,
pose estimation, and action recognition. It is implemented on the PyTorch
framework and is compatible with various operating systems, including Windows,
Linux, and MacOS. The main drawbacks of the MMCV library are that it lacks the
presence of user interfaces such as CLI and GUI and requires a good knowledge
of the previously mentioned tasks to select the suitable model among the
variety of models they provide for each task. These drawbacks are not found in
the DroneVis library, as we offer multiple user interfaces to make the library
user-friendly and suitable for any user, regardless of their programming
knowledge, as depicted in figures 2 and 3. Furthermore, we provide a default
model for each task in DroneVis, suitable for users who may lack familiarity
with the evaluation metrics of various computer vision models and might be
unsure about model selection criteria. Additionally, we offer users the
flexibility to change the default model based on their preferences.

Table 2 presents a comprehensive comparison between DroneVis and MMCV
libraries, encompassing the aforementioned aspects and delving into key
factors. These factors include:

  * •

Test Coverage: Test coverage is a metric that measures the proportion of a
codebase covered by automated tests, enhancing code reliability by identifying
untested code paths. Both libraries demonstrate commendable test coverage,
assuring code quality.

  * •

Type Checking: Type checking is implemented in both libraries to maintain code
consistency and prevent errors.

  * •

Issue and Pull Request (PR) Templates: These templates streamline
contributions, simplifying the process for users to report issues or submit
PRs.

  * •

Recent Activity (Measured by Last Commit): Active development is evident, with
both libraries having a last commit as recent as one day ago.

  * •

Approved PRs (Signifying Active Community Engagement): The number of approved
Pull Requests in the past six months showcases active community engagement,
with DroneVis having 36 and MMCV 64 approved PRs.

These factors collectively contribute to the libraries’ reliability,
accessibility, and vitality of their respective communities.

Table 2. Comparison to related software | Drone-Vis | MMCV
---|---|---
Backend | TFLite/Pytorch | Pytorch
User Guide/Tutorials |  ✔/ ✔ |  ✔/ ✔
API Documentation | ✔ | ✔
Pretrained Models | ✔ | ✔
Default Models | ✔ | ✘
CLI/GUI |  ✔ / ✔ |  ✘/✘
Test Coverage | 83% | 69%
Type Checking | ✔ | ✔
Issue PR Template | ✔ | ✔
Last Commit (age) | 1 day | 1 day
Approved PRs (6 month) | 40 | 64

##  6\. Vision and Drone Integration

DroneVis excels in its seamless integration of computer vision models with
drone technology, enabling intelligent and autonomous scene tracking. An
illustrative example is its integration of the YOLOv8 object detection model
with drones for autonomous subject tracking. This approach involves real-time
analysis of video frames captured by the drone’s camera. YOLOv8 then predicts
bounding box coordinates for detected objects, enabling the drone to take
appropriate actions based on object position and size. This integration not
only underscores DroneVis’s adaptability but also opens the door to
incorporating various vision models, offering endless potential for precise
and automated computer vision tasks in real-world applications.

Here is a code snippet, accompanied by its output in Fig. 15, demonstrating
just how effortlessly DroneVis facilitates the integration of drone control
with computer vision models. The process is user-friendly, allowing developers
to concentrate on crafting intelligent drone applications without being
burdened by the technical complications of drone communication and control.

⬇

import cv2

from dronevis.drone_connect import Drone

from dronevis.models import YOLOv8Detection

from dronevis.utils.general import write_on_image

drone = Drone(); drone.connect() # Connect to drone network

model = YOLOv8Detection(track=True)

EPS_HORIZONTAL = 0.1; EPS_VERTICAL = 0.3; EPS_HEIGHT = 0.3

def take_action(results):

boxes = results[0].boxes

x, y, _, h = boxes[0].xywhn.detach().cpu().numpy()[0]

if (abs(x - 0.5) <= EPS_HORIZONTAL and abs(y - 0.5) <= EPS_VERTICAL

and abs(h - 0.5) <= EPS_HEIGHT):

drone.hover(); return "hover"

if x >= 0.5 + EPS_HORIZONTAL:

drone.right(); return "right"

if x <= 0.5 - EPS_HORIZONTAL:

drone.left(); return "left"

if y >= 0.5 + EPS_VERTICAL:

drone.downward(); return "down"

if y <= 0.5 - EPS_VERTICAL:

drone.upward(); return "up"

if h >= 1 + EPS_HEIGHT:

drone.backward(); return "backward"

drone.forward(); return "forward"

def operating_callback(frame, _):

results = model.raw_predict(frame, classes=0, verbose=False)

action = "hover"

if len(results[0].boxes) > 0: action = take_action(results)

frame = write_on_image(results[0].plot(), action)

cv2.imshow("frame", frame); cv2.waitKey(1)

model.load_model()

none_lambda = lambda: None # dummy callback for closing the video

drone.connect_video(none_lambda, operating_callback, "None")

Furthermore, this example highlights the potential for extending DroneVis to
more advanced tasks, such as Simultaneous Localization and Mapping (SLAM). By
integrating additional components and algorithms into DroneVis, developers can
create drones that not only track objects but also navigate complex
environments, map their surroundings, and execute more advanced missions
autonomously. This showcases the versatility and adaptability of DroneVis for
various drone applications and underscores its role as a powerful tool for
pushing the boundaries of drone technology.

(a) Drone steering left.

(b) Drone advancing.

Figure 15. Output of the drone tacking for left and forward cases.

##  7\. Conclusion and Future Work

In this work, we have introduced the DroneVis library, which serves as a
comprehensive solution for automating computer vision algorithms on Parrot
drones. The library boasts a wide range of features, including comprehensive
documentation, a demonstrative testing environment, adherence to coding
standards, and a reliable test coverage percentage. Furthermore, DroneVis
offers diverse user interfaces, encompassing both graphical and command-line
interfaces. Notably, it also enables drone control through innovative methods
such as hand gesture recognition, significantly expanding the range of drone
applications.

The library supports an extensive selection of computer vision tasks, making
Parrot drones suitable for applications ranging from surveillance and security
to rescue and disaster management. Additionally, for each task, users have the
flexibility to choose from a variety of neural network models, tailoring their
approach to their specific preferences.

For future work, our primary objective is to integrate a localization module
into the library. This enhancement will empower drones with autonomous
navigation capabilities, opening up new horizons for their utilization. In
addition an module for ensuring the automo

## References

  * (1)
  * Aharon et al. (2022) Nir Aharon, Roy Orfaig, and Ben-Zion Bobrovsky. 2022.  BoT-SORT: Robust associations multi-pedestrian tracking.  _arXiv preprint arXiv:2206.14651_ (2022).
  * Akbari et al. (2021) Younes Akbari, Noor Almaadeed, Somaya Al-Maadeed, and Omar Elharrouss. 2021.  Applications, databases and open computer vision research from drone videos and images: a survey.  _Artificial Intelligence Review_ 54 (2021), 3887–3938.
  * Arafat et al. (2023) Muhammad Yeasir Arafat, Muhammad Morshed Alam, and Sangman Moh. 2023.  Vision-based navigation techniques for unmanned aerial vehicles: Review and challenges.  _Drones_ 7, 2 (2023), 89.
  * Arnab et al. (2021) Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. 2021.  Vivit: A video vision transformer. In _Proceedings of the IEEE/CVF international conference on computer vision_. 6836–6846.
  * Barbedo (2019) Jayme Garcia Arnal Barbedo. 2019\.  A review on the use of unmanned aerial vehicles and imaging sensors for monitoring and assessing plant stresses.  _Drones_ 3, 2 (2019), 40.
  * Bazarevsky et al. (2020) Valentin Bazarevsky, Ivan Grishchenko, Karthik Raveendran, Tyler Zhu, Fan Zhang, and Matthias Grundmann. 2020.  Blazepose: On-device real-time body pose tracking.  _arXiv preprint arXiv:2006.10204_ (2020).
  * Bazarevsky et al. (2019) Valentin Bazarevsky, Yury Kartynnik, Andrey Vakunov, Karthik Raveendran, and Matthias Grundmann. 2019\.  Blazeface: Sub-millisecond neural face detection on mobile gpus.  _arXiv preprint arXiv:1907.05047_ (2019).
  * Bertasius et al. (2021) Gedas Bertasius, Heng Wang, and Lorenzo Torresani. 2021\.  Is space-time attention all you need for video understanding?. In _ICML_ , Vol. 2. 4.
  * Dalal and Triggs (2005) Navneet Dalal and Bill Triggs. 2005.  Histograms of oriented gradients for human detection. In _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR)_ , Vol. 1. 886–893.
  * Ding and Tao (2016) Changxing Ding and Dacheng Tao. 2016.  A comprehensive survey on pose-invariant face recognition.  _ACM Transactions on intelligent systems and technology (TIST)_ 7, 3 (2016), 1–42.
  * Elmenshawii (2023) Fares Elmenshawii. 2023\.  Face Detection Dataset.  https://www.kaggle.com/datasets/fareselmenshawii/face-detection-dataset/data.
  * Feichtenhofer et al. (2019) Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. 2019.  Slowfast networks for video recognition. In _Proceedings of the IEEE/CVF international conference on computer vision_. 6202–6211.
  * Geiger et al. (2013) Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. 2013.  Vision meets robotics: The kitti dataset.  _The International Journal of Robotics Research_ 32, 11 (2013), 1231–1237.
  * Girshick (2015) Ross Girshick. 2015\.  Fast r-cnn. In _Proceedings of the IEEE international conference on computer vision_. 1440–1448.
  * Girshick et al. (2014) Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014.  Rich feature hierarchies for accurate object detection and semantic segmentation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 580–587.
  * Google (2023) Google. 2023.  MediaPipe.  https://github.com/google/mediapipe.
  * Han et al. (2016) Song Han, William Shen, and Zuozhen Liu. 2016.  Deep drone: Object detection and tracking for smart drones on embedded system.  _URL https://web. stanford. edu/class/cs231a/prev_projects_2016/deepdrone-object__2_. pdf_ (2016).
  * He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015\.  Spatial pyramid pooling in deep convolutional networks for visual recognition.  _IEEE transactions on pattern analysis and machine intelligence_ 37, 9 (2015), 1904–1916.
  * Henriques et al. (2012) Joao F Henriques, Rui Caseiro, Pedro Martins, and Jorge Batista. 2012.  Exploiting the circulant structure of tracking-by-detection with kernels. In _Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part IV 12_. Springer, 702–715.
  * Henriques et al. (2014) João F Henriques, Rui Caseiro, Pedro Martins, and Jorge Batista. 2014.  High-speed tracking with kernelized correlation filters.  _IEEE transactions on pattern analysis and machine intelligence_ 37, 3 (2014), 583–596.
  * Huang et al. (2007) Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller. 2007.  _Labeled faces in the wild: A database for studying face recognition in unconstrained environments_.  Technical Report Technical Report 07-49. University of Massachusetts, Amherst.
  * Jocher (2020) Glenn Jocher. 2020\.  _YOLOv5 by Ultralytics (Version 7.0)_.  https://doi.org/10.5281/zenodo.3908559
  * Jocher et al. (2023) Glenn Jocher, Anand Chaurasia, and Jiarui Qiu. 2023\.  _YOLO by Ultralytics (Version 8.0.0)_.  https://github.com/ultralytics/ultralytics
  * Kay et al. (2017) Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. 2017\.  The kinetics human action video dataset.  _arXiv preprint arXiv:1705.06950_ (2017).
  * Kim et al. (2022) Doyeon Kim, Woonghyun Ka, Pyungwhan Ahn, Donggyu Joo, Sehwan Chun, and Junmo Kim. 2022\.  Global-local path networks for monocular depth estimation with vertical cutdepth.  _arXiv preprint arXiv:2201.07436_ (2022).
  * Krekel et al. (2004) Holger Krekel, Bruno Oliveira, Ronny Pfannschmidt, Floris Bruynooghe, Brianna Laugher, and Florian Bruhin. 2004.  pytest 7.4.  https://github.com/pytest-dev/pytest
  * Lin et al. (2017) Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. 2017\.  Feature pyramid networks for object detection. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2117–2125.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context. In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_. Springer, 740–755.
  * Liu et al. (2016) Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. 2016.  Ssd: Single shot multibox detector. In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14_. Springer, 21–37.
  * Madhavan et al. (2018) Raj Madhavan, Tatiana Silva, Flavia Farina, Rodrigo Wiebbelling, Leonardo Renner, and Edson Prestes. 2018.  Unmanned aerial vehicles for environmental monitoring, ecological conservation, and disaster management. In _Technologies for Development: From Innovation to Social Impact_. Springer International Publishing, 31–39.
  * MMCV Contributors (2018) MMCV Contributors. 2018\.  _OpenMMLab Computer Vision Foundation_.  https://github.com/open-mmlab/mmcv
  * Mohsan et al. (2023) Syed Agha Hassnain Mohsan, Nawaf Qasem Hamood Othman, Yanlong Li, Mohammed H Alsharif, and Muhammad Asghar Khan. 2023.  Unmanned aerial vehicles (UAVs): Practical aspects, applications, open challenges, security issues, and future trends.  _Intelligent Service Robotics_ 16, 1 (2023), 109–137.
  * Parrot (2012) Parrot. 2012.  _AR Drone 2.0_.  http://ardrone2.parrot.com/ Parrot’s official website.
  * Redmon et al. (2016) Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016.  You only look once: Unified, real-time object detection. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 779–788.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015\.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ 28 (2015).
  * Rohan et al. (2019) Ali Rohan, Mohammed Rabah, and Sung-Ho Kim. 2019\.  Convolutional neural network-based real-time object detection and tracking for parrot AR drone 2.  _IEEE access_ 7 (2019), 69575–69584.
  * Sindagi and Patel (2017) Vishwanath A Sindagi and Vishal M Patel. 2017.  Cnn-based cascaded multi-task learning of high-level prior and density estimation for crowd counting. In _2017 14th IEEE international conference on advanced video and signal based surveillance (AVSS)_. IEEE, 1–6.
  * Soomro et al. (2012) Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012\.  UCF101: A dataset of 101 human actions classes from videos in the wild.  _arXiv preprint arXiv:1212.0402_ (2012).
  * Tong et al. (2022) Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. 2022\.  Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  _Advances in neural information processing systems_ 35 (2022), 10078–10093.
  * Uijlings et al. (2013) Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. 2013.  Selective search for object recognition.  _International journal of computer vision_ 104 (2013), 154–171.
  * Viola and Jones (2001) Paul Viola and Michael J. Jones. 2001.  Rapid object detection using a boosted cascade of simple features. In _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR)_ , Vol. 1. I–I.
  * Wang et al. (2021) Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. 2021\.  Scaled-yolov4: Scaling cross stage partial network. In _Proceedings of the IEEE/cvf conference on computer vision and pattern recognition_. 13029–13038.
  * Wu et al. (2022) Dong Wu, Man-Wen Liao, Wei-Tian Zhang, Xing-Gang Wang, Xiang Bai, Wen-Qing Cheng, and Wen-Yu Liu. 2022.  Yolop: You only look once for panoptic driving perception.  _Machine Intelligence Research_ 19, 6 (2022), 550–562.
  * Yao et al. (2020) Rui Yao, Guosheng Lin, Shixiong Xia, Jiaqi Zhao, and Yong Zhou. 2020.  Video object segmentation and tracking: A survey.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ 11, 4 (2020), 1–47.
  * Yu et al. (2018) Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, Trevor Darrell, et al. 2018\.  Bdd100k: A diverse driving video database with scalable annotation tooling.  _arXiv preprint arXiv:1805.04687_ 2, 5 (2018), 6\.
  * Zhang et al. (2020) Fan Zhang, Valentin Bazarevsky, Andrey Vakunov, Andrei Tkachenka, George Sung, Chuo-Ling Chang, and Matthias Grundmann. 2020.  Mediapipe hands: On-device real-time hand tracking.  _arXiv preprint arXiv:2006.10214_ (2020).
  * Zhang et al. (2022) Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang. 2022.  Bytetrack: Multi-object tracking by associating every detection box. In _European Conference on Computer Vision_. Springer, 1–21.