# Adapt-and-Distill: Developing Small, Fast and Effective
Pretrained Language Models for Domains

Yunzhi Yao†, Shaohan Huang‡, Wenhui Wang‡, Li Dong‡, Furu Wei‡
†Shandong University, Jinan, China
‡Microsoft Research, Beijing, China
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Contribution during internship at Microsoft Research.

###### Abstract

Large pretrained models have achieved great success in many natural language
processing tasks. However, when they are applied in specific domains, these
models suffer from domain shift and bring challenges in fine-tuning and online
serving for latency and capacity constraints. In this paper, we present a
general approach to developing small, fast and effective pretrained models for
specific domains. This is achieved by adapting the off-the-shelf general
pretrained models and performing task-agnostic knowledge distillation in
target domains. Specifically, we propose domain-specific vocabulary expansion
in the adaptation stage and employ corpus level occurrence probability to
choose the size of incremental vocabulary automatically. Then we
systematically explore different strategies to compress the large pretrained
models for specific domains. We conduct our experiments in the biomedical and
computer science domain. The experimental results demonstrate that our
approach achieves better performance over the $\text{BERT}_{\text{BASE}}$
model in domain-specific tasks while 3.3× smaller and 5.1× faster than
$\text{BERT}_{\text{BASE}}$. The code and pretrained models are available at
https://aka.ms/adalm.

##  1 Introduction

Pre-trained language models, such as GPT Radford et al. (2018), BERT Devlin et
al. (2019), RoBERTa Liu et al. (2019) and UniLM Dong et al. (2019) have
achieved impressive success in many natural language processing tasks. These
models usually have hundreds of millions of parameters. They are pre-trained
on a large corpus of general domain and fine-tuned on target domain tasks.
However, it is not optimal to deploy these models directly to edge devices in
specific domains. First, heavy model size and high latency makes it difficult
to deploy on resource-limited edge devices such as mobile phone. Second,
directly fine-tuning a general pre-trained model on a domain-specific task may
not be optimal when the target domain varies substantially from the general
domain. Thirdly, many specialized domains contain their own specific terms,
which are not included in pre-trained language model vocabulary.

Figure 1: The four alternatives when distilling BERT into specific domains.
All strategies are task-agnostic.

In this paper, we introduce AdaLM, a framework that aims to develop small,
fast and effective pre-trained language models for specific domains. To
address domain shift problem, recent studies Lee et al. (2020); Gururangan et
al. (2020) conduct continual pre-training to adapt a general domain pre-
trained model to specific domains. However, specific domains contain many
common in-domain terms, which may be divided into bite-sized pieces (e.g.,
lymphoma is tokenized into [l, ##ym, ##ph, ##oma]). Gu et al.(2020) mentions
that domain-specific vocabularies play a vital role in domain adaptation of
pre-trained models. Specifically, we propose a domain-specific vocabulary
expansion in the adaptation stage, which augments in-domain terms or subword
units automatically given in-domain text. Also, it is critical to decide the
size of incremental vocabulary. Motivated by subword regularization Kudo
(2018), AdaLM introduces a corpus occurrence probability as a metric to
optimize the size of incremental vocabulary automatically.

We systematically explore different strategies to compress general BERT models
to specific domains (Figure 1): (a) From scratch: pre-training domain-specific
small model from scratch with domain corpus; (b) Distill-then-Adapt: first
distilling large model into small model, then adapting it into a specific
domain; (c) Adapt-then-Distill: first adapting BERT into a specific domain,
then distilling model into small size; (d) Adapt-and-Distill: adapting both
the large and small models, then distilling with these two models initializing
the teacher and student models respectively.

We conduct experiments in both biomedical and computer science domain and
fine-tune the domain-specific small models on different downstream tasks.
Experiments demonstrate that Adapt-and-Distill achieves state-of-the-art
results for domain-specific tasks. Specifically, the 6-layer model of 384
hidden dimensions outperforms the $\text{BERT}_{\text{BASE}}$ model while 3.3×
smaller and 5.1× faster than $\text{BERT}_{\text{BASE}}$.

##  2 Related Work

#### Domain adaptation of pre-trained model

Most previous work on the domain-adaptation of pre-trained models targets
large models. Lee et al. (2020) conduct continual pre-training to adapt the
BERT model to the biomedical domain using the PubMed abstracts and the PMC
full text. Gururangan et al. (2020) also employ continual pre-training to
adapt pre-trained models into different domains including biomedical, computer
science and news. However, many specialized domains contain their own specific
words that are not included in pre-trained language model vocabulary. Gu et
al.(2020) propose a biomedical pre-trained model PubMedBERT, where the
vocabulary was created from scratch and the model is pre-trained from scratch.
Furthermore, in many specialized domains, large enough corpora may not be
available to support pre-training from scratch. Zhang et al. (2020) and Tai et
al. (2020) extend the open-domain vocabulary with top frequent in-domain words
to resolve this out-of-vocabulary issue. This approach ignores domain-specific
sub-word units (e.g., blasto-, germin- in biomedical domain). These subword
units help generalize domain knowledge and avoid unseen words.

#### Task-agnostic knowledge distillation

In recent years, tremendous progress has been made in model compression Cheng
et al. (2017). Knowledge distillation has proven to be a promising way to
compress large models while maintaining accuracy Sanh et al. (2019); Jiao et
al. (2020); Sun et al. (2020); Wang et al. (2020). In this paper, we focus on
task-agnostic knowledge distillation approaches, where a distilled small pre-
trained model can be directly fine-tuned on downstream tasks. DistilBERT Sanh
et al. (2019) employs the soft label and embedding outputs to supervise the
student. TinyBERT Jiao et al. (2020) and MobileBERT Sun et al. (2020)
introduce self-attention distributions and hidden states to train the student
model. MiniLM Wang et al. (2020) avoids restrictions on the number of student
layers and employs the self-attention distributions and value relation of the
teacher’s last transformer layer to supervise the student model. Because this
method is more flexible, we implement MiniLM to compress large models in this
work. No previous work systematically explores different strategies to achieve
an effective and efficient smaller model in specific domains.

##  3 Methods

###  3.1 Overview

We systematically explore different strategies to achieve an effective and
efficient small model in specific domains. We summarize them into four
strategies: from scratch, distill-then-adapt, adapt-then-distill and adapt-
and-distill.

#### Pretrain-from-scratch

Domain-specific pretraining from scratch employs a random initialization of a
pretrained model and pretrains a small model directly on domain-specific
corpus. In this work, we conduct pretraining from scratch on different
vocabularies including BERT original vocabulary, from scratch vocabulary, and
expanded vocabulary.

#### Distill-then-adapt

These approaches first distill the large general pretrained model which
pretrained on Wikipedia and BookCorpus. Then it continues the pretraining
process using a domain-specific corpus. In this work, we first distill the
BERT model into a small model using task-agnostic knowledge distillation in
MiniLM Wang et al. (2020). Then we initialize the small model with it and
conduct continual training with both the BERT original vocabulary and the
expanded vocabulary.

#### Adapt-then-distill

In this work, we select different large models as teacher models such as BERT
and large models with different vocabularies. We first adapt these models into
domain-specific models and then implement MiniLM to compress them to small
models.

#### Adapt-and-distill

In the previous part, when doing knowledge distill, we initialized the student
model randomly. In order to get a better domain-specific small model, we try
to explore the impact of the initialization of the student model. In this
part, we adapt large and small models into specific domains separately, then
use these two models to initialize the teacher and student model respectively.

###  3.2 Domain Adaptation

AdaLM contains a simple yet effective domain adaptation framework for a
pretrained language model. As shown in Figure 2, it takes a general pretrained
language model, original vocabulary and a domain specific corpus as input.
Through vocabulary expansion and continual pretraining, AdaLM adapts general
models into specific domains.

Figure 2: The pipeline of domain adaptation. Here we adapt the BERT model into
the biomedical domain with PubMed dataset. Figure 3: Concatenate original
embedding with expanded embedding.

The core pipeline of domain adaptation consists of the three steps described
below:

  1. 1.

Given original vocabulary and a domain-specific corpus, the vocabulary
expansion module aims to augment original vocabulary with domain-specific
subword units or terms. We augment domain-specific vocabulary from the target
domain, while keeping the original BERT vocabulary unchanged. We describe them
in more detail in Section 3.3.

  2. 2.

Due to the size of the vocabulary having changed, we cannot initialize our
model with BERT directly. As illustrated in Figure 3, we initialize the
original embedding and Transformer encoder with weights from BERT (the green
part in Figure 3). For incremental vocabulary, we first tokenize them into
sub-words with the original vocabulary and then use an average pooling of
their own sub-words embedding to initialize. As shown in Figure 3, the word
‘lymphoma’ is not included in BERT vocabulary. We tokenize it into three sub-
words (lym, ##pho, ##ma). The embedding vector of ‘lymphoma’ is initialized by
the average embedding vector of ‘lym’, ‘##pho’and ‘##ma’.

  3. 3.

After model initialization and data preprocessing, we continually pretrain our
model with domain-specific corpus using masked language model loss. Following
BERT, we randomly replace 15% of tokens by a special token (e.g., [MASK]) and
ask the language model to predict them in continual pretraining.

###  3.3 Vocabulary Expansion

Vocabulary expansion is the core module of AdaLM. It augments domain-specific
terms or subword units to leverage domain knowledge. The size of the
incremental vocabulary is a vital parameter for vocabulary expansion.
Considering that unigram language modeling Kudo (2018) aligns more closely
with morphology and avoids problems stemming from BPE’s greedy construction
procedure, as proposed in Bostrom and Durrett (2020), we followed Kudo (2018)
and introduced a corpus occurrence probability as a metric to optimize the
size of incremental vocabulary automatically. We assume that each subword
occurs independently and we assign to each subword in the corpus a probability
equal to its frequency in the corpus.

$\displaystyle\forall
i\,\,x_{i}\in\mathcal{V},\,\,\,\sum_{x_{i}\in\mathcal{V}}p(x_{i})=1,$ (1)

where $\mathcal{V}$ is a pre-determined vocabulary. The probability of a
subword sequence $\mathbf{x}=(x_{1},\ldots,x_{M})$ can be computed by the
product of the subword appearance probabilities $p(x_{i})$. We convert it to
logarithmic form:

$\displaystyle P(\mathbf{x})=\sum_{i=1}^{M}log(p(x_{i})),$ (2)

Given a domain-specific corpus $D$, the occurrence probability of corpus $D$
is formulated as:

$\displaystyle P(D)=\sum_{\mathbf{x}}^{|D|}log(P(\mathbf{x})),$ (3)

where $\mathbf{x}$ represents tokenized sentence in corpus $D$.

We sample 550k sentences from the PubMed corpus and compute the occurrence
probability $P(D)$ with different vocabulary sizes. The results are shown in
Figure 4. We compare the occurrence probability with BERT and PubMedBERT
vocabularies. We observe that $P(D)$ reveals a logarithmic trend with
substantial increases at the beginning and little influence after vocabulary
size of 70k in the biomedical domain. The PubMedBERT vocabulary performs
similarly to the 40k size vocabulary. We present the occurrence probability of
different vocabulary sizes in Appendix A.

Figure 4: The $P(D)$ of different vocab sizes under biomedical domain. We use
the BERT’s vocabulary as the 30k vocabulary without vocabulary expanding. The
PubMedBERT vocabulary is also 30k.

We propose a simple method to decide the size of the incremental vocabulary.
Assume the probability at the time step $i-1$ is $P_{i-1}(D)$ and at the time
step $i$ is $P_{i}(D)$. If the rise $\frac{P_{i}(D)-P_{i-1}(D)}{P_{i-1}(D)}$
is lower than a threshold $\delta$, we regard the vocabulary size at the time
step $i$ as the final size.

Input: Original vocabulary $raw\\_vocab$, domain corpora $D$, threshold
$\delta$ and vocabulary size step $V_{\Delta}$

Output: $vocab_{final}$

$token\\_count\leftarrow$ whitespace split from $D$;

$P_{0}\leftarrow$ computed from $raw\\_vocab$;

$V_{0}\leftarrow|raw\\_vocab|$;

do

vocabulary size $V_{i}\leftarrow V_{i-1}+V_{\Delta}$;

$sub\\_count\leftarrow$ split token to subwords;

Sort $sub\\_count$ by frequency;

$incr\\_vocab\leftarrow$keep $(V_{i}-V_{0})$ subwords;

$vocab_{i}\leftarrow$ $raw\\_vocab+incr\\_vocab$;

$P_{i}\leftarrow$ computed from $vocab_{i}$

while  _ $\frac{P_{i}-P_{i-1}}{P_{i-1}}>\delta$ _;

return $vocab_{final}\leftarrow vocab_{i}$ ;

Algorithm 1 Vocabulary Expansion

We expand the domain-specific vocabulary with the process shown in Algorithm
1. We implement our vocabulary expansion algorithm referring to
SubwordTextBuilder in
tensor2tensor111https://github.com/tensorflow/tensor2tensor. In experiments,
we set the threshold $\delta$ as $1\%$ and vocabulary size step $V_{\Delta}$
as 10k. Finally, we obtain the expanded vocabulary size of biomedical as 60k
and computer science domain as 50k.

##  4 Experiment Details

We conduct our experiments in two domains: biomedical and computer science.

###  4.1 Datasets

#### Domain corpus:

For the biomedical domain, we collect a 16GB corpus from
PubMed222https://pubmed.ncbi.nlm.nih.gov/ abstracts to adapt our model. We use
the latest collection and pre-process the corpora with the same process as
PubMedBERT (we omit any abstracts with less than 128 words to reduce noise.).

For the computer science domain, we use the abstracts text from the
arXiv333https://www.kaggle.com/Cornell-University/arxiv Dataset. We select
abstracts in computer science categories, collecting 300M entries for the
corpus.

#### Fine-tuning tasks:

For the biomedical domain, we choose three tasks: named entity recognition
(NER), evidence-based medical information extraction (PICO), and relation
extraction (RE). We perform entity-level F1 in NER task and word-level
macro-F1 in the PICO task. The RE task uses the micro-F1 of positive classes
evaluation. JNLPBA (Collier and Kim, 2004) NER dataset contains 6,892 disease
mentions, which are mapped to 790 unique disease concepts with BIO tagging
Ramshaw and Marcus (1995). EBM PICO (Nye et al., 2018) datasets annotates text
spans with four tags: Participants, Intervention, Comparator and Outcome.
ChemProt Krallinger et al. (2017) dataset consists of five interactions
between chemical and protein entities. We list the statistics of those tasks
in Table 5.

We fine-tune two downstream tasks in the computer science domain. They are
both classification tasks. The ACL-ARC Jurgens et al. (2018) dataset mainly
focuses on analyzing how scientific works frame their contributions through
different types of citations. SCIERC Luan et al. (2018) dataset includes
annotations for scientific entities, their relations, and coreference
clusters. The statistics are available in Table 2.

Dataset | Train | Dev | Test
---|---|---|---
JNLPBA | 46,750 | 4551 | 8,662
EBM PICO | 339,167 | 85321 | 16,364
ChemProt | 18,035 | 11268 | 15,745
Table 1: Biomedical dataset used in our experiment. All selected from BLURB555https://microsoft.github.io/BLURB/ Dataset | Train | Task | Test | Classes
---|---|---|---|---
ACL-ARC | 1,688 | 114 | 139 | 6
SCIERC | 3,219 | 455 | 974 | 7
Table 2: Computer science dataset used in our experiment. We use the same
train, development, and test splits as Gururangan et al. (2020)

###  4.2 Implementation

We use the uncased version of $\text{BERT}_{\text{BASE}}$ ($12$ layers, $768$
hidden size) as the large model and the MiniLM ($6$ layers, $384$ hidden size)
as the small model.

To adapt the large model, we set the batch size at 8192 and the training step
at 30,000. The peak learning rate was set to 6e-4. To adapt the small model,
we set the batch size as 256 and the training step as 200,000. The learning
rate is set to 1e-4. The maximum length of the input sequence was 512 and the
token masking probability was 15% for both the large model and the small
model.

We implement MiniLM to compress large models and follow the setting of MiniLM,
where the batch size was set to 256 and peak learning rate as 4e-4. We set the
training step as 200,000.

For biomedical tasks, we follow the setting of PubMedBERT Gu et al. (2020) to
fine-tune these three tasks. For computer science tasks, we use the same
setting as Gururangan et al. (2020). The concrete parameters are shown in
Appendix B.

##  5 Results

Config | Type | Model | Teacher | JNLPBA | PICO | Chemprot | Average
---|---|---|---|---|---|---|---
$L$=12; $d$=786 | Large model | BERT† | - | 78.63 | 72.34 | 71.86 | 74.28
BioBERT† | - | 79.35 | 73.18 | 76.14 | 76.22
PubMedBERT† | - | 80.06 | 73.38 | 77.24 | 76.89
AdaLM♢ | - | 79.46 | 75.47 | 78.41 | 77.74
$L$=6; $d$=384 | Small model | MiniLM | - | 77.44 | 71.69 | 68.08 | 72.40
From scratch | BERT vocab (a) | - | 77.89 | 72.97 | 70.21 | 73.69
PubMed vocab (b) | - | 77.82 | 73.82 | 70.32 | 73.99
AdaLM vocab (c) | - | 77.80 | 73.39 | 70.86 | 74.02
Distill-then-Adapt | BERT vocab (d) | - | 78.63 | 74.00 | 72.28 | 74.97
PubMed vocab (e) | - | 78.36 | 73.91 | 71.33 | 74.53
AdaLM vocab (f) | - | 78.77 | 74.23 | 72.29 | 75.10
Adapt-then-Distill | Random initial (g) | BERT | 77.98 | 72.38 | 68.86 | 73.07
Random initial (h) | PubMedBERT | 78.78 | 74.20 | 70.89 | 74.62
Random initial (i) | AdaLM♢ | 78.98 | 74.78 | 71.51 | 75.09
Adapt-and-Distill | Model (f) initial (j) | AdaLM♢ | 79.04 | 74.91 | 72.06 | 75.34

Table 3: Comparison between different strategies on biomedical tasks. The
AdaLM♢ means we just adapt the large model without distillation. Scores of the
methods marked with † are taken from Gu et al. (2020). Underlined data marks
the small models whose performances surpass the BERT model’s performance. $L$
and $d$ indicate the number of layers and the hidden dimension of the model.

Config | Type | Model | Teacher | ACL-ARC | SCIERC | Average
---|---|---|---|---|---|---
$L$=12; $d$=786 | Large model | BERT | - | 64.92 | 81.14 | 73.03
AdaLM♢ | - | 73.61 | 81.91 | 77.76
$L$=6; $d$=384 | Small model | MiniLM | - | 61.5 | 72.88 | 67.19
From scratch | BERT vocab (a) | - | 62.48 | 74.93 | 68.70
AdaLM vocab (b) | - | 59.57 | 74.93 | 67.25
Distill-then-Adapt | BERT vocab (c) | - | 65.75 | 79.13 | 72.44
AdaLM vocab (d) | - | 65.93 | 79.88 | 72.91
Adapt-then-Distill | Random initial (e) | BERT | 63.12 | 77.89 | 70.50
Random initial (f) | AdaLM♢ | 66.21 | 77.04 | 71.62
Adapt-and-Distill | Model (d) initial (g) | AdaLM♢ | 68.74 | 78.88 | 73.81

Table 4: Comparison between different strategies on computer science tasks.
The AdaLM♢ is the adapted large model without compressing. We report averages
across five random seeds. Data marked with underlines are the results of small
models which outperform the BERT model’s. $L$ and $d$ indicate the number of
layers and the hidden dimension of the model.

The results of the tasks are shown in the Table 3 and 4. We structure our
evaluation by stepping through each of our three findings:

(1) Domain-specific vocabulary plays a significant role in domain-specific
tasks and expanding vocabulary with the general vocabulary is better than just
using domain-specific vocabulary.

We observe improved results via the expanded vocabulary with both the large
and small models. For large model, AdaLM achieves the best results under each
domain, where 77.74 on biomedical domain tasks, beating BioBERT and PubMedBERT
and 77.76 on the computer science domain tasks.

For small models, in the biomedical domain, whether we train from scratch or
distill-then-adapt with small models, incremental vocabulary models always
perform better than the general vocabulary or just the domain-specific
vocabulary. (When distill-then-adapt with the PubMed vocabulary, we initialize
the word embedding in the same way as mentioned in Section 3.2). In addition,
with distill-then-adapt, the model (f) (75.10) can surpass the BERT model
(74.28).

In the computer science domain, distill-then-adapt models with incremental
vocabulary also show great performance. Model (d) achieves a comparable result
of 72.91 as BERT and outperforms BERT in the ACL-ARC datasets with 65.93
(+1.01 F1). We also observe that when training from scratch, the results of
Model (b) with incremental vocabulary are lower (1.45 lower) than that of
model (a). This may be because after vocabulary expansion, a from-scratch
model needs to be pretrained with more unlabeled data.

(2) Continual pretraining on domain-specific texts from general language
models is better than pretraining from scratch.

Gu et al. (2020) finds that for domains with abundant unlabeled texts,
pretraining language models from scratch outperforms continual pretraining of
general-domain language models. However, in our experiments, we find that
general-domains model can help our model to learn the target domain better. In
the biomedical domain, we use MiniLM model to initialize the model (d), (e)
and (f) in distill-then-adapt setting. No matter which vocabulary is used,
continual pretraining on domain-specific texts from general language models is
better than pretraining from scratch. For AdaLM vocabulary, the model (f) gets
75.10, outperforming the model (c) trained from scratch with the same
vocabulary by 1.08. On the other hand, for domains that do not have enormous
unlabeled texts such as the computer science domain in our experiments,
continual pretraining also showed better results. With continual pretraining,
model (d) achieves higher results exceeding both model (b) (+5.66 F1) and
model (c) (+0.47 F1).

(3) Adapt-and-Distill is the best strategy to develop a task-agnostic domain-
specific small pretrained model.

In the Adapt-then-Distill part, our findings supports evidence from previous
observations (Wang et al., 2020) that a better teacher model leads to a better
student model. Using AdaLM which performs best among large models as the
teacher model can yield good results: 75.09 in the biomedical domain and 71.62
in the computer science, better than other domain-specific large models.
Furthermore, we find that a better student model for initialization can also
help to get a better small model. In the Adapt-and-Distill part, we adapt
large and small models into specific domains separately and then compress the
adapted large model as the teacher with the adapted small model as
initialization. In the biomedical domain, the model (j), initialized from
model (i), achieves the best result of 75.34 among the small models. It also
outperforms the BERT model (+1.06 F1). In the computer science domain, model
(g), initialized by model (d), is the only small model that outperforms BERT
(+0.78 F1).

##  6 Analysis

###  6.1 Inference Speed

We compare AdaLM’s parameters’ size and inference speed with the BERT model in
the biomedical domain in Table 5.

Type | Model | #Params | Speedup
---|---|---|---
Large | BERT | 109M | $\times 1.0$
PubMedBERT | 109M | $\times 1.0$
AdaLM vocab | 132M | $\times 1.07$
Small | BERT vocab | 22M | $\times 5.0$
AdaLM | 34M | $\times 5.1$

Table 5: Comparison of model’s parameter size and the inference speed. The
inference speedup is computed by the classification task ChemProt and
evaluated on a single NVIDIA P100 GPU.

First we can find that the vocabulary expansion yields marginal improvements
on the model’s inference speed. We added about 20M parameters in the embedding
weights in the large model using AdaLM vocabulary, but its inference speed is
slightly faster than BERT and PubMedBERT. Since most domain-specific terms are
shattered into fragmented subwords, the length of the token sequence we get by
using the incremental vocabulary is shorter than the length of the sequence
got by the original vocabulary, which reduces the computation load. We list
the change of the sequence length of the downstream tasks in Appendix C.
Meanwhile, in the embedding layers, the model just needs to map the sub-words’
id to their dense representations, which is little affected by the parameters’
size. The small model shows the same trend.

In addition, the small model AdaLM shows great potential. Compared with the
12-layer model of 768 hidden dimensions, the 6-layer model of 384 hidden
dimensions is 3.3x smaller and 5.1x faster in the model efficiency, while
performing similarly to or even better than $\text{BERT}_{\text{BASE}}$.

###  6.2 Impact of Training Time

Pre-training often demands lots of time. In this section, we examine the
adapted model’s performance as a function of training time. Here we use the
biomedical domain since its unlabelled texts are abundant and compare the
large domain-specific adapted model with BioBERT. For every 24 hrs of
continual pre-training, we fine-tuned the adapted model on the downstream
tasks. For comparison, we convert the training time of BioBERT to the time it
may take with the same computing resource of this work (16 V100 GPUs).

Model | Training Time | Average
---|---|---
AdaLM | 0 hrs | 74.25
24 hrs | 76.80
48 hrs | 77.36
72 hrs | 77.74
BERT | 0 hrs | 74.28
BioBERT | 120 hrs | 76.22
Table 6: Results with different pre-training time. In the table, AdaLM is the
adapted large model without compressing.

We list the results in Table 6, we denote the large adapted model as AdaLM in
the table. AdaLM at 0 hrs means that we fine-tune the initialized model
directly without any continual pre-training. We find that BERT is slightly
better than 0hr AdaLM and after 24 hrs, AdaLM outperforms BioBERT, which
demonstrates that domain-specific vocabulary is very critical for domain
adaption of pre-trained model. Our experiments demonstrate promising results
in the biomedical domain. Under constrained computation, AdaLM achieves better
performance compared to BioBERT. More details can be found in Appendix D

###  6.3 Impact of Vocabulary Size

To understand the impact of the vocabulary size, we conduct some experiments
with different vocabulary sizes in the biomedical domain. We select the
biomedical large AdaLM model and to reduce the computation load, we set the
batch size as 256 and step as 250K in our ablation studies. We show
performance of the model with different sizes in Table 7.

| 40k | 50k | 60k | 70k | 80k
---|---|---|---|---|---
JNLPBA | 78.84 | 79.02 | 78.91 | 78.94 | 79.01
PICO | 75.09 | 74.81 | 74.99 | 74.58 | 75.00
ChemProt | 76.10 | 76.80 | 77.21 | 76.40 | 76.85
Average | 76.67 | 76.87 | 77.03 | 76.64 | 76.95

Table 7: The performance of different vocabulary sizes

We observe that the model of 60k achieves the best results in our ablation
studies. The result is a bit surprising. Despite having a larger vocabulary,
the 70k and 80k model does not show a stronger performance. A possible
explanation for these results may be that a larger vocabulary set may contain
some more complicated but less frequent words, which cannot be learnt well
through continual pre-training. For example, the word _ferrocytochrome_ exists
in 70k and 80k vocabularies but is split into (_‘ferrocy’, ‘##tochrom’,
‘##e’_) in the 60k vocabulary. In our sampled data (about 550k sentences), ‘
_ferrocytochrome_ ’ appears less than 100 times, while the subword ‘
_##tochrom_ ’ appears more than 10k times and ‘ _ferrocy_ ’ appears more than
200 times. The representation of those rare words cannot be learnt well due to
the sparsity problem.

###  6.4 Vocabulary Visualization

The main motivation for using an the expanded vocabulary set is to leverage
domain knowledge better. Compared to PubMedBERT which just uses the domain-
specific vocabulary and initializes the model randomly, the keep of the
general vocabulary and the general language model’s weights may help us make
good use of the existing knowledge and word embedding.

To assess the importance of the expanded vocabulary, we compute the
$L2$-distance of the embedding weights before and after pre-training in our
AdaLM model in the biomedical domain in Figure 5.

Figure 5: The $L2$-distance of the embedding layer. The deeper the color, the
farther the distance.

We observe that the domain-specific vocabulary part changes a lot during the
pre-training time, which indicates that our model learns much information
about these domain-specific terms. We also observe that there is little change
in many original sub-words’ embedding weights, which indicates that many
general vocabularies can be used directly in continual training.

##  7 Conclusion

In this paper, we investigate several variations to compress general BERT
models to specific domains. Our experiments reveal that the best strategy to
obtain a task-agnostic domain-specific pretrained model is to adapt large and
small models into specific domains separately and then compress the adapted
large model with the adapted small model as initialization. We show that the
adapted 6-layer model of 384 hidden dimensions outperforms the
$\text{BERT}_{\text{BASE}}$ model while 3.3× smaller and 5.0× faster than
$\text{BERT}_{\text{BASE}}$. Our findings suggest that domain-specific
vocabulary and general-domain language model play vital roles in domain
adaptation of a pretrained model. In the future, we will investigate more
directions in domain adaptation, such as data selection and efficient
adaptation.

## References

  * Bostrom and Durrett (2020) Kaj Bostrom and Greg Durrett. 2020.  Byte pair encoding is suboptimal for language model pretraining.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4617–4624, Online. Association for Computational Linguistics.
  * Cheng et al. (2017) Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. 2017.  A survey of model compression and acceleration for deep neural networks.  _arXiv preprint arXiv:1710.09282_.
  * Collier and Kim (2004) Nigel Collier and Jin-Dong Kim. 2004.  Introduction to the bio-entity recognition task at JNLPBA.  In _Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA/BioNLP)_ , pages 73–78, Geneva, Switzerland. COLING.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.  Unified language model pre-training for natural language understanding and generation.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 13042–13054.
  * Gu et al. (2020) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020.  Domain-specific language model pretraining for biomedical natural language processing.  _arXiv preprint arXiv:2007.15779_.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8342–8360, Online. Association for Computational Linguistics.
  * Jiao et al. (2020) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020.  TinyBERT: Distilling BERT for natural language understanding.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4163–4174, Online. Association for Computational Linguistics.
  * Jurgens et al. (2018) David Jurgens, Srijan Kumar, Raine Hoover, Dan McFarland, and Dan Jurafsky. 2018\.  Measuring the evolution of a scientific field through citation frames.  _Transactions of the Association for Computational Linguistics_ , 6:391–406.
  * Krallinger et al. (2017) Martin Krallinger, Obdulia Rabal, Saber A Akhondi, Martın Pérez Pérez, Jesús Santamaría, GP Rodríguez, et al. 2017.  Overview of the biocreative vi chemical-protein interaction track.  In _Proceedings of the sixth BioCreative challenge evaluation workshop_ , volume 1, pages 141–146.
  * Kudo (2018) Taku Kudo. 2018.  Subword regularization: Improving neural network translation models with multiple subword candidates.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 66–75, Melbourne, Australia. Association for Computational Linguistics.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Luan et al. (2018) Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018.  Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3219–3232, Brussels, Belgium. Association for Computational Linguistics.
  * Nye et al. (2018) Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain Marshall, Ani Nenkova, and Byron Wallace. 2018.  A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 197–207, Melbourne, Australia. Association for Computational Linguistics.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.
  * Ramshaw and Marcus (1995) Lance Ramshaw and Mitch Marcus. 1995.  Text chunking using transformation-based learning.  In _Third Workshop on Very Large Corpora_.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Sun et al. (2020) Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020\.  MobileBERT: a compact task-agnostic BERT for resource-limited devices.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2158–2170, Online. Association for Computational Linguistics.
  * Tai et al. (2020) Wen Tai, HT Kung, Xin Luna Dong, Marcus Comiter, and Chang-Fu Kuo. 2020.  exbert: Extending pre-trained models with domain-specific vocabulary under constrained training resources.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , pages 1433–1439.
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Zhang et al. (2020) Rong Zhang, Revanth Gangi Reddy, Md Arafat Sultan, Vittorio Castelli, Anthony Ferritto, Radu Florian, Efsun Sarioglu Kayi, Salim Roukos, Avi Sil, and Todd Ward. 2020.  Multi-stage pre-training for low-resource domain adaptation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5461–5468, Online. Association for Computational Linguistics.

##  Appendix A Occurrence probability of different vocabulary sizes

Vocabulary | P(D)
---|---
BERT | -255.92
PubMed | -218.49
40k vocab | -220.06
50k vocab | -214.40
60k vocab | -211.88
70k vocab | -210.44
80k vocab | -209.57
90k vocab | -208.86
100k vocab | -208.42
Table 8: The $P(D)$ of different vocabulary under biomedical domain. Vocabulary | P(D)
---|---
BERT | -211.14
40k vocab | -194.08
50k vocab | -192.56
60k vocab | -191.87
70k vocab | -191.45
80k vocab | -191.09
90k vocab | -190.76
100k vocab | -190.53
Table 9: The $P(D)$ of different vocabulary under computer science domain.

##  Appendix B Fine-tuning hyperparameters for downstream tasks

Hyperameter | Assignment
---|---
| NER | PICO | RE
Batch size | 32 | {16,32} | 32
Learning rate | {1e-5,3e-5,5e-5}
Epoch | {30-40} | {10,15} | {40-50}
Dropout | 0.1
Table 10: Hyparameters we used to finetune on biomedical tasks. Hyperameter | Assignment
---|---
| ACL-ARC | SCIERC
Batch size | 16
Learning rate | 2e-5
Epoch | 20
Dropout | 0.1
Table 11: Hyparameters we used to finetune on computer science tasks.

##  Appendix C Sequence Length

After the vocabulary expansion, the length of the token sequence may get
shorter. We compute the average sentence length of the downstream tasks. We
list the results in Table 12

Dataset | Original Vocab | Incr. Vocab
---|---|---
ChemProt | 66 | 53
EBM PICO | 36 | 31
JNLPBA | 41 | 32
ACL-ARC | 53 | 50
SCIERC | 45 | 42
Table 12: The sequence length tokenized by the original vocabulary and
expanded vocabulary.

##  Appendix D Results of different training time

We list the biomedical tasks’ results of each pretraining time in the
following table.

| 0h | 24h | 48h | 72h
---|---|---|---|---
JNLPBA | 77.56 | 79.14 | 79.11 | 79.46
PICO | 73.29 | 74.22 | 75.28 | 75.36
ChemProt | 71.91 | 77.06 | 77.69 | 78.42
Average | 74.25 | 76.80 | 77.36 | 77.74
Table 13: The performance of models with different pretraining time


# Insights to the Forensic Investigation of a Custom Built UAV

Tanay Kumar 111Research Associate, Department of Aerospace Engineering;
<EMAIL_ADDRESS>Indian Institute of Technology Kanpur, Kanpur, UP, India,
208016  Mangal Kothari222Professor, Department of Aerospace Engineering;
<EMAIL_ADDRESS>Member AIAA. Indian Institute of Technology Kanpur, Kanpur,
UP, India, 208016

###### Abstract

Unmanned Aerial Vehicles (UAVs) have revolutionized numerous application
fields since their introduction. Alongside this, advancements in mechanics and
electronics have simplified and decreased the cost of system design. As a
consequence, UAVs have dominated the global market across all sectors from
recreational products to military applications. However, the widespread use of
UAVs has also contributed to an increase in criminal activity involving them.
UAVs have become a common means of delivering narcotics, arms, and ammunition,
collecting sensitive personal information, and snooping in restricted areas,
among other activities. This could constitute a threat to national security.
Despite its growing significance, UAV forensics is still a relatively
unexplored field of study. In this paper, we present new insights on UAV
forensic analysis in terms of preliminary analysis, accessing the digital
containers of the UAV, and retrieving vital data. With the aid of a simulated
scenario, a comprehensive methodology for the analysis of retrieved data to
establish conclusions on the flight path, source of origin, recover flight
data, and acquire media file content is presented.

##  1 Introduction

###  1.1 Drones in the modern world

Unmanned Aerial Vehicles (UAVs) have grown in popularity in recent years and
have become an integral part of numerous industries, such as agriculture,
media, recreation, disaster management, delivery, the military, etc[1, 2, 3].
In parallel with this, advancements in mechanics and electronics have
simplified and decreased the cost of system design. UAVs have revolutionized
the collection of data and execution of duties in the air. They have the
unique ability to deploy from anywhere and to transport a variety of payloads.
Due to these benefits, they have become a popular area of research and
consequently, tech giants such as Amazon, Google, Intel, and Boeing are
investing heavily in research and development.

Nonetheless, the proliferation of UAVs has posed new difficulties for law
enforcement and forensic investigators. UAVs have been used to commit offenses
such as contraband smuggling, illegal surveillance, and commercial flight
interference[4]. In the recent past, drones have also been captured
inadvertently violating no-fly zones, and there is a growing fear that
terrorists could use them to cause panic or other damage. Therefore, an
effective and expedient method of investigating UAV-related incidents is
required. The potential misuse of UAVs in criminal and terror-related
activities has intrigued the interest of forensic analysts. Due to their
vertical takeoff-and-landing (VTOL) and hover capabilities, multirotor UAVs
have become popular for malicious purposes. Small and nimble, they are ideal
for conducting surveillance and reconnaissance in secret. They can also
contain payloads such as cameras or small packages, allowing them to transport
contraband or illegal goods across borders or into restricted areas.
Furthermore, these UAVs are widely available, relatively affordable, and
simple to operate, making them accessible to individuals with malicious
intent. It is important to note, however, that any form of UAV can be used for
illegal activities if operated improperly, and the vast majority of UAV
operators use them for legal and beneficial purposes.

###  1.2 Motivation and aim

Due to the threats posed by the malicious use of UAVs, UAV forensic programs
are essential for advancing our understanding of drone technology. In
addition, UAV forensics can be utilized for crash analysis, which is prevalent
in the research and development industry. There is only little work in this
area; therefore, UAV forensics is an underexplored discipline. In order to
address the lack of a standardized drone-based collection and analysis model
in the field of forensics, [5] highlights the extant forensic models proposed
for drone forensics and proposes a novel comprehensive collection and analysis
forensic model applicable to drone forensics. A generic drone forensic model
that improves the digital investigation procedure for general UAV forensics
applicable to a broader spectrum of UAVs has been proposed in [6]. In
addition, the paper describes the forensic analysis model of the various
components, including a camera and Wi-Fi. In [7], the authors provide new
insights into drone forensics in terms of accessing the digital contents of an
intercepted drone and retrieving all the data that can assist digital forensic
investigators in establishing ownership, recovering flight data, and acquiring
media file content. The results of the examination of the exposed FTP channel
and data extraction from the memory chip of the DJI Mavic Air 2 Drone are
discussed in [8]. In the case study [9], the collection, examination,
correction, and analysis of vital data from a DJI Spark drone’s recorded
flight data are described. The investigation of multiple drones produced by
industry-leading manufacturers such as DJI, Bebop, and Syma is done in [10].
Extracted are forensically pertinent data such as location, captured images
and videos, UAV’s flight paths, and ownership information. In addition, the
paper establishes results to aid law informants in this domain. A study on the
identification of what data can be extracted from UAV devices is examined in
[11]. Further, the utility of the extracted data and whether consumers can
obfuscate the data in an effort to evade detection using data collected from
UAV flight testing have also been discussed.

Although UAV forensics has attracted the attention of researchers and forensic
analysts in recent years, the vast majority of studies are conducted using
commercially available drones such as DJI or Parrot. Using a serial number,
commercial UAVs are registered with the company and frequently with the
government as well. Such UAVs may also include a capability for storing flight
history in the cloud. These features facilitate the tracking of the UAV’s
proprietor and the retrieval of any other data stored on the UAV. Moreover,
such UAVs are costly because of the advanced technology, quality components,
and manufacturing costs. Due to these factors and the abundance of data
available online for the design and manufacture of UAVs, drone operators have
begun to employ custom-made UAVs for malicious purposes. The use of custom
technologies and components enables the operator to customize the UAV,
particularly its software, in a way that may not be possible with commercial
UAVs. This makes identification and tracking of the UAV difficult.
Consequently, it is necessary to examine the scope of UAV forensics in such
systems. This research paper provides a comprehensive methodology of UAV
forensics for collecting and analyzing digital evidence from UAVs with custom-
built components. The study examines a custom-made UAV, its components, and
the methodologies for data extraction. In addition, the paper discusses the
processing and analysis of extracted data in order to draw conclusions. The
findings contribute to the development of effective investigative techniques
for UAV-related incidents and provide valuable insights into the field of UAV
forensics.

The remainder of the is structured as follows: Section 2 provides an overview
of UAVs, their classification, design and development, and payload details.
Section 3 establishes the groundwork for the problem statement. This section
discusses the current threats posed by UAVs, the significance of UAV
forensics, the evidence and data available to combat these threats, and the
difficulties associated with UAV forensics. Using a simulated scenario,
Section 3 presents a methodological approach to the UAV forensics problem. The
latter portions of the section discuss the data retrieval processes, the
analysis, and the obtained results. In Section 5, concluding remarks are
presented.

##  2 Unmanned Aerial Vehicles

###  2.1 Classification of UAVs

It is imperative to classify UAVs due to multiple reasons. Classification
helps to standardise the various types of available UAVs, making it easier to
comprehend their capabilities, specifications, and applications. It enables
operators to select the most appropriate UAV for a particular application
based on size, range, payload capacity, and endurance, ensuring that the UAV
is optimized for the task at hand. Classification also aids regulators in the
development and implementation of safety and operational standards for various
UAV categories. UAVs can be categorized based on the parameters listed below.

  1. 1.

Size: UAVs can be classified as micro, small, medium, and large based on their
size. Micro UAVs typically weigh just a few grams, whereas small UAVs can
weigh up to 25 kg. Medium UAVs can weigh up to 250 kg, and large UAVs can
weigh over 500 kg. Each size classification of UAVs has its unique features,
capabilities, and applications. Micro and small UAVs are ideal for indoor use
and aerial photography, while medium and large UAVs are more suitable for
long-range surveillance and military operations.

  2. 2.

Range: UAVs can also be classified based on their range. Short-range UAVs can
fly up to 50 km from their launch point, while medium-range UAVs can travel up
to 150 km. Long-range UAVs can travel over 150 km from their launch point, and
some have an endurance of over 36 hours. The range of UAVs depends on factors
like size, propulsion systems, and fuel capacity.

  3. 3.

Application: UAVs have numerous uses, including military, civilian, and
commercial applications. The main tasks of military UAVs are reconnaissance,
surveillance, and combat missions. UAVs are also utilized for search and
rescue operations, disaster management, and environmental monitoring in the
civilian sector. Commercial UAVs are used in industries such as agriculture,
construction, and surveying. UAVs utilized for aerial photography and
videography are also gaining popularity. Each UAV application demands unique
characteristics, including payload capacity, range, and endurance.

  4. 4.

Configuration: There are two primary configurations of UAVs: fixed-wing and
rotary-wing. Fixed-wing UAVs are built similarly to conventional aircraft.
They are typically employed for long-range missions and are can fly at high
altitudes and velocities for extended periods of time. Alternatively, rotary-
wing UAVs are capable of vertical takeoff and landing. They are typically
employed for shorter-range missions and have hover capability. These two
configuration types also include subcategories depending on the number of
rotors, the design of the wing, etc. In addition to this, hybrid UAVs, which
incorporate characteristics of fixed-wing and rotary-wing UAVs, are gaining
popularity due to their adaptability and ability to take off vertically and
fly forward like fixed-wing UAVs. Further, flapping-wing UAVs are also gaining
popularity. However, they have not matured enough for commercial use.

###  2.2 Components of UAVs

The UAV components depend solely on their configuration, size, and mission.
However, as stated previously, quadrotor UAVs are the most common, so we will
focus primarily on quadrotor UAV components. Each element on any UAV can be
identified as one of the following despite the fact that not every UAV will
contain all of the enumerated components.

  1. 1.

Frame and chassis: The frame is typically made of lightweight materials such
as carbon fiber and acts as the core fuselage to house all the other
components.

  2. 2.

Motors and propellers: A quadrotor has a set of four motors and propellers to
provide the lift and propulsion for the UAV. The diagonally opposite rotors
always spin in the same direction to control the pitch, roll, and yaw motion.
Long range and/or high endurance multi-rotor can have more than four rotors.
Among all multi-rotors, quadcopters, hexacopter, and octarotor are very
popular.

  3. 3.

Electronic Speed Controllers (ESCs): The ESCs control the speed of the motors
and the propellers. ESCs generate the desired thrust by driving motors to the
specified speed using a specialized combination of hardware and firmware.

  4. 4.

Flight controller: The flight controller is the brain of the UAV and is
responsible for the control and stability of the UAV. It uses sensor data
(gyroscopes, accelerometers, and magnetometers) combined with GPS for
autonomous navigation. In addition to this, the vehicle can be controlled
remotely by radio control.

  5. 5.

GPS receiver: This component is required for autonomous flight in order to
manage UAV position and return to home functionality. It also helps to
autonomously pilot the UAV beyond the visual line of sight (BVLOS).

  6. 6.

Radio receiver: Used to receive control input signals sent by the pilot using
a ground based transmitter

  7. 7.

Transmitter: Transmits manual input from the operator on the ground to the
UAV.

  8. 8.

Battery: The battery provides power to the motors, the flight controller, and
other electronic components of the UAV.

  9. 9.

Telemetry: The telemetry module permits bidirectional communication between
the UAV and the ground station. All of the UAV’s real-time data can be
transmitted to the ground station for monitoring, allowing the operator to act
accordingly and send commands back to the UAV.

  10. 10.

Ground Control Software (GCS): The GCS allows the operator to remotely control
and monitor the UAV, including its position, attitude, and speed. It also
provides information about the UAV’s vital status, such as battery and sensor
data, enabling efficient mission planning and execution. This software is used
to control pre-determined navigation and effectively plan flight missions.

Figure 1: Components of a typical multirotor UAV[12]

###  2.3 UAV Payload

UAVs are versatile platforms that can be equipped with various types of
payloads depending on the intended application. Some common types of payloads
are listed below.

  1. 1.

Imaging and sensing payload: UAVs can be equipped with various types of
cameras and sensors, such as RGB cameras, thermal cameras, LiDAR sensors, and
multispectral sensors. These payloads are used for aerial photography,
surveying, inspection, mapping, geo-tagging, and remote sensing.

  2. 2.

Cargo: In recent years, the use of UAVs to provide timely and efficient
deliveries has become an area of increasing investment, with Amazon’s Prime
Air service garnering the most attention. While the primary emphasis is on
commercial and retail deliveries, this technology could also benefit other
industries such as healthcare.

  3. 3.

Scientific instruments: UAVs can be used to carry scientific equipment like
atmospheric and biological sensors for research and monitoring applications.

  4. 4.

Weapons: UAVs may be equipped with combative weapons for transportation or to
conduct attacks at the desired target location.

  5. 5.

Electronics and communication payload: Communications payloads are not yet
prevalent, but they have attracted considerable interest in recent years. UAVs
can be used to carry jammers, signal spoofers, and other signal intelligence
systems.

##  3 Problem Overview

###  3.1 Evidence Sources and Data Retrieved

Various forms of data and evidence can be retrieved from the UAV. Proper
analysis of the evidence can help track its source. Some of the sources of
evidence in drone forensics include:

  1. 1.

Video and image data: Reconnaissance and spying drones always carry cameras.
In most instances, captured images and videos are stored onboard and can serve
as a primary data source. This data can be analyzed to provide evidence of the
flight path of the drone, its location, and any nearby objects or persons.

  2. 2.

Other payload content and storage devices: Secondary storage devices may be
present on the UAV to record huge quantities of sensor data. This data is
frequently geo-tagged and contains other crucial data, when analyzed, can
yield vital information.

  3. 3.

Cloud-based data platforms: This type of storage is uncommon at present, but
it is gaining popularity as 5G technology advances. The user could use cloud
data to reduce the need for local storage.

  4. 4.

Flight logs: Majority of the drones are equipped with flight loggers that
record all information about the drone’s flight. These flight records can be
used to reconstruct the drone’s flight path and identify its source location.

  5. 5.

GPS data: The GPS receivers used in drones often have a memory of their own to
record the real-time GPS data. The logging feature allows position fixes and
arbitrary byte strings from the host to be logged in flash memory attached to
the receiver. This is incredibly useful when data logging is manually disabled
on the flight controller of a custom drone. The data can be utilized to
analyze the flight path of the vehicle.

  6. 6.

Network packet data: The UAV wirelessly communicates with the ground station
using a communication protocol (Mavlink is popular). The data packets can be
intercepted and decrypted to obtain information about the drone’s flight path
and other vitals. Reading the data packets can, however, be a challenging task
as they are encrypted with the AES encryption key.

  7. 7.

Physical evidence: Components of drones, such as propellers, batteries, and
other elements, can be analyzed to determine the type of drone, the
manufacturer, and other useful information. Batteries and propellers can
provide an approximation of the drone’s range, which when combined with other
evidence, can help identify its origin. Components and payload utilized may
have a serial number that can be tracked. Typically, commercially available
drones such as DJI and Parrot sync previous flight logs and other data to
cloud-based platforms that can be accessed using the drone’s serial number.
Additionally, the drone and its associated equipment may contain moist
evidence such as fingerprints, DNA, etc.

  8. 8.

Witness testimonies: Witnesses who saw or heard the drone flying can provide
valuable information about the drone’s flight path and the location of the
drone.

###  3.2 Current Threats and Offences

With the advancement of technology, UAVs have become a widely available
commodity including in the form of toys for children. Drones are part of
everyday life, from recreation to the media and entertainment. However, as
with any other form of technology, if not handled properly, UAVs can be used
for malicious purposes, posing a threat to the general public. The increased
availability of UAVs to the general public has also led to their use in
illegal activities. Multiple instances of privacy invasion by spy drones have
been reported. They are used to collect sensitive personal information such as
credit card information and private communications. Additionally, UAVs have
been observed in restricted or prohibited areas, such as airports and
government buildings, where they pose a safety risk. As per a news article
dated December 21, 2018, numerous UAVs were spotted near the runway of Gatwick
Airport (Fig. 1), because of which all the aircraft activities had to be shut
down [13]. This was not only a security threat but also resulted in the
disruption of flight operations and a loss in revenue. Drones have become a
common method for smuggling narcotics, arms, and ammunition across
international borders, as well as aiding terrorist operations. In January
2015, a drug-toting drone crashed into a supermarket parking lot in Mexico, a
couple of miles from the U.S. border shown in Fig. 2. The article states this
was not the first incident of such a kind [14]. Another instance of UAVs
carrying explosives was caught by the BSF in Srinagar, India in February 2021
[15]. The UAV shown in Fig. 3 was carrying small yet powerful IEDs and could
be attached to vehicles and detonated remotely. UAVs with internet
connectivity can be hacked and used for malicious purposes, such as data theft
or cyberattacks. Although extremely uncommon, UAVs can be armed to conduct
targeted assassinations or infrastructure attacks. Unauthorised or improperly
monitored UAV use in public areas can pose security risks such as explosives
and chemical attacks.

Figure 2: UAV caught spying around the
London Gatwick airport in 2018 111Image credits: ©Wired.co.uk /John
Stillwell/PA Wire/PA Images

Figure 3: Mexican drug cartels using drones to
smuggle narcotics into the U.S. in 2015 222Image credits: AP Photo/Secretaria
de Seguridad Pública Municipal de Tijuana

The aforementioned events are common occurrences that made headlines. The use
of UAVs in illicit activity is limitless due to the escalating technological
advances and falling prices of UAVs. This poses the law enforcement community
with new challenges. As a result of such threats, drone forensic programmes
are crucial for advancing the comprehension of drone technology, lowering the
crime rate, and ensuring the safety and security of individuals,
organisations, and the government. Not always is a criminal activity
associated with UAV forensics. Accidents involving UAVs are extremely
prevalent in the research and development industry, and drone forensics can
help determine the cause and parties involved. This can help prevent future
accidents of a similar nature. There are restrictions on where drones can fly
and how they can be used in a number of nations. Using drone forensics, it is
possible to determine whether or not these regulations have been violated and
to identify the offenders. Despite the benefits a drone forensics programme
can provide to law enforcement agencies, this is still a relatively unexplored
area [7].

Figure 4: Pakistani drone carrying arms shot down by the Indian army in
2020333Image credits: File photo taken from The Times of India

###  3.3 Forensic Challenges

The reliability of the analysis and, consequently, its conclusions are highly
dependent on the quality and quantity of the evidence gathered. Analysis of
UAVs for forensic purposes presents numerous challenges. Collecting relevant
data from drones, which can fly at high altitudes and are often difficult to
track, is one of the greatest obstacles. Drones can collide or be destroyed
during flight, making it difficult to retrieve data or any other physical
evidence. Typically, drones employ encryption to safeguard data, making it
difficult to extract pertinent information. Encryption also makes it difficult
to decode the data packets exchanged between the UAV and the ground station
using the telemetry module. Forensic investigators must possess the skills and
equipment necessary to decrypt data without damaging it. In addition, modern
drones are intricate machines with sophisticated software and hardware
components, necessitating an in-depth comprehension of technical details by
investigators. Drones may capture images or other data that violates an
individual’s privacy, raising privacy concerns. Lastly, as drone technology
continues to evolve, forensic investigators must keep abreast of the most
recent advancements in the field.

##  4 Analysis and Results

In this section, a comprehensive method for the forensic analysis of UAVs is
outlined using a simulated scenario. Notably, acquiring and analysing evidence
and data from drones can be a complex process requiring specialised skills and
equipment. The various components of UAV forensics enable law enforcement
agencies to construct an extensive understanding of how drones have been used,
the evidence they contain, and even new discoveries regarding drone data. For
thorough examination and analysis of the evidence, it is necessary to adhere
to the chronological order depicted in Fig. 5.

Figure 5: UAV forensic analysis model

Acquisition is the process of gathering and organising all physical and non-
physical data that can be utilized for analysis. During the examination and
analysis phase, collected data is observed and evaluated to draw conclusions.
Each of these steps can be repeated until conclusive results are reached. The
presentation phase necessitates presenting findings in a manner that
authorities can comprehend. After completing the analysis phase, the examiner
must compile the findings and results into a forensic report.

###  4.1 Simulation Scenario

To mimic a situation in which a drone is discovered conducting reconnaissance
in a sensitive area, a fictitious scenario is simulated. The drone is
immediately identified and apprehended, as depicted in Fig. 4. All physical
evidence is collected meticulously and sent for forensic analysis. A custom
drone is used for this purpose as it is challenging to retrieve and analyze
the data as compared to commercially available drones such as DJI. The flight
test is carried out at the Flight Laboratory, IIT Kanpur. What follows is
comprehensive forensics of the evidence.

Figure 6: UAV intercepted while spying in restricted area 444Image blurred due
to security concerns.

###  4.2 Physical Examination

Physical examination of a drone in UAV forensics typically involves inspecting
the drone for any physical damage, collecting data from the drone’s internal
components and analyzing the collected data to understand the drone’s behavior
and any potential tampering. The physical evidence is collected and sent to
the laboratory for careful examination. The images of the UAV are given in
Fig. 7. Examination of the exterior revealed no form of tampering or sign of
wet evidence. The UAV was then carefully dismantled and the components were
studied. A list of all the physical components present on the UAV are given in
Table 1.

Figure 7: Physical examination of the retrieved UAV Table 1: Physical components retrieved from the UAV Component | Description
---|---
Frame | Custom designed carbon fibre frame with ‘X’ quadrotor configuration
Motors | KDE 2814XF 775KV
ESC | KDEXF-UAS35
Propellers | 12$\times$6 hard plastic
Battery | LiPo 4s 6500mAh
Autopilot | Pixhawk cube orange
GPS Module | Here2 GPS
Radio Receiver | FrSky X8R capable of long range communication
Telemetry module | RFD900
Camera | SJCAM SJ4000 HD camera
Weight | 2.36kg
Size | 60cm

The use of carbon fiber in the design of the frame imparts strength yet
keeping it lightweight. Due to its low radar cross-section (RCS), carbon fiber
is also challenging to detect by radar, making it the ideal material for such
applications [16]. The motor propeller combination can generate high thrust
while utilizing minimal electrical power. Due to the UAV’s high electrical
efficiency, its endurance is increased. Pixhawk autopilot used here is the
industry-leading platform for UAV control due to its stability, dependability,
and precision, as well as its versatility and adaptability to various types of
UAV operations. The Here 2 GPS receiver is able to provide precise position
estimates, ensuring the accuracy and reliability of autonomous missions.
Utilizing long range radio and telemetry modules enables the UAV to maintain
long-range connectivity with the GCS and the operator. RFD900 is capable of
transmission up to 40km, a high data transfer rate of up to 256 kbps, and low
power consumption. It is designed with features such as error correction,
automatic retransmission, and adaptive frequency hopping to ensure a stable
and secure connection between the UAV and the ground station. The SJ4000 is a
popular action camera with an ultra-wide 170∘ lens that can capture images and
videos in high resolution. The camera is also waterproof, making it suitable
for UAV use regardless of weather conditions.

###  4.3 Range Estimation using Battery

The recovered UAV is an electric-powered vehicle, and the first thing that
must be performed is to determine its range. The approximate range assuming
ideal conditions, can be determined based on the battery status. This helps to
narrow down the search range for the possible source of origin. When the
flight log data is corrupted or when data logging is manually disabled, this
method is incredibly useful. The UAV was discovered carrying a four-cell LiPo
battery which can be charged up to 16.8V. The total amount of energy contained
in a completely charged battery is indicated by

$E_{total}=V\frac{\text{Current rating (mAh)}}{1000}Wh$ (1)

The voltage reading of the battery when the UAV was retrieved is known from
the physical examination to be 16.2V. A linear relation between battery
voltage and discharge capacity (mAh) can be assumed in this region [17] to get
the discharge rating at 16.2V. Using Eq. (1), the total energy used can be
calculated as

$E_{used}=E_{total}-E_{remaining}$ $E_{used}=\frac{16.8\times 6500-16.2\times
5300}{1000}=23.34\;Wh$

The power utilised by electronic components is necessary for estimating flight
time and, by extension, range. The principal electrical power consumers
onboard are the motors and telemetry module. The discovered camera is powered
by a distinct battery, and no video transmitter was discovered. The power
consumption of the telemetry module is detailed in the manufacturer-supplied
datasheet. To calculate the electrical power consumed by the motor, we need to
know the operating rpm of the motors so that they can generate sufficient
propulsion for the vehicle’s cruise speed. The vehicle’s cruise speed in
autonomous mode can be discovered in the autopilot’s parameter list, which is
detailed in subsequent sections. The thrust and electrical power required to
attain cruise speed can be found on the datasheet for the motor. The official
datasheets are frequently generated under ideal conditions; therefore, it is
more reliable to determine the power consumption of the motor propeller
combination through thrust testing as shown in Fig. 8. To attain greater
precision, flight log data can be analysed to determine the input PWM signals
sent to the motors, and then thrust test data can be used to determine the
power. The following equation can be used to compute the duration of the UAV’s
flight:

$T_{flight}=\frac{E_{used}}{Total\;power\;consumed}\text{(hrs)}$ (2)

From the thrust test data, it was observed that each motor requires 94W of
power to produce thrust for the UAV to attain an average forward speed of
2m/s, while the telemetry module consumes 5W. Using this, the vehicle’s
optimal flight time can be calculated as follows

$T_{flight}=\frac{23.34\times 3600}{94\times 4+5}=220.5sec$

It should be noted that the calculated flight time and range presume ideal
conditions. The actual values may be lower because wind disturbances,
environmental conditions, operating temperature, noise, component
deterioration over time, and other factors increase battery consumption.

Figure 8: Propeller thrust test setup

The maximum range of the UAV can be given by

$R_{max}=T_{flight}V_{avg}$ (3) $R_{max}=220.5\times 2=441m$

###  4.4 Autopilot Flash Data Analysis

The Pixhawk Cube Orange has 2MB of flash memory for storing firmware,
configuration parameters, bootloader, calibration data, and mission-specific
information [18]. The Ground Control Station (GCS) software must be accessed
in order to extract the UAV’s firmware and other information. The GCS software
provides an interface for the operator to monitor and control the UAV’s flight
and mission via wired or wireless communication. This makes it an essential
component of any UAV system, allowing for the safe and efficient operation of
the vehicle. QGroundControl (QGC) GCS software was utilized in this analysis
[19]. QGC offers complete flight management and mission planning for any
MAVLink-enabled UAV. Using the correct COM port and baud rate, the autopilot
can be accessed via a USB interface from the QGC. The firmware is intended to
provide a software platform for tasks such as sensor data acquisition, motor
control, advanced flight modes, external communication, and system-level
function administration. Pixhawk operates with either the Ardupilot or Px4
firmware. The UAV’s firmware specifications can be accessed from the vehicle
setup section in QGC and are shown in Fig. 9. The safety and failsafe measures
employed by the UAV are enlisted in Table 2.

Figure 9: Firmware and airframe specifications of the UAV autopilot Table 2: Failsafe settings in the UAV Parameter | Value | Description
---|---|---
[.5]0 Low battery failsafe trigger | Action: Warning | Warning when the battery reaches the
Level: 15% | specified warn level
Object detection | - | Disabled
RC loss failsafe trigger | Action: Return mode | The UAV will automatically return to the
RC loss timeout: 2 sec | home point if the RC communication is
| lost for more than 2sec
Data link failsafe trigger | - | Telemetry failure does not activate a failsafe
Geofence failsafe trigger | - | Disabled
Return to launch settings | Altitude: 30m | UAV acquires an altitude of 30m, returns
Post RTL: Land immediately | to the home point and lands immediately
Land descent rate | 0.7m/s | -
Vehicle telemetry logging | Enabled | Autopilot logs all the data on the storage device

The configuration settings are parameters that specify the UAV’s behavior in
different situations. They can be modified to tailor the Pixhawk’s behavior to
a particular UAV. This includes the airframe type, flight mode parameters,
sensor calibration values, etc. The configuration parameters can be accessed
by browsing the full parameter list of the autopilot, as depicted in Fig. 10.
Table 3 lists a number of crucial parameters for the UAV.

Figure 10: Autopilot parameter settings in the GCS software Table 3: Configuration parameters of the UAV Parameter | Value | Inference
---|---|---
COM_ARM_MIS_REQ | 0 | UAV does not require valid mission to arm
COM_ARM_WO_GPS | 1 | Allows arming without GPS fix
COM_FLTMODE1 | Stabilized | First flight mode
COM_FLTMODE4 | Altitude | Fourth flight mode
COM_FLTMODE6 | Mission | Sixth flight mode
COM_OBS_AVOID | 0 | Obstacle avoidance disabled
MIS_LTRMIN_ALT | 10m | Minimum loiter altitude
MPC_LAND_SPEED | 0.7m/s | Landing descent rate
MPC_TILTMAX_AIR | 45.0∘ | Maximum tilt angle in air
MPC_TKO_SPEED | 1.50m/s | Takeoff climb rate
MPC_XY_CRUISE | 5.0m/s | Maximum horizontal velocity in mission
MPC_Z_VEL_MAX_DN | 1.0m/s | Maximum vertical descent velocity
MPC_Z_VEL_MAX_UP | 3.0m/s | Maximum vertical descent velocity
FD_ESCS_EN | 1 | Enable failure detector check on ESCs that
report the arming state
GPS_1_GNSS | 0 | GNSS systems for the primary receiver.
Use GPS system (with QZSS)
GPS_1_PROTOCOL | u-blox | Protocol for primary GPS
MAV_TYPE | Quadrotor | MAVLink airframe type
PWM_AUX_RATE | 50Hz | PWM output frequency of the auxiliary channels
SDLOG_BOOT_BAT | 0 | Logging will start from boot even if
battery power is not detected
SDLOG_MODE | When armed until disarmed | Logging mode
SYS_AUTOCONFIG | Keep parameters | Automatically configure the values
SYS_MC_EST_GROUP | ekf2 | Set multicopter estimator group

The mission specifics can be used to retrieve autonomous mission details such
as the home location, takeoff and landing points, waypoints, flight path, and
failsafes. This aids in determining the possible origin, flight path, and
other objectives of the UAV. Fig. 11 depicts how the pre-planned mission
details can be read directly from flash memory via the flight plan section of
the QGC software. The location marked ‘L’ represents the home or launch point
of the UAV, while the location marked ‘T’ represents the launching point. The
numbers and the line connecting them depict, respectively, the location of the
waypoints and the path the UAV is instructed to follow. It should be noted
that the number of waypoints does not increase at regular intervals. This is
due to the fact that the UAV is commanded to perform additional tasks at these
locations, such as velocity changes, loitering, etc.

Figure 11: UAV’s pre-planned mission retrieved from the autopilot flash memory

The UAV is initially instructed to take off at the height of 10m and move
along the runway to waypoint 2 at a speed of 5m/s (specified in Table 3). It
then enters the taxiway and pursues the remaining path at a speed of 2m/s. The
waypoint description indicates that between waypoints 6 and 10, the UAV must
sustain a heading of 180 degrees and hold the position at the tarmac’s edge
(visible in Fig. 11) for 5sec. The holding period is established so that a
sufficient quantity of high-quality images can be collected at these
locations. Once the UAV has collected data from all the specified waypoints,
it must return to the launch location by retracing the path marked by the red
line.

###  4.5 Flight Log Analysis

Pixhawk has a logging feature that generates log files used for debugging and
analyzing the UAV’s flight behavior. Log files include information such as
sensor readings, vehicle status, and control inputs that were recorded during
the flight. Using the GCS software, the binary data can be converted to a
readable format. Due to their large size, log files cannot be saved in the
flash memory. Pixhawk employs a microSD card to store flight log data in
addition to the boot log file, a backup copy of configuration parameters, and
other crucial information. To access the log files, the SD card must be
removed from the Pixhawk and accessed. Fig.12 depicts the steps required to
access a particular log file.

Figure 12: Log file retrieval procedure

The log files are in ULG format and can be analyzed using Matlab Flight Log
Analyzer, Px4 Flight Review, or any other third-party application. Matlab
Flight Log Analyzer was utilized for this analysis due to its advantages in
terms of customization, data synchronization, interactive data exploration,
and data exporting. Table 4 summarizes the UAV’s entire flight course.

Table 4: Flight summary Total flight time | 2min 26sec
---|---
Total distance | 295.1m
OS Version | NuttX, v11.0.0
Estimator | EKF2
Average speed | 2m/s
Max speed | 5.06m/s
Max up speed | 2.8m/s
Max down speed | 1.25m/s
Max tilt angle | 19.8∘

The trajectory followed by the UAV is shown in Fig. 13. The autopilot
integrates the barometer altitude and GPS altitude to provide a more precise
fused altitude estimation. Similarly, precise x-y coordinates are estimated by
combining GPS and IMU data. As discussed in subsection 4.4, the UAV does not
successfully complete its mission by returning to the launch position. This is
because the UAV was intercepted and captured as it approached waypoint 10 in
Fig. 11.

(a) Map view

(b) 3D trajectory with takeoff point as the origin

Figure 13: Flight path of the UAV

Fig. 15 and Fig. 15 depict the attitude and x-y-z velocity of the UAV in the
body frame, respectively. Due to the low magnitude of the reference
velocities, the UAV’s attitude remains small for the majority of flight time.
The erratic peaks observed in the attitude are the result of unanticipated
wind turbulence. Notably, the time axis in these graphs does not begin at 0
seconds. This is because the Pixhawk autopilot’s clock begins when the system
is powered on, whereas data recording does not begin until the UAV is armed.
Therefore, it can be inferred that the UAV was armed and initiated the mission
after approximately 481 seconds of being powered on.

Figure 14: Vehicle attitude

Figure 15: Vehicle velocity in the body frame

Since the problem statement necessitates tracing the precise flight path of
the surveillance UAV, only the pertinent flight log data and corresponding
results have been discussed. In addition to the data and results discussed
earlier, the log files also contain information such as attitude rate, angular
velocity and acceleration, radio inputs, actuator outputs, vibration metrics,
GPS uncertainty, magnetic field, logged messages, etc. The flight history of
the UAV can be more accurately estimated by combining these data.

###  4.6 Other Data Retrieved

After a comprehensive analysis of the autopilot data, it is imperative to
search for any additional storage devices on the UAV, as the data may provide
additional information about the UAV’s whereabouts, origin, and operator. As
specified in section 4.2, the UAV is equipped with an HD camera for gathering
images. The camera includes a 16GB micro SD card because high-resolution
images and videos necessitate a secondary large storage medium. A thorough
examination of the SD card revealed that the UAV captured numerous images from
a variety of locations and perspectives during its flight, some of which may
contain sensitive information. Several of these images are depicted in Fig. 5.
Although the images are not geo-tagged and identifying the individuals
captured by the camera is difficult, the time stamp of the images can be
combined with the flight log data to generate a visual representation of the
UAV’s flight path. This can aid in accurately pinpointing the origin of the
UAV.

(a) Image 1

(b) Image 2

(c) Image 3

(d) Image 4

Figure 16: Images extracted from the camera onboard the UAV 555Images blurred
due to security concerns.

##  5 Conclusion

This study highlights the significance of UAV forensics as well as the
associated difficulties. As the use of UAVs continues to expand in multiple
industries, there is an increasing need for forensic investigation
capabilities to ensure the safe and responsible operation of these systems.
Using a custom reconnaissance UAV as a case study, a detailed methodology for
the forensic analysis of a UAV has been outlined. The novel contribution of
this study is the use of a cheaper custom UAV for the analysis, as they can be
tailored to a particular application, making them difficult to monitor and
neutralize. This was accomplished by analyzing the various components of the
recovered UAV in chronological order using open-source software and MATLAB.
The analysis contributes novel insights to the existing corpus of knowledge on
drone forensics. Through a combination of physical analysis, data recovery,
and data analysis, we have demonstrated that UAV forensics has the potential
to yield invaluable insights into the operation of these complex systems. The
data has been meticulously analyzed, and conclusions have been derived based
on evidence from a variety of sources. The results indicate that forensic
analysis of UAVs can aid in the investigation of accidents, incidents, and
criminal activities involving these systems. However, UAV forensics is a
difficult and rapidly evolving field that requires additional research and
development to enhance the efficacy and dependability of forensic techniques.

## References

  * Shakhatreh et al. [2019] Shakhatreh, H., Sawalmeh, A. H., Al-Fuqaha, A., Dou, Z., Almaita, E., Khalil, I., Othman, N. S., Khreishah, A., and Guizani, M., “Unmanned aerial vehicles (UAVs): A survey on civil applications and key research challenges,” _Ieee Access_ , Vol. 7, 2019, pp. 48572–48634.
  * Mondal et al. [2020] Mondal, S., Williamson, A., Xu, Z., and Tsourdos, A., “Autonomous architecture for uav-based agricultural survey,” _AIAA Scitech 2020 Forum_ , 2020, p. 2298.
  * Muhammad et al. [2020] Muhammad, W., Esposito, F., Rajashekar, S. C., and Gururajan, S., “Vocal intent programmability for uas in disaster scenarios,” _AIAA Scitech 2020 Forum_ , 2020, p. 0736.
  * Sathyamoorthy [2015] Sathyamoorthy, D., “A review of security threats of unmanned aerial vehicles and mitigation steps,” _J. Def. Secur_ , Vol. 6, No. 1, 2015, pp. 81–97.
  * Alotaibi et al. [2022] Alotaibi, F. M., Al-Dhaqm, A., Al-Otaibi, Y. D., and Alsewari, A. A., “A comprehensive collection and analysis model for the drone forensics field,” _Sensors_ , Vol. 22, No. 17, 2022, p. 6486.
  * Jain et al. [2017] Jain, U., Rogers, M., and Matson, E. T., “Drone forensic framework: Sensor and data identification and verification,” _2017 IEEE Sensors Applications Symposium (SAS)_ , IEEE, 2017, pp. 1–6.
  * Bouafif et al. [2018] Bouafif, H., Kamoun, F., Iqbal, F., and Marrington, A., “Drone forensics: challenges and new insights,” _2018 9th IFIP International Conference on New Technologies, Mobility and Security (NTMS)_ , IEEE, 2018, pp. 1–6.
  * Lan and Lee [2022] Lan, J. K. W., and Lee, F. K. W., “Drone Forensics: A Case Study on DJI Mavic Air 2,” _2022 24th International Conference on Advanced Communication Technology (ICACT)_ , IEEE, 2022, pp. 291–296.
  * Kao et al. [2019] Kao, D.-Y., Chen, M.-C., Wu, W.-Y., Lin, J.-S., Chen, C.-H., and Tsai, F., “Drone forensic investigation: DJI spark drone as a case study,” _Procedia Computer Science_ , Vol. 159, 2019, pp. 1890–1899.
  * Al-Room et al. [2021] Al-Room, K., Iqbal, F., Baker, T., Shah, B., Yankson, B., MacDermott, A., and Hung, P. C., “Drone forensics: A case study of digital forensic investigations conducted on common drone models,” _International Journal of Digital Crime and Forensics (IJDCF)_ , Vol. 13, No. 1, 2021, pp. 1–25.
  * Atkinson et al. [2021] Atkinson, S., Carr, G., Shaw, C., and Zargari, S., “Drone forensics: the impact and challenges,” _Digital Forensic Investigation of Internet of Things (IoT) Devices_ , 2021, pp. 65–124.
  * Burdziakowski [2017] Burdziakowski, P., “Low cost hexacopter autonomous platform for testing and developing photogrammetry technologies and intelligent navigation systems,” 2017.
  * Stokel-Walker [2018] Stokel-Walker, C., “Why the Gatwick drones were so hard to stop,” _Wired_ , 2018.  URL https://www.wired.co.uk/article/gatwick-drones.
  * Bever [2015] Bever, L., “The meth-toting drone that crashed in Tijuana,” _The Washington Post_ , 2015.  URL https://www.washingtonpost.com/news/morning-mix/wp/2015/01/22/a-meth-toting-drone-just-crashed-in-tijuana/.
  * JK [2021] “Arrival of ’sticky bombs’ in Kashmir sets off alarm bells,” _The Times of India_ , 2021.  URL https://timesofindia.indiatimes.com/india/arrival-of-sticky-bombs-in-kashmir-sets-off-alarm-bells/articleshow/81257273.cms.
  * Semkin et al. [2020] Semkin, V., Haarla, J., Pairon, T., Slezak, C., Rangan, S., Viikari, V., and Oestges, C., “Analyzing radar cross section signatures of diverse drone models at mmWave frequencies,” _IEEE Access_ , Vol. 8, 2020, pp. 48958–48969.
  * Liu et al. [2022] Liu, S., Chen, J., Zhang, C., Jin, L., and Yang, Q., “Experimental study on lithium-ion cell characteristics at different discharge rates,” _Journal of Energy Storage_ , Vol. 45, 2022, p. 103418.
  * px4 [accessed 03 May 2023] “CubePilot Cube Orange Flight Controller,” _PX4 User Guide_ , accessed 03 May 2023.  URL https://docs.px4.io/main/en/flight_controller/cubepilot_cube_orange.html.
  * qgc [accessed 03 May 2023] “QGC - Qgroundcontrol - drone control,” , accessed 03 May 2023.  URL http://qgroundcontrol.com/.