# Gradient Boosting Machine: A Survey

Zhiyuan<EMAIL_ADDRESS>Point Zero One Technology  Fordham University
Danchen Lin  Point Zero One Technology  Imperial College London  Thomas Lau
Point Zero One Technology  Mike Wu  Point Zero One Technology  Massachusetts
Institute of Technology

###### Abstract

In this survey, we discuss several different types of gradient boosting
algorithms and illustrate their mathematical frameworks in detail: _1._
introduction of gradient boosting leads to _2._ objective function
optimization, _3._ loss function estimations, and _4._ model constructions.
_5._ application of boosting in ranking.

##  1 Introduction

Proposed by Freund and Schapire (1997), boosting is a general issue of
constructing an extremely accurate prediction with numerous roughly accurate
predictions. Addressed by Friedman (2001, 2002) and Natekin and Knoll (2013),
the Gradient Boosting Machines (GBM) seeks to build predictive models through
back-fittings and non-parametric regressions. Instead of building a single
model, the GBM starts by generating an initial model and constantly fits new
models through loss function minimization to produce the most precise model
(Natekin and Knoll, 2013).

This survey concentrates on the mathematical derivations of the gradient
boosting algorithms. In Section 2, we analyze the optimization methods for
parametric and non-parametric models. Section 3 covers the definitions of
different types of loss functions. In Section 4, we present different types of
boosting algorithms, while in Section 5, we explore the combination of
boosting algorithms and ranking algorithms to rank the real-world data.

##  2 Basic Framework

The ultimate goal of the GBM is to find a function $F(x)$, which minimize its
loss function $L(y,F(x))$ as

$F^{*}=\operatorname*{argmin}_{F}E_{y,x}L(y,F(x)),$

through iterative back-fitting.

###  2.1 Numerical Optimization

By definition, a boosted model is a weighted linear combination of the base
learners

$F(x;\\{\beta_{m},a_{m}\\}_{1}^{M})=\sum_{m=1}^{M}\beta_{m}h(x:a_{m}),$

where $h(x;a)$ is a base learner parameterized by $a$. Regarded as weak
learners, the base learners produce hypotheses that only predict slightly
better than random guessing, and it was proved that recursive learning with
weak learners can perform just as good as a strong learning algorithm
(Schapire, 1990).

If the base learner is a regression tree, the parameter $a$ is usually the
splitting nodes of tree branches (Friedman, 2002). Tree-based models divide
the input variable space into regions and apply a series of rules to identify
the regions that have the strongest responses to the inputs (Elith et al.,
2008). Each region is then fitted with a regression tree taking the mean
response of the observations (Elith et al., 2008). Decision trees are
constructed through binary splits, and recursive splits generate a large tree,
which is then pruned to drop out the weak branches.

The optimization process can be written as follows

$P^{*}=\operatorname*{argmin}_{p}\Phi(P)\Phi(P)=E_{y,x}L(y,F(x;P))F^{*}(x)=F(x;P^{*}),$

where $P^{*}=\sum_{m=0}^{M}p_{m},$ and $p_{m}$ are the consecutive boosting
steps.

One of the approaches to generate these steps, $p_{m}$, is to use the
steepest-descent algorithm by calculating the gradient

$g_{m}=\\{g_{jm}\\}=\left\\{\frac{\delta\Phi(P)}{\delta
P_{j}}\bigg{|}_{P=P_{m-1}}\right\\},$

where $P_{m-1}=\sum_{i=0}^{m-1}p_{i}.$ The boosting step in the previous
function is given by

$p_{m}=-\rho_{m}g_{m},$

and $\rho_{m}=\operatorname*{argmin}_{\rho}\Phi(P_{m-1}-\rho g_{m})$ is the
line search on the direction of steepest-descent.

In the non-parametric case, the function $F$ is solved by minimizing

$\Phi(F)=E_{y,x}L(y,F(x))=E_{x}[E_{y}(L(y,F(x)))|x]=E_{y}[L(y,F(x))|x],$

and the optimum is reached at $F^{*}(x)=\sum_{m=0}^{M}f_{m}(x),$ where
$f_{m}(x)=-\rho_{m}g_{m}(x).$

The gradient of the non-parametric model is

$g_{m}(x)=\frac{\delta\Phi(F(x))}{\delta
F(x)}\bigg{|}_{F(x)=F_{m-1}(x)}=\frac{\delta E_{y}[L(y,F(x))|x]}{\delta
F(x)}\bigg{|}_{F(x)=F_{m-1}(x)}.$

The differentiation and integration of the gradient function can be switched
interchangeably, and the gradient function can be simplified to

$g_{m}(x)=E_{y}\left[\frac{\delta\Phi(F(x))}{\delta
F(x)}\bigg{|}x\right]_{F(x)=F_{m-1}(x)}.$

According to the above equation, the line search is solved as

$\rho_{m}=\operatorname*{argmin}_{\rho}E_{y,x}L(y,F_{m-1}(x)-\rho g_{m}(x)).$

###  2.2 Finite Dataset

With finite number of samples, a non-parametric function $F(x)$ can be
obtained by a greedy stage-wise algorithm. Different from stepwise strategy,
the stage-wise strategy does not make adjustments to the previous boosting
steps, and thus can be demonstrated as

$(\beta_{m},a_{m})=\operatorname*{argmin}_{\beta,a}\sum_{i=1}^{N}L(y_{i},F_{m-1}(x_{i})+\beta
h(x_{i};a)).$

Thus, $F(x)$ can be obtained iteratively

$F_{m}(x)=F_{m-1}(x)+\beta_{m}h(x;a_{m}).$

##  3 Estimation

Under the framework of GBM, different loss functions can be applied to solve
different tasks (Natekin and Knoll, 2013; Koenker and Hallock, 2001; Friedman,
2002).

###  3.1 Continuous Response

$L_{1}$ loss function, known as Laplacian loss, is presented as

$L(y,F)_{L_{1}}=|y-F|,$

which is the absolute value of residuals between the explained variable $y$
and the predictive function $F$, while the most commonly used squared-error
$L_{2}$ loss function is defined as

$L(y,F)_{L_{2}}=\frac{1}{2}(y-F)^{2}.$

In addition, the Huber loss function that merges $L_{1}$ and $L_{2}$ loss
functions described above can be a robust alternative to the $L_{1}$ loss
function

$L(y,F)_{Huber,\delta}=\left\\{\begin{array}[]{rcl}\frac{1}{2}(y-F)^{2}\ \ \
|y-F|\leq\delta\\\ \delta(|y-F|-\frac{\delta}{2})\ \ \
|y-F|>\delta\end{array}\right..$

Quantile loss is inevitably handy in the situations of ordering and sorting
because of its robustness, which is framed as

$L(y,f)_{\alpha}=\left\\{\begin{array}[]{rcl}(1-\alpha)|y-f|\ \ \ y-f\leq 0\\\
\alpha|y-f|\ \ \ y-f>0\end{array}\right.,$

where $\alpha$ designates the targeted quantile in the conditional
distribution. The loss function can be degenerated into the $L_{1}$ loss by
taking $\alpha=0.5$.

###  3.2 Categorical Response

There are two loss functions designed for categorical response that are widely
used, namely the Bernoulli loss function and the exponential loss function.
The Bernoulli loss function is formulated as

$L(y,F)_{Bern}=\log(1+\exp(-2\overline{y}F)),$

while in Adaboost algorithm, the same alteration on $y$ variable in Bernoulli
loss is applied on $y$ variable in exponential loss (Natekin and Knoll, 2013)

$L(y,F)_{Ada}=\exp(-\overline{y}F).$

##  4 Methodology

### AdaBoost

Gradient boosting is a generalization of Adaboost. The design of Adaboost
(Freund and Schapire, 1997), the original boosting algorithm, is to find a
hypothesis with low prediction error relative to a given distribution over the
training samples. Freund and Schapire (1997) demonstrated their algorithm
through a horse gambling example, where a gambler wishes to bet on the horse
that has the greatest chance to win. In order to increase the winning
probability of a bet, the gambler is encouraged to gather the expert opinions
before placing a bet. Such a process of collecting information from different
experts is similar to the ensemble of a class of poor classifiers. In
Adaboost, each expert’s opinion corresponds to a training set (Wang, 2012).
Each sample is initialized with a weight, and the weights of the training sets
are adjusted after each iteration, such that the weights of misclassified
samples are increased, while the weights of correctly classified samples are
decreased.

In each iteration of boosting, the current weak learner of Adaboost chooses a
weak hypothesis from the entire set of weak hypotheses instead of just the
weak hypotheses that are currently found to the point. Since the search of an
entire space of hypotheses can be enormous amount of work, it is often
suitable to apply weak learners that approximately cover the whole set
(Collins et al., 2002).

Boosting algorithms with certain modifications perform well under high bias
and high variance settings. When weighted sampling is implemented for the
training data, the performance of boosting is determined by its ability to
reduce variance (Friedman et al., 2000). Meanwhile, boosting performance
depends on bias reduction when the weighted sampling is replaced with weighted
tree fitting (Friedman et al., 2000).

Additionally, Adaboost is prone to cause model overfitting because of the
exponential loss. The overfitting may be mitigated by minimizing the
normalized sigmoid cost function in exchange (Mason et al., 2000),

$C(F)=\frac{1}{m}\sum_{i=1}^{m}1-\tanh(\lambda y_{i}F(x_{i})).$

In the above function, $F$ is a convex combination of weak hypotheses, and the
parameter $\lambda$ measures the steepness of the margin cost function
$c(z)=1-\tanh(\lambda z)$. Through their experiments, Mason et al. (2000)
showed that a new boosting algorithm optimizing normalized sigmoid cost,
called DOOM II, overall performed better than Adaboost. According to Mason et
al. (1999), AnyBoost is a general boosting algorithm that optimize gradient
descent in an inner product space. The inner product space $S$, which is
inclusive of all linear combinations of weak hypotheses, contains the weak
hypotheses and their combination $F$. The inner product can thus be
represented as,

$\langle F,G\rangle\coloneqq\frac{1}{m}\sum_{i=1}^{m}F(x_{i})G(x_{i}),$

where $F$ and $G$ are the combinations of weak hypotheses that belong to the
set of all linear combinations of weak hypotheses. Only AnyBoost algorithm
that implies the inner product function and normalized sigmoid cost function
is referred to DOOM II (Mason et al., 2000).

### Arc-x4

Arcing, a concept introduced by Breiman (1996) and utilized in Adaboost, is a
technique to adaptively reweighting the training samples. Arc-x4 (Breiman,
1997) performs similarly to the original boosting in training error and
generalization error reduction. At each boosting step, a new training sample
is generated from the training set with probability

$p(n)=\frac{(1+m(n)^{4})}{\sum(1+m(n)^{4})},$

where $m(n)$ is the number of misclassified cases.

### Least Squares Boost

The least-squared loss function in continuous response is one of the most
commonly used loss function. In parametrized model, the optimization using the
least-squared loss has an equation

$(\rho_{m},a_{m})=\operatorname*{argmin}_{a,\rho}\sum_{i=1}^{N}[\widetilde{y_{i}}-\rho
h(x_{i};a)]^{2}.$

Solving for $F(x)$, we obtain a stage-wise model

$F_{m}(x)=F_{m-1}(x)+\rho_{m}h(x;a_{m}).$

### Logitboost

Another well-known boosting Algorithm is Logitboost. Similar to other boosting
algorithms, Logitboost adopts regression trees as the weak leaners. Deriving
from the logistic regression, Logitboost takes the negative of the
loglikelihood of class probabilities (Li, 2012). Defined as $p$, class
probability is formulated as

$p_{i,k}=Pr(y_{i}=k|x_{i})=\frac{e^{F_{i,k}(x_{i})}}{\sum_{s=0}^{K-1}e^{F_{i,s}(x_{i})}},$

where $y_{i}$ is the output vector and $X_{i}$ is the input vector. Thus, the
loss function of Logitboost can be written out

$L=\sum_{i=1}^{N}L_{i},L_{i}=-\sum_{k=0}^{K-1}r_{i,k}\log p_{i,k},$

where $r_{i,k}=0$ if $y_{i}\neq k$ and $r_{i,k}=1$ on the contrary. A
stagewise model follows as

$F_{i,k}=F_{i,k}+v\frac{K-1}{K}(f_{i,k}-\frac{1}{K}\sum_{k=0}^{K-1}f_{i,k}),$

where $v$ is a shrinkage parameter, and $f_{i,k}$ is the objective function.

Beside the class probabilities, another important factor in Logitboost is the
dense Hessian matrix, which is obtained by computing the tree split gain and
node value fitting. However, certain modifications are required in order to
incorporate these factors into optimization. The sum-to-zero constraint of
classifier, implied by the sum-to-one constraint of the class probabilities,
can be settled by adopting a vector tree at each boost. In the vector tree, a
sum-to-zero vector is fitted at each split node in the K-dimensional space.
Moreover, adding the vector tree allows explicit computations of the split
gain and node fitting, which becomes a secondary problem when fitting a new
tree. Such secondary problems can then be used to cope with the dense Hessian
matrix, where only two coordinates are allowed for each of the secondary
problems (Sun et al., 2012).

### LAD Regression

The LAD regression proposed by Friedman (2002) has its loss function as
$L(y,F)=|y-F|$, where $F(x)$ is solved by

$F_{m}(x)=F_{m-1}(x)+\sum_{j-1}^{J}\gamma 1(x\in R_{jm}),\ \ \
\gamma_{jm}=\rho_{m}b_{jm}.$

Moreover, in the LAD regression, the gamma parameter is

$\gamma_{jm}=\operatorname*{median}_{x_{i}\in
R_{jm}}\\{y_{i}-F_{m-1}(x_{i})\\}.$

### M-Regression

M-Regresison (Friedman, 2002) is designed to incorporate with the Huber loss
function

$\gamma_{jm}=\widetilde{r_{jm}}+\frac{1}{N_{jm}}\sum_{x_{i}{\in
R}_{jm}}\operatorname*{sign}(r_{m-1}(x_{i})-\widetilde{r_{jm}})\bullet\min(\delta_{m},\operatorname*{abs}(r_{m-1}(x_{i})-\widetilde{r_{jm}})),$

where $\widetilde{r_{jm}}=\operatorname*{median}_{x_{i}\in
R_{jm}}\\{r_{m-1}(x_{i})\\}$ and $r_{m-1}(x_{i})=y_{i}-F_{m-1}(x_{i}).$

### Two Class Logistic Regression

The loss function applied in the logistic regression is a binary function
(Friedman, 2002), which is the Bernoulli loss function. Approximately, the
line search of the logistic regression can be solved from Bernoulli loss

$\gamma_{jm}=\sum_{x_{i}\in R_{jm}}\widetilde{y_{i}}\bigg{/}\sum_{x_{i}\in
R_{jm}}|\widetilde{y_{i}}|(2-|\widetilde{y_{i}}|),j=1,\ldots,J,$

where
$\widetilde{y_{i}}=2\widetilde{y_{i}}/(1+\exp(2y_{i}F_{m-1}(x_{i}))),i=1,\ldots,N.$

### Multiclass Logistic Regression

The loss function applied in the Multi-class logistic regression are as
follows

$L(\\{y_{k},F_{k}(x)\\}_{1}^{K})=-\sum_{k=1}^{K}y_{k}\log p_{k}(x),\ \ \
y_{k}=1\in\\{0,1\\},\ \ \ p_{k}(x)=Pr(y_{k}=1|x),$

where the line search of the multi-class logistic regression is

$\gamma_{jkm}=\frac{K-1}{K}\frac{\sum_{x_{i}\in
R_{jkm}}\widetilde{y_{ik}}}{\sum_{x_{i}\in
R_{jkm}}|\widetilde{y_{ik}}|(1-|\widetilde{y_{ik}}|)}.$

##  5 Ranking Problem

One of the most discussed problems in machine learning is teaching a computer
to rank. Two sets of data are required before constructing a ranking algorithm
(Zheng et al., 2008), i.e., the preference data containing a set of features,
and the ranked targets. Based on these two datasets, the ranking function can
be computed for each dataset under an optimization problem.

The objective function for the ranking problem is

$R(h)=\frac{w}{2}\sum_{i=1}^{N}(\max\\{0,h(y_{i})-h(x_{i})+\tau\\})^{2}+\frac{1-w}{2}\sum_{i=1}^{n}(l_{i}-h(z_{i}))^{2},$

where $x_{i}$ and $y_{i}$ are the features in the preference data, and
$h(x_{i})\leq h(y_{i})+\tau,$ if $x_{i}$ is ranked higher than $y_{i}$.

Wu et al. (2008) proposed a highly effective ranking algorithm LambdaMART
which integrates LambdaRank function and boosting. The LambdaRank function
aims to maximize the Normalized Discounted Cumulative Gain (NDCG)

$N_{i}\coloneqq n_{i}\sum_{j=1}^{T}(2^{r(j)}-1)/\log(1+j),$

where $r(j)$ represents the ranking of the targets. Gamma gradient is used in
the optimization

$\gamma_{i,j}\coloneqq S_{ij}\bigg{|}\triangle NDCG\frac{\delta C_{ij}}{\delta
o_{ij}}\bigg{|},$

where $S_{ij}$ takes the value of 1 or -1 depending on the relevance of the
items. For example, in ranking for webpages, the gamma gradient is used to
determine the relevance of information retrieved online. If a piece of
information $i$ is more relevant than another piece $j$, then $S_{ij}$ equals
to 1; otherwise $S_{ij}$ equals to -1. The $o_{ij}$ represents the difference
between the ranking scores predicted by the ranking function $o_{ij}\coloneqq
F(x_{i})-F(x_{j}),$ and $C_{ij}\coloneqq
C(o_{ij})=F(x_{j})-F(x_{i})+\log(1+e^{s_{i}-s_{j}}).$ Moreover, the gamma
gradient of a specific item $i$ is as follows

$\gamma_{i}=\sum_{j\in P}\gamma_{ij}.$

##  6 Conclusion

In this paper, we summarize the Gradient Boosting Algorithms from several
aspects, including the general function optimization, the objective functions,
and different loss functions. Additionally, we present a set of boosting
algorithms with unique loss functions, and we solve their predictive models
accordingly.

## References

  * Breiman (1996) Breiman, L. (1996). Bias, Variance, and Arcing Classifiers. Statistics Department, University of California, Berkeley, CA, USA. _Tech. Rep._ 460.
  * Breiman (1997) Breiman, L. (1997). ARCING THE EDGE. Statistics Department, University of California, Berkeley, CA, USA. _Tech. Rep._ 486.
  * Collins et al. (2002) Collins, M., Schapire, R. E., & Singer, Y. (2002). Logistic Regression, Adaboost amd Bregman Distances. _Machine Learning_ 48, 253–285.
  * Elith et al. (2008) Elith, J., and Leathwick, J. R., & Hastie, T. (2008). A Working Guide to boosted Regression Rrees. _Journal of Animal Ecology_ 77, 802–813.
  * Freund and Schapire (1997) Freund, Y., and Schapire, R. E. (1997). A Decision Theoretic Generalization of Online Learning and An Application to Boosting. _Journal of Computer and System Sciences_ 55, 119–139.
  * Friedman (2001) Friedman, J. H. (2001). Greedy Function Approximation: A Gradient Boosting Machine. _Annals of Statistics_ , 1189–1232.
  * Friedman (2002) Friedman, J. H. (2002). Stochastic Gradient Boosting. _Computational Statistics & Data Analysis_ 38, 367–378.
  * Friedman et al. (2000) Friedman, J., Hastie, T., & Tibshirani, R. (2000). ADDITIVE LOGISTIC REGRESSION: A STATISTICAL VIEW OF BOOSTING. _The Annals of Statistics_ 28, 337–407.
  * Koenker and Hallock (2001) Koenker, R., and Hallock, K. F. (2001). Quantile Regression _Journal of Economic Perspectives_ 15, 143–156.
  * Li (2012) Li, P. (2012). Robust Logitboost and Adaptive Base Class (abc) Logitboost. _arXiv preprint arXiv:1203.3491_.
  * Mason et al. (1999) Mason, L., Baxter, J., Bartlett, P., & Frean, M. (1999). Boosting Algorithms as Gradient Descent in Function Space.
  * Mason et al. (2000) Mason, L., Baxter, J., Bartlett, P., & Frean, M. (2000). Boosting Algorithms as Gradient Descent. _Advances in Neural Information Processing Systems_ , 512–518.
  * Natekin and Knoll (2013) Natekin, A., and Knoll, A. (2013). Gradient Boosting Machines, A Tutorial. _Frontiers in Neurorobotics_ 7, 21.
  * Schapire (1990) Schapire, R. E. (1990). The Strength of Weak Learnability. _Machine Learning_ 5, 197–227.
  * Sun et al. (2012) Sun, P., Reid, M. D., & Zhou., J. (2012). ASOS-LogitBoost: Adaptive One-Vs-One LogitBoost for Multi-Class Problem. _arXiv preprint arXiv:1110.3907_.
  * Wang (2012) Wang, R. (2012). AdaBoost for Feature Selection, Classification and Its Relation with SVM, A Review. _Physics Procedia_ 25, 800–807.
  * Wu et al. (2008) Wu, Q., Burges, C. J., Svore, K. M., & Gao, J. (2008). Ranking, Boosting, and Model Adaptation.
  * Zheng et al. (2008) Zheng, Z., Zha, H., Zhang, T., Chapelle, O., Chen, K., & Sun, G. (2008). A General Boosting Method and Its Application to Learning Ranking Functions for Web Search. _Advances in Neural Information Processing Systems_ , 1697–1704.


# Global-Local MAV Detection under Challenging Conditions based on Appearance
and Motion

Hanqing Guo, Ye Zheng, Yin Zhang, Zhi Gao, Shiyu Zhao H. Guo is with the
College of Computer Science and Technology at Zhejiang University and the
School of Engineering at Westlake University, Hangzhou, China. (e-mail:
guohanqing@westlake.edu.cn)Y. Zheng, Y. Zhang, and S. Zhao are with the School
of Engineering at Westlake University, Hangzhou, China. (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>zhaoshiyu@westlake.edu.cn)Z. Gao is with the School of Remote Sensing and
Information Engineering, Wuhan University, Wuhan, China. (e-mail:
gaozhinus@gmail.com)(Corresponding author: Shiyu Zhao.)

###### Abstract

Visual detection of micro aerial vehicles (MAVs) has received increasing
research attention in recent years due to its importance in many applications.
However, the existing approaches based on either appearance or motion features
of MAVs still face challenges when the background is complex, the MAV target
is small, or the computation resource is limited. In this paper, we propose a
global-local MAV detector that can fuse both motion and appearance features
for MAV detection under challenging conditions. This detector first searches
MAV targets using a global detector and then switches to a local detector
which works in an adaptive search region to enhance accuracy and efficiency.
Additionally, a detector switcher is applied to coordinate the global and
local detectors. A new dataset is created to train and verify the
effectiveness of the proposed detector. This dataset contains more challenging
scenarios that can occur in practice. Extensive experiments on three
challenging datasets show that the proposed detector outperforms the state-of-
the-art ones in terms of detection accuracy and computational efficiency. In
particular, this detector can run with near real-time frame rate on NVIDIA
Jetson NX Xavier, which demonstrates the usefulness of our approach for real-
world applications. The dataset is available at
_https://github.com/WestlakeIntelligentRobotics/GLAD_. In addition, A video
summarizing this work is available at _https://youtu.be/Tv473mAzHbU_.

###### Index Terms:

Global-local, Appearance and motion features, MAV detection, Air-to-air.

##  I Introduction

Vision-based MAV detection has attracted increasing attention in recent years
due to its application in many tasks such as vision-based swarming[1, 2, 3],
aerial see-and-avoid[4, 5], and malicious MAV detection[6, 7, 8]. Different
from the existing works[9, 10, 11] that consider the ground-to-air scenario,
this work focuses on the air-to-air scenario where a camera carried by a
flying MAV is used to detect another flying target MAV. The air-to-air
scenario is more challenging than the ground-to-air scenario because the
camera itself is moving and the target MAV is often engulfed by complex
background scenes such as buildings, trees, and other man-made things when the
camera looks down from the top (Fig. 1(a)). Moreover, the target MAV may be
extremely small in the image when seen from a distance (Fig. 1(b)). In
addition, the detection algorithm must be sufficiently efficient in order to
be implemented onboard with limited computational resources.

(a) Complex background.

(b) Small MAV (8x8 pixels).

Figure 1: Examples of challenging conditions for MAV detection. Our approach
can work effectively in very challenging conditions such as complex
backgrounds and small-sized MAVs. Red box indicates ground-truth. Yellow box
indicates the predicted bounding box by the local detector of our approach.
(Top right images are better view with 500$\%$ zoom-in)

In recent years, many appearance-based methods that rely on deep learning
techniques have been proposed for vision-based MAV detection. For example,
some state-of-the-art object detection networks such as YOLO series, R-CNN
series, SSD, DETR [12, 13, 14] have been applied to MAV detection. These
methods usually work effectively in relatively simple scenarios where the
target MAV is distinct from the background and its size is relatively large in
the image. However, appearance features are not stable in more complex
scenarios. For example, as shown in Fig. 1(a), the background scene is
extremely complex when the camera looks down from the top. The target MAV can
easily get engulfed in complex background scenes such as trees or buildings.
Moreover, when the target MAV flies far away from the camera, its image may
only occupy a tiny portion of the image. For example, as shown in Fig. 1(b),
an MAV seen from nearly 100 m only occupies 8$\times$8 pixels in an image of
1920$\times$1080 pixels. Although we can zoom in to detect distant MAVs, it
would lead to a smaller field of view which is unfavorable to search and track
moving MAVs. It is therefore necessary to develop high-performance algorithms
to detect MAVs under challenging conditions.

Motion features are helpful in detecting MAVs under challenging conditions.
Many motion-assisted methods which combine appearance features and motion
features have been proposed to detect MAVs based on, for example, background
subtraction[15, 16], low-rank based methods[17, 18], spatio-temporal
information[9, 10, 19], and optical flow [20, 21, 22]. However, motion-
assisted MAV detection still faces the following challenges. First, motion
cues of MAVs are difficult to separate from the background when the background
is non-planar or the camera moves drastically. In particular, most of the
existing motion-assisted methods assume the background scene is planar [15,
23, 24] and use affine transformation or perspective transformation to
estimate the camera movement. Although this assumption is valid in some cases
especially when the camera flies at a high altitude so that the height of the
ground object is neglectable, it is still invalid when the flight altitude is
relatively low. In this case, the ground objects such as buildings, trees, and
lampposts may violate the planar assumption. Second, many existing motion-
assisted methods such as the region-based sliding windows in [20] or the two-
stage approach in [21] are computationally intensive and hence difficult to
implement in onboard computers of MAVs. Third, most of the existing motion-
assisted methods can only detect moving MAVs. Since multirotor MAVs may hover
stationarily, it is necessary to develop a method that can detect both
stationary and moving MAVs by integrating different types of features.

To overcome the limitations of the existing approaches, we propose a new
Global-Local MAV Detector (GLAD) for air-to-air detection of MAVs under
challenging conditions. This algorithm is composed of a global detector, a
local detector, an adaptive search region, and a detector switcher. First, the
global detector is used to search for MAV targets in the full-size image. It
consists of a global appearance-based detection module (GAD) and a global
motion-based detection module (GMD). The GMD serves as a good assistant when
the appearance features are unreliable. Second, after the MAV target has been
detected by the global detector, the local detector is activated to conduct
subsequent detection in a Kalman filter-based adaptive search region cropped
from the neighboring area around the target. This local detector consists of a
local appearance-based detection module (LAD) and a local motion-based
detection module (LMD) as well. It can significantly improve the detection
accuracy under challenging conditions because the target’s resolution in the
adaptive search region is greatly improved compared with the method based on
down-sampling. Third, a detector switcher is designed to adaptively coordinate
the global and local detectors. The detector switcher can adaptively switch
between the global and local detectors based on the detection results of the
previous frames. It can avoid the local detector searching in the wrong search
region when the local detector fails for a while.

The main contributions and novelties of this work can be summarized as
follows.

1) We propose a global-to-local method that can significantly improve the
accuracy and efficiency for MAV detection under challenging conditions. This
has been verified by experimental results on three challenging datasets,
showing that our proposed method outperforms the existing ones including [25,
21, 26].

2) We design a motion-based classifier and an appearance-based classifier that
can effectively and efficiently eliminate the interruptions generated by
imperfect image alignment in non-planar scenes. This is supported by
experimental results in non-planar scenes where 3D structures such as high
buildings, trees, and lampposts are dominant rather than a planar background
assumption which is commonly used in aerial object detection such as [15, 23,
24].

3) We introduce a Kalman filter-based adaptive search region to dynamically
adjust the size and location of the local search region. The experimental
results show that this method can further enhance the robustness of our method
in the presence of occlusion and missing detection.

4) We create a new dataset, named ARD-MAV, which contains 60 videos and
106,665 frames. This dataset contains more challenging scenarios that may
occur in practice. Compared to the existing datasets [15, 20, 14, 27], our
proposed dataset has the smallest average object size. It contains various
real-world challenges such as complex backgrounds, 3D structures, abrupt
camera movement, and small MAVs.

##  II Related Works

###  II-A Appearance-Based MAV Detection

The existing appearance-based MAV detection works can be classified into
conventional methods and deep learning methods. The conventional methods
usually use feature extraction methods to obtain target features and then use
a discriminant classifier to classify the MAV. In particular, the work in [28]
tests Harr-like features, histogram of gradients (HOG), and local binary
patterns (LBP) using cascades of boosted classifiers for MAV detection. The
work in [4] uses the Adaboost algorithm with HOG features for online detection
of MAV. The work in [29] uses 2-dimensional, rotation, and translation
invariant Generic Fourier Descriptor (GFD) features and classifies targets as
a drone or bird by a neural network. Besides, template matching and
morphological filtering[2] have also been considered for MAV detection. These
methods have been verified to be effective in simple cases. Nevertheless, when
the shape and size of the MAV change vastly or the background is too complex,
conventional methods usually have difficulties in these scenarios.

On the other hand, with the fast development of deep learning in object
detection, there emerge many works of MAV detection using deep learning
methods. The work in [13] evaluates eight state-of-the-art deep learning
algorithms on the Det-Fly dataset for MAV detection. Similarly, the authors in
[12] evaluates four state-of-the-art deep learning algorithms on three
representative MAV dataset (MAV-VID, Drone-vs-Bird, Anti-UAV). To further
improve the object detection accuracy, the authors in [30] implement a special
augmentation method and prune the convolution channel and shortcut layer of
YOLOv4 for small drone detection, the authors in [31] propose a novel
comprehensive approach that combines transfer learning based on simulation
data and adaptive fusion to improve small object detection performance.
Although deep learning methods have made great progress compared with
conventional methods, there are still many challenges for appearance-based MAV
detection such as complex backgrounds, motion blur, and small objects.

###  II-B Motion-Assisted MAV Detection

Motion-assisted MAV detection methods aim at detecting the MAV by combining
motion features and appearance features. The existing motion-assisted MAV
detection methods can be classified into stationary cameras and moving
cameras. The works in [32, 11] monitor the sky with a stationary camera and
then use background subtraction and CNN-based object classification for MAV
detection. The works in [19, 10] fuse the spatiotemporal feature of the target
for remote flying drone detection. The work in [9] adopts appearance features
to exclude non-MAV moving targets and then uses a motion-based classification
algorithm to distinguish MAV from other interruptions.

Compared with a stationary camera, MAV detection from a moving camera is much
more challenging since the movement of the background is coupled with the
movement of targets. The works in [33, 15] propose a UAV-to-UAV video dataset
and a general architecture for small MAV detection from a camera mounted on a
moving MAV platform. The authors detect the moving MAV by subtracting
neighboring frames and then identify MAV using a hybrid classifier. Similarly,
the works in [20, 34] create a more challenging dataset for detecting flying
objects using a single moving camera. The authors first employ two CNN
networks in a sliding window fashion to obtain the motion-stabilized spatial-
temporal cubes and then use the third CNN network to classify MAV in each
spatial-temporal cube. The work in [21] proposes a two-stage segmentation
approach. In the first stage, the authors utilize a 2-D convolution network
and channel-pixel-wise attention to extract contextual information based on
overlapping patches. Then, a 3-D convolution network and channel-pixel-wise
attention are used to learn spatiotemporal cues and discover the missing
detections of stage-1. The work in [22] proposes a feature super-resolution-
based UAV detector with motion information extraction based on dense optical
flow. However, these methods are either too time-consuming or only effective
when the target is large enough or the background is very simple, which is
challenging for air-to-air MAV detection in a cluttered environment.

##  III An Overview of the Proposed Method

To effectively detect MAVs under challenging conditions, we propose a global-
local MAV detector called GLAD. The architecture of GLAD is illustrated in
Fig. 2. It consists of a global detector, a local detector, an adaptive search
region, and a detector switcher.

Figure 2: The architecture of our proposed GLAD algorithm. Given an input
image of 1920$\times$1080, we first use the global detector to obtain the
initial position of the target MAV. Then, a Kalman filter-based adaptive
search region is cropped around the center of the last target as the local
search region. The local detector is applied to the local search region for
subsequent MAV detection. Finally, a detector switcher is applied to
adaptively switch between the global and local detectors in order to avoid the
local detector falling into the wrong search region. The solid arrow line
represents the data flow. The dashed arrow line represents signal flow.

Global Detector: The global detector is composed of an appearance-based module
and a motion-based module. First, YOLO is used as an appearance-based
detection module for MAV detection in the full-size image. YOLO is a fast
single-shot object detector based on convolutional neural networks. Our
previous study [13] shows that YOLO can achieve a good balance between
accuracy and speed. In this paper, YOLOv5s is trained with the proposed ARD-
MAV dataset. Detailed information about the dataset is given in Section V-A.To
reduce false alarms as many as possible, a high confidence threshold $T_{0}$
is adopted.

When the appearance-based module fails to detect an MAV under challenging
conditions, the motion-based module is activated. The motion-based module aims
to detect a moving MAV and is composed of motion compensation, moving object
segmentation, and MAV classification. Due to the extra temporal cues, the
motion-based module shows superior performance under challenging conditions.
The procedure of the motion-based detection module is presented in Section IV.
Since the motion-based module is more time-consuming, it will be called only
when the appearance-based module fails. The appearance-based module and the
motion-based module are continuously executed until an MAV is detected.

Local Detector: After the target MAV has been found by the global detector, a
local detector is activated to conduct subsequent MAV detection. Considering
that the target usually does not move too far between two consecutive frames,
a small area around the center of the target is cropped as the search region
for MAV detection. Within the local search region, a local detector is applied
to detect the MAV.

The local detector is composed of an appearance-based module and a local
motion-based module as well. Different from the global detector, the local
detector is trained with cropped images and has different parameters settings.
Since the local search region only focuses on a small patch of the full-size
image, the resolution of the target can be greatly improved compared to the
down-sampling method, and many non-MAV interruptions are removed as well. We
set a low confidence threshold $T_{1}$ to detect as many targets as possible.
Although a low threshold would bring some false targets, confidence ranking
and distance ranking are applied to obtain the most reliable target.
Specifically, the target with the highest confidence score or the closest
distance to the last target is selected as the final target.

The local detector not only improves the detection accuracy but also greatly
improves the computational efficiency. Since the size of the local search
region is much smaller than the size of the original image, the operations
including bilinear interpolation, key points selection, optical flow matching,
frame difference, and object classification in the local detector are much
more efficient. The details about the computational efficiency are introduced
in Section V-F.

Adaptive Search Region: Considering that most of the targets are small, a
fixed size of 300$\times$300 area around the center of the target in the
current frame is usually enough to cover the potential location of the target
in the next frame. Nevertheless, when the detectors fail to detect a target
for multiple frames, a fixed search region might not cover the area of the
potential targets. To improve the robustness of our method when occlusion and
missing detection happen, we design an adaptive search region to dynamically
adjust the size and location of the local search region.

To better predict the position of the local search region, we first use the
Kalman filter to estimate the target position in the next frame. In
particular, we use a Kalman filter to track the relative velocity of the
target rather than tracking the target’s position. We do this because the
position is highly non-linear when the camera moves fast, however, the
relative velocity is continuous in this scenario. The Kalman filter estimates
the target state $x_{t}=(v_{x},v_{y},a_{x},a_{y})$ via a linear difference
equation,

$x_{t}=Mx_{t-1}+w_{t-1},$ (1)

where $(v_{x},v_{y})$ denotes the velocity of the target center,
$(a_{x},a_{y})$ denotes the acceleration of the target center in X-axis and
Y-axis direction, $M\in\mathbb{R}^{4\times 4}$ is the state transition matrix,
$w_{t-1}$ denotes the modeling errors or process noise.

In the prediction step, the target’s velocity is obtained via a dynamic model
expressed as,

$\hat{Z}_{t}=Nx_{t}+v_{t},$ (2)

where $\hat{Z}_{t}$ denotes the predicted measurement,
$N\in\mathbb{R}^{2\times 4}$ is the measurement matrix, and $v_{t}$ is the
measurement noise.

In the updating step, the target state is updated with the actual measure
$Z_{t}$ if the detection is successful. However, if the detection fails, the
Kalman filter will not be updated, and we directly update the target state
with the optimal prediction of the last frame.

After we have obtained the estimated velocity, we estimate the target position
in the current frame with the following equation,

$p_{t}=T(p_{t-1};H)+U_{t-1},$ (3)

where $p_{t}$ denotes the coordinate of the target center, $T$ and $H$ denotes
the 2D perspective transformation and the homography matrix, $U_{t-1}$ is the
velocity of the target center.

Given the estimated target position, we set up a new search region in the next
frame centered at the predicted target position. The size of the search region
(a $L$ $\times$ $L$ square) is decided as following:

$L=300+T_{lost}\times 4,$ (4)

where $T_{lost}$ is the number of frames that detectors fail to detect a
target.

Detector Switcher: Although the proposed local detector can significantly
improve the detection accuracy and the adaptive search region can guide the
local detector to the right search region in most cases, persistent detection
failures may occasionally happen when the background is too complex or the
target is too small to detect. Under these circumstances, the local detector
may concentrate on a wrong search region where the true object is excluded.
Besides, occlusion and abrupt camera movement may also cause the same problem.
Hence, a detector switcher is designed to adaptively coordinate the global and
local detectors.

The switcher adaptively switches between the global detector and the local
detector based on the detection results of the previous frames. Specifically,
when the global and local detectors successfully detect a target MAV, the
detector switcher will switch to or keep on the local detector in the next
frame. When the global detector fails to detect the target MAV, the detector
switcher will continue to execute detection using the global detector in the
full-size image. However, if the local detector fails, the detector switcher
will proceed to execute detection using the local detector in the next frame.
The successive detection in the local search region can help the algorithm
quickly recover from a failure status because the local detector has better
detection accuracy and the target MAV usually does not move too far between
consecutive frames. To avoid the local detector searching in the wrong search
region, the detector switcher will switch to the global detector if the local
detector fails for a while (for example 30 frames).

##  IV Motion-based Detection Module

This section introduces the procedure of the motion-based detection module in
detail. The motion-based detection module is composed of three parts, namely
motion compensation, moving object segmentation, and MAV classification (see
Fig. 3). First, grid-based key points and perspective transformation are used
to compensate for the camera motion. Second, frame difference and
morphological operation are applied to segment moving objects. Third, a
motion-based classifier and an appearance-based classifier are successively
utilized to eliminate non-MAV moving objects and image alignment errors. The
details are given as follows.

Figure 3: The flowchart of the motion-based detection module.

###  IV-A Motion Compensation

To separate the moving objects from the moving background, we must first align
two adjacent frames so that the influence of the camera’s ego-motion can be
eliminated. In this paper, 2D perspective transformation is used for motion
compensation because it exactly models the 2D background motion when the
background results from the relative motion of a 2D plane in the 3D world and
has been widely used for camera motion compensation in aerial view [15, 35,
36]. Perspective transformation requires the background to be planar or the
camera only rotates. In many cases, this is a reasonable approximation since
the pursuing MAV looks at the ground from a high altitude. Hence, most of the
previous works only consider near planar scenes [15, 23]. We noticed that non-
planar scenes containing objects such as high buildings, lampposts, trees, and
wire poles which are quite prevalent in the ARD-MAV dataset can remarkably
influence the motion compensation quality and generate false positives.
Nevertheless, our proposed motion-based classifier and the appearance-based
classifier can remove these interruptions.

For computational efficiency and robustness in textureless regions such as sky
and grassland, grid-based key points are used to calculate the homography
matrix. We sample $30\times 20$ key points uniformly distributed in each row
and column across the previous frame. Then, these key points are tracked by
the pyramidal Lucas-Kanade (LK) algorithm [37] to obtain the corresponding
points in the current frame (one example is shown in Fig. 3(b)). After the key
points are matched over two adjacent frames, the homography matrix $H$ is
calculated with the RANSAC method to reject outliers. The image in the
previous frame $F_{n-1}$ can be aligned with the current frame $F_{n}$ by the
perspective transformation.

$\hat{F}_{n-1}(x,y)=\emph{H}F_{n-1}(x,y).$ (5)

Here, $H$ represents the transformation matrix between $F_{n-1}$ and $F_{n}$,
$\hat{F}_{n-1}$ denotes the motion-compensated previous frame. One example of
the motion-compensated frame is shown in Fig. 3(c).

###  IV-B Moving Object Segmentation

After we have obtained the motion-compensated previous frame, we can highlight
the moving areas with absolute differences between the current frame and the
motion-compensated previous frame. In this paper, frame difference and
morphological operation are used for moving object segmentation.

####  IV-B1 Frame Difference

The frame difference is defined as follows:

$E_{n-1}=|I_{n}(x,y)-\hat{I}_{n-1}(x,y)|,$ (6)

where $I_{n}$ is the gray value of the current frame and $\hat{I}_{n-1}$ is
the gray value of the motion-compensated previous frame.

Next, a threshold $T_{2}$ is applied on $E_{n-1}$ to remove noises and
highlight the silhouette mask of the potential moving foreground. The pixel
values above the threshold $T_{2}$ are set to 255 as foreground, and below the
threshold $T_{2}$ are set to 0 as background. As a result, we obtain the
binarized frame difference $D_{n-1}$. Concerning frame difference, the choice
of threshold $T_{2}$ greatly influences the final result of moving object
segmentation. If $T_{2}$ is too small, the noise will stand out and fill the
image. Conversely, if $T_{2}$ is too big, some parts of the moving foreground
will be wiped out. Moreover, a fixed $T_{2}$ can hardly adapt to the change of
light intensity and moving background. Therefore, we correct the threshold
$T_{2}$ with a light intensity correction term $T_{A}$ and a background motion
correction term $T_{M}$.

$\displaystyle D_{n-1}$ $\displaystyle=\begin{cases}255,&{\text{if
}}E_{n-1}>T_{2}+\alpha T_{A}+\beta T_{M}\\\
{0,}&{\text{otherwise.}}\end{cases}$ (7) $\displaystyle T_{A}$
$\displaystyle=\frac{1}{N_{A}}\sum_{(x,y)\in{A}}|I_{n}(x,y)-I_{n-1}(x,y)|$ (8)
$\displaystyle T_{M}$
$\displaystyle=\frac{1}{N_{S}}\sum_{(x,y)\in{S}}|P_{n}(x,y)-P_{n-1}(x,y)|$ (9)

Here, $A$ represents the whole image pixels, $N_{A}$ represents the total
number of pixels, and $\alpha$ is the light intensity suppress coefficient.
$S$ denotes the set of matched key points, $N_{S}$ represents the total number
of the rectified key points, $P$ is the pixel coordinates of the rectified key
points, and $\beta$ is the background motion correction coefficient.

####  IV-B2 Post Processing

Frame difference method can segment the pixels belonging to moving objects,
but may also contain noises, small holes, and disconnected blobs, which would
cause wrong bounding boxes of the moving objects. Hence, multiple
morphological operations are used to obtain the intact bounding boxes.
Firstly, the morphological open and close operation are used iteratively on
binarized frame difference to eliminate isolated pixels and fill the holes.
Then, connected component analysis is used to obtain the total number of
pixels of each object, and blobs whose area is below 30 pixels are eliminated
because such small objects are usually difficult to recognize in the
subsequent appearance-based classification. Finally, the maximum bounding
rectangle is applied to mark the moving objects and obtain the bounding boxes.
Among these bounding boxes, some adjacent bounding boxes that belong to the
same object may be isolated from each other. Therefore, we merge the bounding
boxes whose distance between another neighboring bounding box is smaller than
$D_{1}$.

###  IV-C MAV Classification

Until now, we have obtained the candidate moving objects. However, there are
still many non-MAV moving objects such as cars, pedestrians, swaying trees,
shimmering water, and image alignment errors (see Fig. 3(d)). Hence, a motion-
based classifier and an appearance-based classifier are successively used to
separate MAV from these interruptions.

####  IV-C1 Motion-based Classifier

In principle, the motion features of MAVs are significantly different from
interruptions such as swaying trees, shimmering water, and image alignment
errors which are usually irregular in moving direction and moving amplitude.
Therefore, most of these interruptions can be filtered by their statistical
features. Assuming that MAV is a non-deformable object, and motion vectors
between two consecutive frames are consistent. We firstly extract Shi-Tomasi
corner points in $k^{th}$ moving object of frame $n$, then define the motion
feature $f^{(k)}_{n}$ as the angle variance of motion vectors in Equation
(10), and $g^{(k)}_{n}$ as the velocity variance of motion vectors in Equation
(12).

$\displaystyle f^{(k)}_{n}$
$\displaystyle=\frac{\sum\limits_{{d_{t}}\in{D^{(k)}_{n}}}({\arctan{{d_{t}}}}-\mu^{(k)}_{n})^{2}}{S^{(k)}_{n}}$
(10) $\displaystyle\mu^{(k)}_{n}$
$\displaystyle=\frac{\sum\limits_{{d_{t}}\in{D^{(k)}_{n}}}{\arctan{{d_{t}}}}}{S^{(k)}_{n}}$
(11) $\displaystyle g^{(k)}_{n}$
$\displaystyle=\frac{\sum\limits_{{d_{t}}\in{D^{(k)}_{n}}}(||{d_{t}}||-\lambda^{(k)}_{n})^{2}}{S^{(k)}_{n}}$
(12) $\displaystyle\lambda^{(k)}_{n}$
$\displaystyle=\frac{\sum\limits_{{d_{t}}\in{D^{(k)}_{n}}}||{d_{t}}||}{S^{(k)}_{n}}$
(13)

Here, $D^{(k)}_{n}$ denotes the set of motion vectors of the $k^{th}$ moving
object of frame $n$ and $S^{(k)}_{n}$ is the number of motion vectors. Given
the above motion features, we build a motion-based classifier to eliminate
these interruptions. We denote $y^{(k)}_{n}$ as a classification label where
the zero value indicates that the $k^{th}$ object is noise, otherwise the
$k^{th}$ object is the candidate MAV. The motion-based classifier is defined
as follows:

$y^{(k)}_{n}=\begin{cases}0,&{\text{if }}{f^{(k)}_{n}>T_{3}}\ or\
{{g^{(k)}_{n}}>T_{4}\ or\ \lambda^{(k)}<T_{5}},\\\
{1,}&{\text{otherwise}}.\end{cases}$ (14)

where $T_{3}$ is the empirical threshold for angle variance, $T_{4}$ is the
empirical threshold for velocity variance, and $T_{5}$ is the empirical
threshold for velocity amplitude.

####  IV-C2 Appearance-based Classifier

After the filtering of the motion-based classifier, the candidate moving
objects still contain some interruptions such as moving cars, pedestrians,
flying birds, and image alignment errors (see Fig. 3(e)). These interruptions
share similar motion features but different appearance features with MAV, we
can use an appearance-based classifier to classify them into MAV and clutter.
Considering that the appearance-based classifier will be called frequently and
most of the common CNN architectures such as ResNet[38] and DenseNet[39] have
a very deep structure that requires large computation resources, a shallow CNN
network (see Fig. 4) is used to extract the feature of MAV and classify
candidate moving objects into MAV and clutter. We train the proposed CNN model
with local images of candidate moving objects (details about the
classification dataset are introduced in Section V-A). Each convolution layer
uses ReLU activation and max pooling. The final layer uses a fully connected
network with a softmax activation to classify the candidate moving objects
into two classes. Up to now, we can obtain the final moving MAV, as shown in
Fig. 3(f).

Figure 4: The architecture of the appearance-based classifier. The input to
the network is a 32$\times$32$\times$3 image. The output represents the image
is an MAV or a clutter.

##  V Experiment

###  V-A Datasets

To evaluate the performance of the proposed GLAD algorithm, we tested our
proposed algorithm on three challenging datasets. Each dataset is briefly
introduced below.

####  V-A1 NPS-Drones dataset[15]

This dataset contains 50 videos of custom delta wing air-frame with a total
number of frames adding up to 70,250. Videos are captured by a GoPro 3 camera
mounted on a custom delta-wing air-frame at HD resolution of 1920$\times$1280
or 1280$\times$960\. Some sample frames are shown in Fig. 5(a). Objects in
this dataset are mainly small drones ranging from 10$\times$8 to
65$\times$21\. The average object size is 0.05$\%$ of the whole image size. We
use the clean version annotations released by [21]. Following the
train/val/test split of [21], we use 40 videos for training and validation and
10 videos for testing.

####  V-A2 Drone-vs-Bird dataset[27]

This dataset is proposed in the International Workshop on Small-Drone
Surveillance, Detection and Counteraction Techniques (WOSDETC), as part of the
16th IEEE International Conference on Advanced Video and Signal based
Surveillance (AVSS). It contains 77 videos of 8 different types of drones with
a total number of frames adding up to 104,760. Many of the videos are recorded
with a static camera but also moving cameras are included. This dataset
exhibits a high variability in difficulty, including sequences with sky or
vegetation as background, different weather conditions, direct sun glare, and
drastic camera movement. Moreover, objects in this dataset are mainly small
drones captured at long distances and often surrounded by small interruptions,
such as birds, and flying insects. Some sample frames are shown in Fig. 5(b).
The average object size is 34$\times$23 (0.1$\%$ of the image size). Due to
some errors and wrong annotations on two videos, we used 60 videos for
training and validation and 15 videos for testing.

(a) Sample frames from NPS-Drones dataset.

(b) Sample frames from Drone-vs-Bird dataset.

Figure 5: Some sample frames from NPS-Drones dataset and Drone-vs-Bird
dataset. Yellow box indicates the target drones.

####  V-A3 ARD-MAV dataset

We created a dataset, named ARD-MAV, which contains 60 video sequences and
106,665 frames. All the videos are taken by the cameras of DJI Mavic2 Pro and
M300 flying at low and medium altitudes. The videos are taken outdoor with
different real-world challenges such as complex backgrounds, non-planar
scenes, occlusion, abrupt camera movement, fast-moving MAVs, and small MAVs
(some examples are shown in Fig. 6(a)). Each video contains only one MAV and
is about one minute long with a 30 FPS frame rate and a resolution of
1920$\times$1080\. All the objects are manually labeled by labelImg. The
object size ranges from 6$\times$3 to 136$\times$75\. The smallest object
refers to an MAV photoed by the aerial camera from more than 150m. The average
object size is only 0.02$\%$ of the image size. As far as we know, this is the
smallest average object size among the existing MAV datasets as shown in Table
I.

TABLE I: Comparison of different MAV datasets Dataset | Count | Max Area | Min Area | Average Area
---|---|---|---|---
NPS-Drones | 70250 | 6.6e-04 | 8.2e-05 | 0.05$\%$
FL-Drones | 38948 | 1.4e-01 | 2.6e-04 | 0.07$\%$
DUT Anti-UAV | 10109 | 7e-01 | 1.9e-06 | 1.3$\%$
Drone-vs-Bird | 104760 | 2.5e-02 | 7.2e-06 | 0.1$\%$
ARD-MAV | 106665 | 3.5e-03 | 1.4e-05 | 0.02$\%$

To train the global and local appearance-based detection module, we use 45
videos for training and validation, and the rest 15 videos for testing. The
training images for the local appearance-based detection module are cropped
around the center of the object with a size of 320$\times$320\. Some examples
of cropped images in the ARD-MAV dataset are shown in Fig. 6(a). Besides, the
position distribution and size distribution are shown in Fig. 6(b).

The training images for the local appearance-based classifier come from the
intermediate results of moving object segmentation. The detection results
after the motion-based classifier contain many non-MAV objects such as cars,
pedestrians, swaying trees, shimmering water, and image alignment errors.
Therefore, we manually divide these candidate moving objects into MAV and
clutter (some examples are shown in Fig. 6(c)). The classification datasets
include 46,268 images of clutter and 17,695 images of MAV. All the images are
resized to 32$\times$32 before training and validation.

(a) Examples of original images and cropped images in ARD-MAV dataset. The
cropped image is displayed at the top right of each image.

(b) MAV’s position distribution and size distribution in each part of ARD-MAV.
The left picture is position distribution and the right is size distribution.

(c) Examples of MAV and clutter. The left picture are examples of MAV, and the
right picture are examples of clutter.

Figure 6: Examples of our ARD-MAV dataset. The global appearance-based
detection module is trained with original images. The local appearance-based
detection module is trained with cropped images. The appearance-based
classifier is trained with local images of MAV and clutter. Yellow box
indicates the target MAVs.

###  V-B Evaluation Metrics and Implementation Details

####  V-B1 Evaluation Metrics

Following the protocol in [21], the performance evaluation is based on
Precision, Recall, F-Score, and AP. We set the intersection over union (IOU)
threshold between predictions and ground truths to 0.5. Therefore, detected
targets matching with ground truth with IOU > 0.5 are counted as true
positives. In particular, the AP is calculated at 0.5 IOU threshold and is
averaged over uniformly spaced 11 points of the precision-recall curve.

####  V-B2 Implementation Details

Our evaluation experiments are implemented on a computer with an NVIDIA
Geforce RTX 3070 GPU. For the training and validation of YOLOv5s, the input
image size is down-sampled to 640$\times$640 as default. We use the Adam
optimizer with a momentum of 0.937 and an initial learning rate of 0.01. We
trained the model for 150 epochs with a batch size of 32. For the appearance-
based classifier, the Adam optimizer is applied with a learning rate of 0.001.
We trained the model for 100 epochs with a batch size of 64.

To test the deployment efficiency of our approach on the mobile platform, we
selected the NVIDIA Jetson Xavier NX processor for deployment experiments.
This device has a 6-core CPU and 8G of GPU memory. We apply the TensorRT
engine and pycuda API to accelerate the inference speed.

Table II gives the parameters settings in this paper. These parameters are
well-designed based on extensive experiments.

TABLE II: Table of parameters settings Notation | Description | Value
---|---|---
$T_{0}$ | Confidence threshold for global detector | 0.5
$T_{1}$ | Confidence threshold for local detector | 0.1
$T_{2}$ | Threshold for frame difference binarization | 5
$T_{3}$ | Threshold for angle variance | 0.8
$T_{4}$ | Threshold for velocity variance | 0.8
$T_{5}$ | Threshold for velocity amplitude | 1
$D_{1}$ | Threshold for bounding box merging | 15

###  V-C Evaluation Results under Different Conditions

We evaluate the proposed GLAD algorithm on 15 videos from the ARD-MAV dataset.
According to the complexity of the background and target size, these videos
are divided into ordinary scenes, complex backgrounds, and small MAVs. As
shown in Table III, our proposed method can achieve a high success rate on
ordinary scenes with nearly 100$\%$ precision and recall. For the cases of
complex background and small MAVs, the detection accuracy degrades especially
for small MAVs. It is important to note that the small MAV here denotes the
MAV with a pixel area smaller than 100, which is difficult even for humans to
recognize in the image. Some examples of detection results are illustrated in
Fig. 7, we see that GLAD could successfully detect the MAV under challenging
conditions.

TABLE III: The detection results of our proposed GLAD algorithm on different
conditions

Conditions | Count | $S_{bbox}{\textsuperscript{1}}$ | Precision | Recall | F-Score | AP
---|---|---|---|---|---|---
Ordinary | 9326 | 726 | 0.99 | 0.96 | 0.97 | 0.91
Complex | 9649 | 265 | 0.94 | 0.86 | 0.90 | 0.81
Small | 9347 | 63 | 0.82 | 0.67 | 0.73 | 0.58
Total | 28322 | 350 | 0.92 | 0.82 | 0.87 | 0.80

  * 1

$S_{bbox}$: Average pixel area of bounding boxes.

(a) Complex Backgrounds

(b) Small MAVs

(c) Ordinary Scenes

Figure 7: Examples of detection results under various challenging conditions.
Blue box indicates the target is detected by the motion-based detection
module, Yellow box indicates the target is detected by the appearance-based
detection module.

###  V-D Comparison with Prior Works

We compare our proposed GLAD algorithm with several state-of-the-art methods
on the ARD-MAV dataset. Specifically, the YOLOv5s-1 and TPH-YOLOv5l-1 uses the
default 640$\times$640 input image size, and YOLOv5s-2 and TPH-YOLOv5l-2 uses
the 1536$\times$1536 input image size for inference. All compared methods are
implemented based on official codes and are fine-tuned using the pre-trained
weights available with public codes.

The quantitative comparison of GLAD with the compared methods on the ARD-MAV
dataset is shown in Table IV. Our proposed algorithm outperforms the existing
methods both on detection accuracy and computational efficiency. It is worth
noting that GLAD significantly outperforms the existing methods on recall
metric.

The quantitative comparison of GLAD with other methods on the NPS-Drones
dataset and Drone-vs-Bird dataset are shown in Table V and Table VI
respectively. The experimental results of the compared methods are from [21]
and [12]. The results demonstrate that our proposed algorithm outperforms the
existing methods on different evaluation metrics, especially on recall metric.

We attribute the better performance of GLAD over compared methods to several
factors. First, GLAD adopts a local search region that greatly retains the
valuable appearance information of small targets and targets under complex
backgrounds. However, the down-sampling method adopted in compared methods
such as YOLOv5, TPH-YOLOv5, and MEGA all lead to huge information loss. The
tremendous improvements on recall metric when enlarging the input image size
of YOLOv5s and TPH-YOLOv5 exactly confirm it. Second, the motion-based
detection module can make good use of the motion features and detect the
target MAV when the appearance features are unreliable under challenging
conditions (some examples are shown in Fig 8). Third, our proposed GLAD
algorithm can detect stationary and moving targets simultaneously because we
use the appearance features and motion features independently on each module.
However, the compared methods such as Dogfight[21] try to detect MAV by
jointly using appearance features and motion cues. As a result, the stationary
target and slow-moving target are hard to detect.

TABLE IV: Quantitative comparison of the GLAD with state-of-the-art methods on ARD-MAV dataset Method | Precision | Recall | F-Score | AP | FPS
---|---|---|---|---|---
YOLOv5s-1 | 0.90 | 0.20 | 0.33 | 0.56 | 149.3
YOLOv5s-2 | 0.78 | 0.41 | 0.54 | 0.61 | 88.5
TPH-YOLOv5l-1 [26] | 0.87 | 0.27 | 0.41 | 0.58 | 51.5
TPH-YOLOv5l-2 [26] | 0.82 | 0.58 | 0.68 | 0.73 | 12.8
Dogfight[21] | 0.54 | 0.27 | 0.36 | 0.22 | 1.0
MEGA[25] | 0.45 | 0.35 | 0.39 | 0.31 | 3.5
GLAD | 0.92 | 0.82 | 0.87 | 0.80 | 146.5
TABLE V: Quantitative comparison of the GLAD with state-of-the-art methods on NPS-Drones dataset Method | Precision | Recall | F-Score | AP
---|---|---|---|---
SCRDet-H[40] | 0.81 | 0.74 | 0.77 | 0.65
SCRDet-R[40] | 0.79 | 0.71 | 0.75 | 0.61
FCOS [41] | 0.88 | 0.84 | 0.86 | 0.83
Mask-RCNN[42] | 0.66 | 0.91 | 0.76 | 0.89
MEGA[25] | 0.88 | 0.82 | 0.85 | 0.83
SLSA[43] | 0.47 | 0.67 | 0.55 | 0.46
Dogfight[21] | 0.92 | 0.91 | 0.92 | 0.89
GLAD | 0.92 | 0.95 | 0.93 | 0.89
TABLE VI: Quantitative comparison of the GLAD with state-of-the-art methods on Drone-vs-Bird dataset Method | Faster-RCNN | SSD512 | YOLOv3 | DETR | GLAD
---|---|---|---|---|---
AP | 0.632 | 0.629 | 0.546 | 0.667 | 0.701
Figure 8: Some examples of the successful detection of GMD when GAD fails to
detect a target.

###  V-E Ablation Studies

In this section, we analyze different components of GLAD to verify their
effectiveness. The experimental results in Table VII show that each component
of our proposed method is important and contributes to the final performance.

TABLE VII: Ablation study of different components of our method Methods | Precision | Recall | F-Score | AP
---|---|---|---|---
GAD | 0.76 | 0.17 | 0.28 | 0.18
GMD | 0.81 | 0.30 | 0.43 | 0.25
GAD+LAD | 0.90 | 0.51 | 0.65 | 0.54
GMD+LMD | 0.89 | 0.30 | 0.45 | 0.34
GAD+GMD+LAD | 0.90 | 0.78 | 0.84 | 0.72
GAD+GMD+LAD+LMD | 0.91 | 0.81 | 0.86 | 0.80
GLAD | 0.92 | 0.82 | 0.87 | 0.80

####  V-E1 Influence of Motion-based Detection Module

The experimental results in Table VII demonstrate that the motion-based
detection module significantly improves detection accuracy, especially on
recall metric compared with simple appearance-based methods. This improvement
on recall metric might be due to the reason that the GMD can spot the subtle
change between consecutive frames. Therefore, it can help generate an initial
position for the local detector when the GAD fails to find an MAV under
complex backgrounds and small MAV conditions. In addition, the LMD can also
help the local detector maintain the right local search region when the LAD
fails to detect a target.

####  V-E2 Influence of Appearance-based Detection Module

The experimental results in Table VII show that the appearance-based detection
module also plays an important role especially when there is a local search
region compared with simple motion-based detection methods. On the one hand,
this is because the motion-based module can only deal with moving targets.
Hence, hovering and slow-moving MAVs are ignored. On the other hand, it owes
to the higher robustness and reliability of the LAD than the LMD. In some
cases, the LMD has difficulties in motion compensation or has low
discernibility towards similar objects. Some failure examples of the local
motion-based module are illustrated in Fig. 9.

(a) Missed detection for hovering MAV

(b) False detection for similar objects in local search region

Figure 9: Failure examples of the motion-based detection module.

####  V-E3 Influence of Local Search Region

In theory, the local search region is conducive to improving the resolution of
targets and removing interruptions. To verify the effectiveness of the local
search region, we tested the performance of the appearance-based module and
the motion-based module with and without a local search region. As shown in
the first and third row of Table VII, the local search region greatly improves
the recall of the appearance-based module from 0.17 to 0.51. This huge
improvement primarily owes to the higher resolution of the target in the local
search region. When down-sampling the image from 1920$\times$1080 to
640$\times$640, the effective pixels of the target are dropped by nearly
80$\%$ and the appearance information is lost. In contrast, since the size of
the local search region is smaller than the input size of YOLOv5s, the
valuable appearance information of the target is completely reserved and the
detection accuracy is greatly improved.

On the other hand, the motion-based module benefits little from the local
search region (as shown in the second and fourth row of Table VII). The
motion-based module primarily relies on the motion cues and local appearance
features to locate and classify the MAV. However, the local appearance
features in the local search region are the same as in the full-size image.
Therefore, it just benefits from fewer interruptions in the local search
region. In addition, a small patch occasionally impairs the motion-based
module by inaccurate image alignment due to the sparse matched key points. As
a result, the improvement of accuracy for the motion-based module is not as
remarkable as the appearance-based module.

We have also tested the influence of the adaptive search region. As shown in
the sixth and seventh row of Table VII, the adaptive search region can improve
the precision and recall. When occlusion and missing detection happen, a
Kalman filter-based tracker can predict the target position in the coming
frame, and an enlarged search region size has a better view than a fixed
search region. Some examples are shown in Fig. 10.

(a) The adaptive search region predicts a right target position when missing
detection happens.

(b) The adaptive search region has a better view than a fixed search region.

Figure 10: Some examples to show the advantage of an adaptive search region.
Yellow box indicates the position of the target MAV, Blue box indicates the
adaptive search region, and White box indicates the fixed search region. The
blue dot indicates the predicted target center.

###  V-F Inference Time

Table VIII demonstrates the frames per second (FPS) of each module on PC and
Jetson Xavier NX respectively. We see that YOLOv5s is greatly accelerated with
the TensorRT engine, it can reach 28.5 FPS on Jetson Xavier NX. The frame rate
of the GMD and LMD is 10.6 FPS and 5.1 FPS on Jetson Xavier NX respectively.

Our method executes different modules based on the detection results of the
previous frames rather than executing all of them sequentially. Since the
detection difficulty varies across different videos, the inference time for
each test video may differ. Therefore, we calculate the average FPS across all
test videos to evaluate the performance of our proposed algorithm. The average
running speed on PC and Jetson Xavier NX is 146.5 FPS and 23.6 FPS
respectively. Although the motion-based detection modules cannot achieve real-
time inference, they are called only when the GAD/LAD fails to detect a target
under challenging conditions. Therefore, the average running speed can reach
nearly real-time.

TABLE VIII: The FPS of each module on PC and Jetson Xavier NX Module | PC | Jetson Xavier NX
---|---|---
GAD/LAD | 149.3 | 28.5
GMD | 41.3 | 5.1
LMD | 183.9 | 10.6
Average FPS on all test videos | 146.5 | 23.6

##  VI Conclusion

In this paper, we proposed a global-local MAV detector for air-to-air
detection of MAVs under challenging conditions. The adaptive search region
adopted in our approach significantly improved the detection accuracy by
improving the resolution of the target using a small cropped image. Besides,
we developed a motion-based detection module, which serves as a good assistant
when the appearance features are unreliable. To evaluate the effectiveness of
the proposed algorithm, we created a new dataset, named ARD-MAV, which
contains various challenging conditions such as complex backgrounds, non-
planar scenes, occlusion, abrupt camera movement, fast-moving MAVs, and small
MAVs. Specifically, this dataset has the smallest average object size among
current MAV detection datasets. Experiments on the three challenging datasets
verified that the proposed algorithm can effectively detect MAV under various
challenging conditions and outperforms state-of-the-art methods. Importantly,
the experiments on the Jetson Xavier NX platform indicated that the proposed
algorithm can be deployed on an aerial platform with real-time running speed.

In the future, our proposed algorithm will be extended to more types of MAVs.
Moreover, an end-to-end network is necessary to be designed to simplify the
training process, reduce the empirical parameters, and more effectively make
use of the motion clues.

## References

  * [1] Y. Tang, Y. Hu, J. Cui, F. Liao, M. Lao, F. Lin, and R. S. Teo, “Vision-aided multi-UAV autonomous flocking in GPS-denied environment,” IEEE Transactions on Industrial Electronics, vol. 66, no. 1, pp. 616–626, 2018.
  * [2] R. Opromolla, G. Fasano, and D. Accardo, “A vision-based approach to UAV detection and tracking in cooperative applications,” Sensors, vol. 18, no. 10, pp. 3391–3417, 2018.
  * [3] M. Vrba and M. Saska, “Marker-less micro aerial vehicle detection and localization using convolutional neural networks,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 2459–2466, 2020.
  * [4] K. R. Sapkota, S. Roelofsen, A. Rozantsev, V. Lepetit, D. Gillet, P. Fua, and A. Martinoli, “Vision-based unmanned aerial vehicle detection and tracking for sense and avoid systems,” in Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1556–1561, 2016.
  * [5] R. Opromolla and G. Fasano, “Visual-based obstacle detection and tracking, and conflict detection for small UAS sense and avoid,” Aerospace Science and Technology, vol. 119, pp. 107167–107186, 2021.
  * [6] J. Zhang, K. Zhang, J. Wang, et al., “A survey on anti-UAV technology and its future trend,” Advances in Aeronautical Science and Engineering, vol. 9, no. 1, pp. 1–8, 2018.
  * [7] E. Unlu, E. Zenou, N. Riviere, and P.-E. Dupouy, “Deep learning-based strategies for the detection and tracking of drones using several cameras,” IPSJ Transactions on Computer Vision and Applications, vol. 11, no. 1, pp. 1–13, 2019.
  * [8] M. Pawełczyk and M. Wojtyra, “Real world object detection dataset for quadcopter unmanned aerial vehicle detection,” IEEE Access, vol. 8, pp. 174394–174409, 2020.
  * [9] Y. Zheng, C. Zheng, X. Zhang, F. Chen, Z. Chen, and S. Zhao, “Detection, localization, and tracking of multiple MAVs with panoramic stereo camera networks,” IEEE Transactions on Automation Science and Engineering, pp. 1–18, 2022.
  * [10] J. Xie, C. Gao, J. Wu, Z. Shi, and J. Chen, “Small low-contrast target detection: Data-driven spatiotemporal feature fusion and implementation,” IEEE Transactions on Cybernetics, vol. 52, no. 11, pp. 11847–11858, 2022\.
  * [11] U. Seidaliyeva, D. Akhmetov, L. Ilipbayeva, and E. T. Matson, “Real-time and accurate drone detection in a video with a static background,” Sensors, vol. 20, no. 14, pp. 3856–3874, 2020.
  * [12] B. K. S. Isaac-Medina, M. Poyser, D. Organisciak, C. G. Willcocks, T. P. Breckon, and H. P. H. Shum, “Unmanned aerial vehicle visual detection and tracking using deep neural networks: A performance benchmark,” in Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pp. 1223–1232, 2021.
  * [13] Y. Zheng, Z. Chen, D. Lv, Z. Li, and S. Zhao, “Air-to-air visual detection of micro-UAVs: An experimental evaluation of deep learning,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 1020–1027, 2021.
  * [14] J. Zhao, J. Zhang, D. Li, and D. Wang, “Vision-based anti-UAV detection and tracking,” IEEE Transactions on Intelligent Transportation Systems, pp. 1–12, 2022.
  * [15] J. Li, D. H. Ye, M. Kolsch, J. P. Wachs, and C. A. Bouman, “Fast and robust UAV to UAV detection and tracking from video,” IEEE Transactions on Emerging Topics in Computing, vol. 10, no. 3, pp. 1519–1531, 2021.
  * [16] J. Wang, G. Zhang, K. Zhang, Y. Zhao, Q. Wang, and X. Li, “Detection of small aerial object using random projection feature with region clustering,” IEEE Transactions on Cybernetics, vol. 52, no. 5, pp. 3957–3970, 2020.
  * [17] L. Du, C. Gao, Q. Feng, C. Wang, and J. Liu, “Small UAV detection in videos from a single moving camera,” in Proceedings of the 2017 CCCV Communications in Computer and Information Science (CCIS), vol. 773, pp. 187–197, 2017.
  * [18] C. Wang, T. Wang, E. Wang, E. Sun, and Z. Luo, “Flying small target detection for anti-UAV based on a gaussian mixture model in a compressive sensing domain,” Sensors, vol. 19, no. 9, pp. 2168–2182, 2019.
  * [19] J. Xie, J. Yu, J. Wu, Z. Shi, and J. Chen, “Adaptive switching spatial-temporal fusion detection for remote flying drones,” IEEE Transactions on Vehicular Technology, vol. 69, pp. 6964–6976, 2020.
  * [20] A. Rozantsev, V. Lepetit, and P. V. Fua, “Detecting flying objects using a single moving camera,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, pp. 879–892, 2017.
  * [21] M. W. Ashraf, W. Sultani, and M. Shah, “Dogfight: Detecting drones from drones videos,” in Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7063–7072, 2021.
  * [22] H. Wang, X. Wang, C. Zhou, W. Meng, and Z. Shi, “Low in resolution, high in precision: UAV detection with super-resolution and motion information extraction,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5, IEEE, 2023.
  * [23] S. Srigrarom and K. H. Chew, “Hybrid motion-based object detection for detecting and tracking of small and fast moving drones,” in Proceedings of the 2020 International Conference on Unmanned Aircraft Systems (ICUAS), pp. 615–621, 2020.
  * [24] P. Zhang, J. Zhao, C. Bo, D. Wang, H. Lu, and X. Yang, “Jointly modeling motion and appearance cues for robust rgb-t tracking,” IEEE Transactions on Image Processing, vol. 30, pp. 3335–3347, 2021.
  * [25] Y. Chen, Y. Cao, H. Hu, and L. Wang, “Memory enhanced global-local aggregation for video object detection,” in Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10334–10343, 2020.
  * [26] X. Zhu, S. Lyu, X. Wang, and Q. Zhao, “TPH-YOLOv5: Improved YOLOv5 based on transformer prediction head for object detection on drone-captured scenarios,” in Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pp. 2778–2788, October 2021\.
  * [27] F. Coluccia, Angelo et al., “Drone-vs-Bird detection challenge at IEEE AVSS2019,” in Proceedings of the 2019 IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1–7, 2019\.
  * [28] G. Fatih, U. Gokturk, S. Erol, and K. Sinan, “Vision-based detection and distance estimation of micro unmanned aerial vehicles,” Sensors, vol. 15, no. 9, pp. 23805–23846, 2015.
  * [29] E. Unlu, E. Zenou, and N. Riviere, “Using shape descriptors for UAV detection,” Electronic Imaging, no. 9, pp. 1–5, 2018.
  * [30] H. Liu, K. Fan, Q. Ouyang, and N. Li, “Real-time small drones detection based on pruned YOLOv4,” Sensors, vol. 21, no. 10, pp. 3374–3390, 2021.
  * [31] C. Rui, G. Youwei, Z. Huafei, and J. Hongyu, “A comprehensive approach for UAV small object detection with simulation-based transfer learning and adaptive fusion,” arXiv preprint arXiv:2109.01800, 2021.
  * [32] A. Schumann, L. Sommer, J. Klatte, T. Schuchert, and J. Beyerer, “Deep cross-domain flying object classification for robust UAV detection,” in Proceedings of the 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2017.
  * [33] J. Li, D. H. Ye, T. Chung, M. Kolsch, J. Wachs, and C. Bouman, “Multi-target detection and tracking from a single camera in unmanned aerial vehicles (UAVs),” in Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4992–4997, 2016.
  * [34] A. Rozantsev, V. Lepetit, and P. Fua, “Flying objects detection from a single moving camera,” in Proceedings of the 2015 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4128–4136, 2015.
  * [35] S. Minaeian, J. Liu, and Y.-J. Son, “Effective and efficient detection of moving targets from a UAV’s camera,” IEEE Transactions on Intelligent Transportation Systems, vol. 19, no. 2, pp. 497–506, 2018.
  * [36] I. Delibasoglu, “UAV images dataset for moving object detection from moving cameras,” arXiv preprint arXiv:2103.11460, 2021.
  * [37] J.-Y. Bouguet et al., “Pyramidal implementation of the affine lucas kanade feature tracker description of the algorithm,” Intel corporation, vol. 5, no. 1-10, p. 4, 2001.
  * [38] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the 2016 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016.
  * [39] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the 2017 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261–2269, 2017.
  * [40] X. Yang, J. Yang, J. Yan, Y. Zhang, T. Zhang, Z. Guo, X. Sun, and K. Fu, “SCRDet: Towards more robust detection for small, cluttered and rotated objects,” in Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8231–8240, 2019.
  * [41] Z. Tian, C. Shen, H. Chen, and T. He, “FCOS: Fully convolutional one-stage object detection,” in Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9626–9635, 2019.
  * [42] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” in Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980–2988, 2017.
  * [43] H. Wu, Y. Chen, N. Wang, and Z.-X. Zhang, “Sequence level semantics aggregation for video object detection,” in Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9216–9224, 2019\.