# Episodic Koopman Learning of Nonlinear Robot Dynamics with Application to
Fast Multirotor Landing

Carl Folkestad∗, Daniel Pastor∗, and Joel W. Burdick *Both authors contributed
equallyAll authors are with the Division of Engineering and Applied Sciences,
California Institute of Technology, Pasadena, CA 91125, USA
<EMAIL_ADDRESS>
###### Abstract

This paper presents a novel episodic method to learn a robot’s nonlinear
dynamics model and an increasingly optimal control sequence for a set of
tasks. The method is based on the Koopman operator approach to nonlinear
dynamical systems analysis, which models the flow of observables in a function
space, rather than a flow in a state space. Practically, this method estimates
a nonlinear diffeomorphism that lifts the dynamics to a higher dimensional
space where they are linear. Efficient Model Predictive Control methods can
then be applied to the lifted model. This approach allows for real time
implementation in on-board hardware, with rigorous incorporation of both input
and state constraints during learning. We demonstrate the method in a real-
time implementation of fast multirotor landing, where the nonlinear ground
effect is learned and used to improve landing speed and quality.

##  I Introduction

While modeling and identification (ID) techniques are well developed for some
robotic mechanisms, such as manipulators, there are an increasing number of
applications where modeling and ID are difficult. Consider a multirotor drone,
whose basic flight mechanics in open air are well understood [1, 2]. However,
when multirotors fly close to the ground or a wall, or inside a narrow tunnel
(e.g., for non-invasive inspection), the unmodeled effects of the complex
vehicle-air-environment interaction can substantially reduce the drone’s path
tracking accuracy, and perhaps its stability. While ground effect models can
be incorporated, their accuracy is limited, and their parameters must still be
estimated in a slow process. This particular problem motivates this paper.
There are many other applications, such as soft robotic structures or robotic
manipulation of soft materials, where first principles modeling and parameter
identification remain challenging in practice. Moreover, one must still design
a controller for the nonlinear mechanics that are identified.

Learning can capture the salient aspects of a robot’s complex mechanics and
environmental interactions. Gaussian process dynamical systems models [3] can
identify nonlinear affine control models in a non-parametric way. Yet,
effective nonlinear control design after identifying the model can be
challenging. Model-free reinforcement learning (MFRL) [4] learns feedback
policies that implicitly incorporate the robot’s dynamics. However, sample
efficiency is very low. Moreover, while safety during MFRL is now possible [5,
6], one cannot yet guarantee that learned policies will satisfy performance
requirements or state and actuator limits.

Figure 1: From left to right: hovering before the sequence start, high speed
descent with learned dynamics, and soft landing.

This paper presents a new method, based on Koopman spectral analysis, to learn
nonlinear robot dynamical models. Conventionally, a system’s behavior is
characterized via its state space flows. In contrast, Koopman-based approaches
study the evolution of observables, which are functions over the state-space.
In this space, the system can be represented by a linear (but possibly
infinite dimensional) operator [7, 8]. Practically, the nonlinear dynamics are
lifted to a higher dimensional space where they are linear. Efficient linear
and optimal control principles can be applied to the lifted system.

Koopman inspired modeling and identification techniques have received
substantial recent attention [9]. The Dynamic Mode Decomposition (DMD) and
extended DMD (EDMD) methods [10] have been successfully used in the field of
fluid mechanics to capture low-dimensional structure in complex flows. More
recently, Koopman-style modeling has been extended to controlled nonlinear
systems [11, 12].

In practice, to model and identify a nonlinear system in the Koopman
framework, one can identify a finite dimensional approximation to the linear
Koopman operator, or identify the Koopman eigenfunctions. We use the latter
approach (see Section II-B). Previous methods for identifying Koopman
eigenfunctions (e.g., [13, 14]) depend upon assumptions that are problematical
for robotic systems: the ID data is gathered while the robot operates under
open loop controls, which can lead to catastrophic system damage.

In very recent work [15], the authors and collaborators have developed a
Koopman Eigenfunction Extended Dynamic Mode Decomposition (KEEDMD) method to
identify/learn an unknown nonlinear dynamical system from a batch of data
gathered while the (robotic) system operates under a stabilizing (but not
necessarily optimal) linear controller.

This paper substantially extends our prior work [15] to make it practically
useful for robotics. First, the method gathers data while the system operates
under any nonlinear stabilizing controller. This enables input vector field
nonlinearities to be captured, unlike prior Koopman-based model ID approaches.
Second, we introduce an episodic learning procedure, by considering the
closed-loop dynamics obtained with a non-linear controller as the autonomous
dynamics for the next episode. This feature increases sample efficiency (i.e.,
fewer learning trials) for improving specific tasks, and enables nonlinear
actuation effects, which are important in robotics, to be captured in the
Koopman eigenfunctions. Third, it should be noted that data collected from
robots executing trajectories formally violates the i.i.d. assumption
underlying the performance guarantees of most learning paradigms. In practice,
this fact can lead to error cascades and poor performance guarantees. Episodic
learning mitigates this problem [16]. Finally, our method integrates Model
Predictive Control (MPC) [17] into its structure, thereby allowing control and
state constraints to be satisfied during the learning process.

Previous approaches have implemented MPC in real time on modest computational
hardware [18], on multirotors using an explicit solution in simulation [19],
and designed feedback linearizing controllers for multirotors in real
experiments [20, 21]. Bouffard et al. [22] also used MPC to learn ground
effects using an experimental multirotor, but used an Extended Kalman Filter
(EKF) in combination with Learning-Based MPC (LBMPC). Shi et al.[23]
experimentally demonstrated using a spectrally-normalized neural network to
learn the ground effect and improve drone landing by designing a feedback
linearizing controller utilizing the learned model. We introduce a new
approach to solve this problem and aim to demonstrate that our method
represents a first step towards practical Koopman-based learning and control
of real-world robotic systems.

Section II reviews relevant facts about the Koopman operator and KEEDMD [15].
Sections III and IV introduce and develop the first main paper
contribution–the episodic KEEDMD algorithm. Section V presents our second
contribution, the experimental demonstration that our method can learn the
ground effect in a fast landing multirotor.

##  II Preliminaries on the Koopman Eigenfunction Extended Dynamic Mode
Decomposition

###  II-A The Koopman Operator

Consider the autonomous dynamical system:

$\mathbf{\dot{x}}=f(\mathbf{x})=A\mathbf{x}+v(\mathbf{x})$ (1)

with the state $x\in\mathcal{X}\subset\mathbb{R}^{n}$ and $f$ Lipschitz
continuous on $\mathcal{X}$. We assume that the system (1) has a fixed point
at the origin, $f(0)=0$. The flow of this dynamical system is denoted by
$S_{t}(x)$ and is defined as

$\frac{d}{dt}S_{t}(x)=f(S_{t}(x))$ (2)

for all $x\in\mathcal{X}$ and all $t\geq 0$. The Koopman operator semi-group
$(U_{t})_{t\geq 0}$, from now on simply denoted as the Koopman operator, is
defined as

$U_{t}g=g\circ S_{t}$ (3)

for all $g\in\mathcal{C}(\mathcal{X})$, where $\circ$ denotes the function
composition and $\mathcal{C}(\mathcal{X})$ is the space of all continuous
functions on $\mathcal{X}$. Each element of the Koopman operator maps
continuous functions to continuous functions,
$U_{t}:\mathcal{C}(\mathcal{X})\rightarrow\mathcal{C}(\mathcal{X})$.
Crucially, each $U_{t}$ is a linear operator. An eigenfunction of the Koopman
operator associated to an eigenvalue $e^{\lambda}\in\mathbb{C}$ is any
function $\phi\in\mathcal{C}(\mathcal{X})$ that defines a coordinate evolving
linearly along the flow of (1) satisfying

$(U_{t}\phi)(x)=\phi(S_{t}(x))=e^{\lambda t}\phi(x).$ (4)

###  II-B Data-driven Construction of Koopman Eigenfunctions

For any sufficiently smooth autonomous dynamical system that is asymptotically
stable to a fixed point, Koopman eigenfunctions can be constructed by finding
the eigenfunctions of the system’s linearization around the fixed point and
then composing them with a diffeomorphism [13]. The linearization of the
dynamics (1) around the origin is

$\mathbf{\dot{y}}=\mathbf{D}f(0)\mathbf{y}=\hat{A}\mathbf{y},\text{
}\mathbf{y}\in\mathcal{Y}$ (5)

The following proposition describes how to construct eigenfunction-eigenvalue
pairs for the linearized system (5).

######  Proposition 1.

Let $\hat{A}$ denote the linearization (5) of nonlinear system (1) and let
$\\{\mathbf{v}_{1},\dots,\mathbf{v}_{n}\\}$ be a basis of the eigenvectors of
$\hat{A}$ corresponding to nonzero eigenvalues
$\\{\lambda_{1},\dots,\lambda_{n}\\}$. Let
$\\{\mathbf{w}_{1},\dots,\mathbf{w}_{n}\\}$ be an adjoint basis to
$\\{\mathbf{v}_{1},\dots,\mathbf{v}_{n}\\}$ such that
$\langle\mathbf{v}_{q},\mathbf{w}_{r}\rangle=\delta_{qr}$ and $\mathbf{w}_{q}$
is an eigenvector of $\hat{A}^{*}$ at eigenvalue $\bar{\lambda}_{q}$. Then,
the linear functional

$\psi_{q}(\mathbf{y})=\langle\mathbf{y},\mathbf{w}_{q}\rangle$ (6)

is a nonzero eigenfunction of $U_{\hat{A}}$, the Koopman operator associated
to $\hat{A}$. Furthermore, for any $(m_{1},\dots,m_{d})\in\mathbb{N}_{0}^{d}$

$\bigg{(}\prod_{q=1}^{d}e^{m_{q}\lambda_{q}},\prod_{q=1}^{d}\psi_{q}^{m_{q}}\bigg{)}$
(7)

is an eigenpair of the Koopman operator $U_{\hat{A}}.$

These linear functionals (6), termed principal eigenfunctions, are used to
construct the eigenfunctions associated with the Koopman operator of the
nonlinear dynamics through the use of a conjugacy map (See [9], Prop. 7).

######  Proposition 2.

Assume that the nonlinear system (1) is topologically conjugate to the
linearized system (5) via the diffeomorphism
$h:\mathcal{X}\rightarrow\mathcal{Y}$. Let $B\in\mathcal{X}$ be a simply
connected, bounded, positively invariant open set in $\mathcal{X}$ such that
$h(B)\subset Q_{r}\subset\mathcal{Y}$, where $Q_{r}$ is a cube in
$\mathcal{Y}$. Scaling $Q_{r}$ to the unit cube $Q_{1}$ via the smooth
diffeomorphism $g:Q_{r}\rightarrow Q_{1}$ gives $(g\circ h)(B)\subset Q_{1}$.
Then, if $\psi$ is an eigenfunction for $U_{\hat{A}}$ at $e^{\lambda}$, then
$\psi\circ g\circ h$ is an eigenfunction for $U_{f}$ at eigenvalue
$e^{\lambda}$, where $U_{f}$ is the Koopman operator associated with the
nonlinear dynamics (1).

An extension of the Hartman-Grobman theorem ([7], Theorem 2.3) guarantees the
existence of a $\mathcal{C}^{1}$ diffeomorphism

$y=c(\mathbf{x})=\mathbf{x}+h(\mathbf{x})$ (8)

between the linearized and nonlinear systems in the entire basin of attraction
of a fixed point, such that $\mathbf{D}c(\mathbf{0})=I$.

###  II-C KEEDMD with Trajectory-tracking Control Laws

Algorithm 1 Data-driven Koopman Eigenpair Construction
---
Require: Data set
$\mathcal{D}=\big{(}(\mathbf{x}_{k}^{j},\mathbf{u}_{k}^{j})_{k=0}^{M_{s}}\big{)}_{j=1}^{M_{t}}$,
nominal model matrices $A_{nom}$, $B_{nom}$, nominal control gains $K_{nom}$,
desired trajectory $\boldsymbol{\tau}(t)$, number of lifting functions $N$,
$N$ power combinations
$(m_{1}^{(i)},\dots,m_{d}^{(i)})\in\mathbb{N}_{0}^{d},i=1,\dots,N$
Construct principal eigenpairs for the linearized dynamics:
$(e^{\lambda}_{j},\psi_{j}(\mathbf{y}))\leftarrow{}(e^{\lambda}_{j},\langle\mathbf{y},\mathbf{w}_{j}\rangle),\qquad
j=1,\dots,n$
Construct $N$ eigenpairs from the principal eigenpairs:
$(\tilde{e}^{\lambda}_{i},\tilde{\psi}_{i})\leftarrow\big{(}\prod_{j=1}^{d}e^{m_{j}^{(i)}\lambda_{j}},\prod_{j=1}^{d}\psi_{j}^{m_{j}^{(i)}}\big{)},\quad
i=1,\dots,N$
Fit diffeomorphism estimator:
$h(\mathbf{y})\leftarrow\text{ERM}(\mathcal{H}_{h},\mathcal{L}_{h},\mathcal{D})$
Construct scaling function: $g(\mathbf{y})\leftarrow
g:\mathcal{Q}_{r}\rightarrow\mathcal{Q}_{1}$
Construct $N$ eigenpairs for the nonlinear dynamics:
$(\tilde{e}^{\lambda}_{i},\phi_{i})\leftarrow(\tilde{e}^{\lambda}_{i},\tilde{\psi}_{i}(g(h(\mathbf{y})))),\qquad
i=1,\dots,N$
Output:
$\Lambda=\text{diag}(\tilde{e}^{\lambda}_{1},\dots,\tilde{e}^{\lambda}_{N}),\qquad\boldsymbol{\phi}=[\phi_{1},\dots,\phi_{N}]^{T}$

Data-driven construction of Koopman eigenfunctions for nonlinear dynamics of
the form $\dot{\mathbf{x}}=a(\mathbf{x})+B\mathbf{u}$ is based on Section
II-B, and summarized in Algorithm 1. General nonlinear dynamics will be
considered in Section III. For trajectory-tracking control laws, the algorithm
assumes that a linearized nominal dynamics model
$\mathbf{\dot{y}}=A_{nom}\mathbf{y}+B_{nom}\mathbf{u}$ is known, along with a
nominal stabilizing feedback control law
$\mathbf{u}(t)=K_{nom}(\mathbf{y}(t)-\boldsymbol{\tau}(t))$, where
$\boldsymbol{\tau}(t)\in\mathcal{X}$ is the trajectory we want the system to
track. Typically, $A_{nom},B_{nom}$ is the linearization of the dynamics
around the origin and $K_{nom}$ the controller gains determined by e.g. LQR on
the nominal state space model. Principal eigenfunctions, with eigenvalues
$e^{\lambda}_{j}$, of the Koopman operator for the closed loop linearized
dynamics $\mathbf{\dot{y}}=(A_{nom}+B_{nom}K_{nom})\mathbf{y}$ are constructed
using Prop. 1: $\psi_{j}(\mathbf{y})=\langle\mathbf{y},\mathbf{w}\rangle$,
where $\mathbf{w}$ is an adjoint basis of the eigenvectors of
$A_{cl}=(A_{nom}+B_{nom}K_{nom})$. Products and powers (7) generate
arbitrarily many eigenpairs of the linearized system before applying the
diffeomorphism (8) between the nonlinear and linearized dynamics. This
diffeomorphism $h:\mathcal{X}\rightarrow\mathcal{Y}$ is learned in a
supervised way (e.g. a neural network trained with gradient descent) by
performing empirical risk minimization (ERM) of an appropriate loss function
over a model class $\mathcal{H}_{h}$. For the trajectory-tracking case, the
loss function is of the form

$\displaystyle\begin{split}&\mathcal{L}_{h}(\mathbf{x},\mathbf{\dot{x}},A_{cl}\mathbf{x}-\mathbf{\dot{x}},\boldsymbol{\tau}(t))=\\\
&\quad\,||\dot{h}(\mathbf{x})-A_{cl}h(\mathbf{x})-(A_{cl}\mathbf{x}-\mathbf{\dot{x}})+B_{nom}K_{nom}\boldsymbol{\tau}||^{2}\\\
\end{split}$ (9)

(see [15] for details). Finally, a function scaling
$\mathcal{Y}\subset\mathcal{Q}_{r}$ into $Q_{1}$, where $\mathcal{Q}_{r}$ is a
hyper cube of the same dimension as $\mathcal{Y}$ with radius $r$, is
constructed (i.e. by scaling each coordinate into a unit cube) and approximate
Koopman eigenpairs for the unknown, nonlinear dynamics are constructed:
$(e^{\lambda_{i}},\phi_{i})=(\tilde{e}^{\lambda}_{i},\tilde{\psi}_{i}(g(h(\mathbf{y}))))$,
where $\tilde{e}^{\lambda}_{i},\tilde{\psi}_{i}$ are the eigenpairs
constructed with (7).

KEEDMD uses the constructed eigenfunctions to lift the system states to a
higher dimensional space where a linear dynamical model of the form
$\mathbf{\dot{z}}=A\mathbf{z}+B\mathbf{u}$ can be identified. For Lagrangian
dynamics, as the example in Section V, we use
$\mathbf{z}=[\mathbf{x},\boldsymbol{\phi}(\mathbf{x})]^{T}$ as the lifted
state, where $\mathbf{x}=[\mathbf{p}\quad\mathbf{v}]^{T}$, p the position,
$\dot{\boldsymbol{p}}=\textbf{v}$ the velocity, and $\boldsymbol{\phi}$ is a
vector of the eigenfunctions. While helping data efficiency, the method does
not generally require any a priori information of the structure of the
dynamics. Since the time evolution of the eigenfunctions is dictated by their
eigenvalues, we can show that the lifted state space model has the structure

$\begin{bmatrix}\mathbf{\dot{p}}\\\ \mathbf{\dot{v}}\\\
\boldsymbol{\dot{\phi}}\bigg{(}\\!\\!\begin{bmatrix}\mathbf{p}\\\
\mathbf{v}\end{bmatrix}\\!\\!\bigg{)}\end{bmatrix}=\underbrace{\begin{bmatrix}\begin{array}[]{cc}0&\quad
I\end{array}{}&0\\\
\begin{array}[]{cc}A_{\mathbf{v}\mathbf{p}}&A_{\mathbf{v}\mathbf{v}}\end{array}{}&A_{\mathbf{v}\boldsymbol{\phi}}\\\
-B_{\boldsymbol{\phi}}K_{nom}&\begin{array}[]{c}\\\
\end{array}{}\Lambda\begin{array}[]{c}\\\
\end{array}{}\end{bmatrix}}_{A}\begin{bmatrix}\mathbf{p}\\\ \mathbf{v}\\\
\boldsymbol{\phi}\bigg{(}\\!\\!\begin{bmatrix}\mathbf{p}\\\
\mathbf{v}\end{bmatrix}\\!\\!\bigg{)}\end{bmatrix}+\underbrace{\begin{bmatrix}B_{\mathbf{p}}\\\
B_{\mathbf{v}}\\\ B_{\boldsymbol{\phi}}\begin{array}[]{c}\\\
\end{array}{}\end{bmatrix}}_{B}\mathbf{u}$

where $0,I,\Lambda,K_{nom}$ are fixed matrices and
$A_{\mathbf{v}\mathbf{p}},A_{\mathbf{v}\mathbf{v}}$,
$A_{\mathbf{v}\boldsymbol{\phi}}$, $B_{\mathbf{p}}$, $B_{\mathbf{v}}$,
$B_{\boldsymbol{\phi}}$ are determined from data, and the term
$-B_{\boldsymbol{\phi}}K_{nom}$ accounts for the nominal controller’s state
feedback effect on the evolution of the eigenfunctions. The desired trajectory
effect of the controller is captured by the learning framework. The unknown
elements of $A$ and $B$ can be found using linear regression on the data
collected under the nominal control law (see [15] for details). We define
$\mathcal{L}_{z}$ as the mean squared error for the regression problem, where
different forms of regularization can be included if needed. As the feedback
controller is state-dependent, it is not possible to disambiguate its effect
from the passive uncontrolled system dynamics in the learning process. To
avoid an ill-conditioned KEEDMD regression problem, Brownian noise is added to
perturb the nominal controller [24]. Brownian noise is chosen in this instance
because pure sampling from e.g. a Gaussian distribution leads to perturbations
that have too high frequency to perturb the movement of the multirotor. This
perturbation is also used by our episodic learning framework (Section IV).
When the lifted state space model is identified, state estimates can be
obtained as $\mathbf{x}=C\mathbf{z}$, where $C=[I\quad 0]$. $C$ is denoted the
projection matrix of the lifted state space model.

##  III Episodic KEEDMD learning

Figure 2: Flow chart showing the different elements for each episode.

###  III-A Problem Setup and Dynamics Modeling

Assume that we have selected a fixed trajectory $\boldsymbol{\tau}$ to be
tracked by the robot during episodic learning. Further assume a nominal
controller $\mathbf{\hat{u}}(\mathbf{x},\boldsymbol{\tau},t)$ that can
stabilize the system to $\boldsymbol{\tau}$ within a region of attraction
$\Omega$ around the trajectory. This controller might be the outcome of a
previous learning episode (see below), or the simple linear nominal controller
from KEEDMD (Section II). Finally, the system’s governing dynamics are assumed
to be unknown

$\mathbf{\dot{x}}=f(\mathbf{x},\mathbf{u})$ (10)

where
$\mathbf{x}\in\mathcal{X}\subset\mathbb{R}^{n},u\in\mathcal{U}\subset\mathbb{R}^{m}$.
and $f(\mathbf{x},\mathbf{u})$ is assumed to be Lipschitz continuous on
$\mathcal{X}\times\mathcal{U}$.

###  III-B Learning with Arbitrary Stabilizing Control Laws

KEEDMD (Section II) requires batch training data to be collected from system
executions under a nominal linear control law:
$\mathbf{u}_{nom}(\mathbf{x})=K_{nom}(\mathbf{x}-\boldsymbol{\tau}(t))$. A
main contribution of this paper is to iteratively learn an improving sequence
of eigenfunctions and nonlinear controllers. Specifically, we will iteratively
use the lifted state-space model to design an MPC-controller to track learning
trajectories (see Figure 2).

If a candidate nonlinear controller
$\mathbf{\hat{u}}(\mathbf{x},\boldsymbol{\tau},t)$ can stabilize system (10)
to a given trajectory $\boldsymbol{\tau}$, the controlled system can be
described by the autonomous dynamics

$\mathbf{\dot{x}}=f(\mathbf{x},\mathbf{\hat{u}}(\mathbf{x},\boldsymbol{\tau},t))=F_{\mathbf{\hat{u}},\boldsymbol{\tau}}(\mathbf{x},t).$
(11)

Importantly, for the autonomous dynamics (11), there exists an associated
Koopman operator $U_{F_{\mathbf{\hat{u}},\boldsymbol{\tau}}}$ that depends on
control law $\mathbf{\hat{u}}$ and trajectory $\boldsymbol{\tau}$. Therefore,
approximate eigenpairs for $U_{F_{\mathbf{\hat{u}},\boldsymbol{\tau}}}$ can be
constructed (see Section II-B) from the gathered state and control samples. A
lifted state-space model can be constructed from these eigenpairs.

However, unlike the framework reviewed in Section II, we aim to learn a
dynamical model that assumes that the system is already regulated by the
nominal controller $\mathbf{\hat{u}}(\mathbf{x},\boldsymbol{\tau},t)$. As a
result, the $A$-matrix of the lifted state space model captures the autonomous
dynamics under the nominal control law (Eq. 11), and the $B$-matrix captures
the effect of control variations around the nominal controller:

$\mathbf{\dot{\hat{z}}}=A\mathbf{\hat{z}}+B(\mathbf{u}(\mathbf{x},\boldsymbol{\tau},t)-\mathbf{\hat{u}}(\mathbf{x},\boldsymbol{\tau},t)).$
(12)

This model is used in an MPC framework below to design an augmenting control
law that adds optimal control actions to the nominal controller. The
augmenting controller leverages the improved system model to make corrections
to sub-optimal actions taken by the nominal controller.

###  III-C Modifications to Allow the Diffeomorphism to Capture Nonlinear
Control and Dynamics Effects

To enable the learning framework to capture nonlinear effects caused by the
nonlinear controller and actuated dynamics, a minor modification to the
function approximator of $h$ is necessary. Namely, since the diffeomorphism is
affected by the forcing signal $\boldsymbol{\tau}(t)$ it must be included in
the inputs of $h$. This is motivated by the form of the diffeomorphism loss
function (9). In the case considered in the preliminaries however, the
actuated dynamics and controller are assumed to be linear. This causes the
effect of the forcing signal $\boldsymbol{\tau}(t)$ to cancel out such that
the diffeomorphism is independent of the desired trajectory. In the general
nonlinear case however, the effect is not canceled out and must be captured by
the diffeomorphism. As a result, the diffeomorphism is modified such that
$h:\mathcal{X}\times\mathcal{X}\rightarrow\mathcal{Y}$ (see [15] for details).

##  IV Episodic Eigenfunction Construction and KEEDMD Inference

This section describes the main contribution of this paper, a substantial
extension of the KEEDMD framework to allow iterative learning and improvement
of the lifted state-space model and its associated controller.

###  IV-A Overview of the Episodic Learning Algorithm

Algorithm 2 summarizes our episodic learning approach, which applies three key
steps per episode. In each episode, $e$, the first key step starts when an
initial condition is sampled from set $X_{0}$ and an experiment is executed
with the controller that results from the previous episode
$\mathbf{u}_{e-1}(\mathbf{x},\boldsymbol{\tau},t)$. The state $\mathbf{x}$,
control actions $\mathbf{u}_{e-1}$, Brownian noise control perturbations
$\mathbf{\tilde{u}}$, and the desired position dictated by the trajectory at
the time associated with the $i$-th sample $\boldsymbol{\tau}_{i}$ are
sampled. State data can be differentiated numerically to find estimates
$\mathbf{\dot{x}}$. The resulting data set is:

$\mathcal{D}_{x}^{(e)}=\left\\{\left(\mathbf{x}_{i}^{(e)},\mathbf{u}_{i}^{(e)},\mathbf{\tilde{u}}_{i}^{(e)},\boldsymbol{\tau}_{i}\right),\mathbf{\dot{x}}_{i}^{(e)}\right\\}_{i=1}^{T_{s}}$
(13)

where $\mathbf{x}_{i}^{(e)}$ denotes the $i$-th timestep of the $e$-th episode
and $T_{s}$ denotes the number of samples in the episode. From
$\mathcal{D}_{x}^{(e)}$ we estimate the diffeomorphism $h$ and construct the
eigenfunctions $\boldsymbol{\phi}^{(e)}(\mathbf{x})$ with associated
eigenvalues $\Lambda^{(e)}$, via Algorithm 1. Since changes in the control law
between episodes are expected to be small, we warm start the learning
algorithm with model coefficients from the previous episode.

Algorithm 2 Episodic KEEDMD
---
Require: Desired trajectory $\boldsymbol{\tau}$, nominal controller
$\mathbf{\hat{u}}(\mathbf{x},\boldsymbol{\tau},t)$, diffeomorphism model class
$\mathcal{H}_{h}$, diffeomorphism loss $\mathcal{L}_{h}$, number of lifting
functions $N$, KEEDMD loss $\mathcal{L}_{z}$
$\mathcal{D}_{z}=\emptyset,\quad\mathbf{u}_{0}(\mathbf{x},\boldsymbol{\tau},t)=\mathbf{\hat{u}}(\mathbf{x},\boldsymbol{\tau},t)$
for $e=1,\dots,N_{ep}$ do
Sample initial condition: $\mathbf{x_{0}}\leftarrow\text{sample}(X_{0})$
Execute experiment:
$\mathcal{D}_{x}^{(e)}\leftarrow\text{run}(\mathbf{x}_{0},\mathbf{u}^{(e-1)}(\mathbf{x},\boldsymbol{\tau},t))$
Fit diffeomorphism estimator:
$h(\mathbf{x})\leftarrow\text{ERM}(\mathcal{H}_{h},\mathcal{L}_{h},\mathcal{D}_{x}^{(e)})$
Construct eigenpairs:
$(\boldsymbol{\phi}^{(e)}(\mathbf{x}),\Lambda^{(e)})\leftarrow
h(g(\boldsymbol{\psi}(\mathbf{x})))$
Construct and aggregate lifted data set:
$\mathcal{D}_{z}\leftarrow\mathcal{D}_{z}\cup\mathcal{D}_{z}^{(e)}$
Fit KEEDMD model:
$\mathbf{\dot{z}}^{(e)}(\mathbf{z})\leftarrow\text{ERM}((\boldsymbol{\phi}^{(e)}\\!,\\!\Lambda^{(e)}),\mathcal{L}_{z},\mathcal{D}_{z})$
Update controller:
$\mathbf{u}^{(e)}\leftarrow\mathbf{u}^{(e-1)}+w^{(e)}\text{MPC}(\mathbf{\dot{z}}^{(e)},\mathbf{u}^{(e-1)})$
end for
Output: Final control law $\mathbf{u}^{(N_{ep})}$

(a)

(b)

(c)

(d)

Figure 3: Evolution of drone altitude $p_{z}$ with accumulated error and
control effort after each episode. Episode 0: baseline controller, Episode
1-3: performance after each episode of learning. Red arrows: duration the
thrust constraint is active.

The second key step is to use the constructed eigenpairs to build a lifted
data set $\mathcal{D}_{z}^{(e)}$.

$\mathcal{D}_{z}^{(e)}=\left\\{\left(\mathbf{z}_{i}^{(e)},\mathbf{u}_{i}^{(e)},\mathbf{\tilde{u}}_{i}^{(e)},\boldsymbol{\tau}_{i}\right),\mathbf{\dot{z}}_{i}^{(e)}\right\\}_{i=1}^{T_{s}}$
(14)

which is the same data as $\mathcal{D}_{x}^{(e)}$, but with the state and its
derivative, $\mathbf{x}_{i}^{(e)},\mathbf{\dot{x}}_{i}^{(e)}$, replaced with
the lifted state and its derivative, $\mathbf{z}_{i}^{(e)}$,
$\mathbf{\dot{z}}_{i}^{(e)}$. Next, data from the current and previous
episodes is aggregated: $\bigcup_{j=1}^{e}\mathcal{D}_{z}^{(j)}$. The lifted
state-space model is constructed from this data using the framework of Section
III-B. This results in a model of the form (12).

In the third and final step, an augmenting MPC is designed (see Section IV-B)
for the lifted state-space model. The evaluation of the previous iteration’s
controllers is necessitated by the fact that the eigenfunctions depend on the
dynamics under closed loop control with the controller deployed in the
previous episodes. The controller augmentations are weighted and added to the
previous episode’s control law:
$\mathbf{u}_{e}=\mathbf{u}_{0}+\sum_{j=1}^{e}w_{j}\mathbf{u}_{j}$, where
$w_{e}$ is a weighting factor indicating the confidence in the augmenting
controller. The weighting factors can be any monotonically increasing sequence
on the interval $[0,1]$ which allows the augmenting controller to have a
bigger impact after a sufficiently rich data set has been collected.

###  IV-B Model Predictive Controller Details

Inspired by [25], we transform the original non-linear optimization problem
into an efficient quadratic program (QP). The QP formulation requires us to
discretize the previously learned linear continuous dynamics. We assume a
known objective function of states and controls only. For simplicity, we use a
quadratic objective function with respect to the state error and control
action, but other objective functions can be used by simply adding it to the
lifting functions. We assume known control bounds
$u_{\min},u_{\max}\in\mathbb{R}^{m}$ and state bounds
$x_{\min},x_{\max}\in\mathbb{R}^{n}$. Because the control input for each MPC
problem refers to the change from the the previous controller, we have to
correct for this change in the control bounds. All these assumptions define
the following optimization problem that is solved at each time step:

$\begin{array}[]{ll}{\underset{\begin{subarray}{c}u\in\mathbb{R}^{m\times
N_{p}}\\\ z\in\mathbb{R}^{N\times
N_{p}}\end{subarray}}{\operatorname{min}}}&{\sum_{p=1}^{N_{p}}\left[\left(C_{j}z_{p}-\tau_{p}\right)^{T}Q\left(C_{j}z_{p}-\tau_{p}\right)+u_{p}^{T}Ru_{p}\right]}\\\
{\text{s.t.}}&{z_{p}=A_{j}z_{p-1}+B_{j}u_{p}}\\\ {}&{x_{\min}\leq
C_{j}z_{p}\leq x_{\max}}\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,p=1,\dots,N_{p}\\\
{}&{u_{\min}\leq u_{p}-\sum_{i=1}^{j-1}w^{(i)}u_{p}^{(i)}\leq u_{\max}}\\\
{}&{z_{0}=\phi_{j}(x_{k})}\end{array}$ (15)

where $Q\in\mathbb{R}^{n\times n}$ and $R\in\mathbb{R}^{m\times m}$ are
positive semidefinite cost matrices, $\tau\in\mathbb{R}^{n\times N_{p}}$ is
the reference trajectory, $A_{j}\in\mathbb{R}^{N\times N}$ and
$B_{j}\in\mathbb{R}^{N\times m}$ are the discrete time versions of (12) for
controller $j$, $C_{j}\in\mathbb{R}^{n\times N}$ is the $j^{th}$ controller’s
projection matrix, and $\phi_{j}\in\mathbb{R}^{N}$ are the $j^{th}$
controller’s eigenfunctions. See Figure 2 to see how each controller is used
as more episodes are being executed. In addition, we add a smoothing
regularizer to avoid chatter that may arise from optimization-based
controllers [26] of the form $\sum_{p=1}^{N_{p}}\alpha_{R}(u_{p}-u_{p-1})^{2}$
where $u_{0}$ is the deployed control action at the previous timestep.

##  V Improving Fast multirotor Descent and Landing by Learning the Ground
Effect

To validate our methodology, we apply it to fast descent and landing of a
multirotor111The code for learning and control is publicly available on
https://github.com/Cafolkes/keedmd. As the vehicle approaches the landing
plane, a ground effect from the interaction of the prop downwash and the
landing surface becomes prominent. This effect induces added upward thrust on
the drone, which can lead to poor tracking performance for control designs
that rely on models which omit these fluid flow interactions.

###  V-A Modeling and Problem Statement

To simplify the discussion, we consider a 1-dimensional nominal model of the
multirotor’s altitude dynamics, consisting of a point mass model having
altitude and its derivative, $[p_{z},\dot{p}_{z}]^{T}$, as states, mass $m$,
and total thrust, $T$, as input:

$\begin{bmatrix}{}\dot{p}_{z}\\\
\ddot{p}_{z}\end{bmatrix}=\begin{bmatrix}0&1\\\
0&0\end{bmatrix}\begin{bmatrix}p_{z}\\\
\dot{p_{z}}\end{bmatrix}+\begin{bmatrix}0\\\ 1/m\end{bmatrix}T.$ (16)

Using this model we design a nominal MPC as described in Section IV-B with the
goal of reaching a fixed point of $0.05\text{\,}\mathrm{m}$ above ground at
zero velocity.

A nominal MPC stabilizes the drone to a fixed point, but uses more control
effort and time to reach that point as a result of its simplified model.
Importantly, the nominal dynamics model does not capture the ground effect.
Our goal is to iteratively learn a better dynamics model (and associated MPC)
that will improve speed and tracking performance in both the air and near-
ground regimes.

###  V-B Implementation and Experimental Details

Our experiments use the Intel Aero RTF Drone. Drone position is measured using
an OptiTrack motion capture system and is fused with the drone’s IMU (stock
PX4 v1.8) to estimate the state. The diffeomorphism, $h$, is parameterized by
a neural network and implemented with PyTorch [27], and the KEEDMD regression
is implemented with elastic net regularization in Scikit-learn [28]. A dense
form MPC-controller is implemented in Python using the QP solver OSQP [29],
and commands are sent to the PX4 flight controller via ROS. All computation
for learning and control is done on board the drone. Each neural network and
MPC evaluation takes 5 ms, limiting us to 5 episodes as update rates below 60
hz lead to poor performance on our hardware. The experiment’s key parameters
are summarized in Table I.

TABLE I: Experiment Parameters State error penalty, $Q$ | $[10,0.1]$ |  Min thrust, $u_{\min}$ | 0.3
---|---|---|---
Control penalty, $R$ | 1 |  Max thrust, $u_{\max}$ | 0.8
Min altitude, $x_{\min}$ | $0.05\text{\,}\mathrm{m}$ |  Hover thrust, $u_{\text{hover}}$ | 0.66

We execute Algorithm 2 as discussed in Section IV on the drone for three
episodes in each campaign. Each episode starts with 3 repetitions of the
following: (1) the drone takes off and moves to an initial point under PX4
control; (2) the lifted controller takes over to stabilize the fixed point and
hovers at that point for a second. After 3 repetitions, the drone lands under
lifted control, fits the diffeomorphism and KEEDMD models, and repeats the
episode. An additional landing sequence is executed to evaluate the
performance of the current episode controller.

###  V-C Results and Discussion

Figure 3 depicts the drone’s trajectory and control effort under the nominal
controller (Episode 0), and then final landing for three episodes of a single
learning campaign. Episode 0 represents the nominal performance before
learning, while episodes 1-3 show the learning effect. Tracking error is
reduced by 19.3 percent by the end of the last episode while the total control
effort increases 4.5 percent as a consequence of the chosen MPC penalty
matrices. Importantly, the thrust constraint is rigorously satisfied, and this
constraint is active for longer duration. As the system learns more accurate
dynamic models, it relies more on the open-loop bang-bang characteristic, as
would be expected from an optimal solution, and less from closed loop control.
Less control effort is needed towards the end of the trajectory, indicating
that our methodology captures the ground effect. The mean and standard
deviation of five independent learning campaigns are reported in Fig. 4. The
tracking performance improves in every episode. Furthermore, the methodology
has low variance between campaigns.

Figure 4: Mean $\pm$ 1 standard deviation of tracking performance after each
episode over 5 independent campaigns.

##  VI Conclusions and Future Work

This paper presented a novel episodic method based on a Koopman eigenfunction
framework to learn a robotic system’s nonlinear dynamics, and learn a near
optimal control strategy for given tasks. By using a Koopman approach, we are
able to implement a real-time MPC framework for optimal system control during
the learning process. The approach improves performance as it gathers more
data, augmenting the controller to avoid constant actuation matrix
limitations, while respecting state and control input bounds. A current
limitation is the addition of a controller in each episode leading to
prohibitive computational complexity as the number of episodes grows. Current
work is addressing this by consolidating the controllers into a finite set,
and by optimally choosing when to switch to the next episode, reducing the
number of episodes needed.

## Acknowledgement

This work has been supported in part by Raytheon Company and the DARPA
Physics-infused AI program. The first author is grateful for the support of
Aker Scholarship Foundation. The authors would like to thank Igor Mezic, Ryan
Mohr, and Maria Fonoberova for helpful discussions.

## References

  * [1] G. M. Hoffmann, H. Huang, S. Waslander, and C. Tomlin, “Quadrotor Helicopter Flight Dynamics and Control: Theory and Experiment,” in Proc. AAIA Guidance and Control Conf., 2007.
  * [2] M. Bangura and R. Mahoney, “Nonlinear Dynamic Modeling for High Performance Control of a Quadrotor,” in Proc. Australasian Conf. on Robotics and Automation, 2012.
  * [3] J. M. Wang, D. J. Fleet, and A. Hertzmann, “Gaussian Process Dynamical Systems,” in Proc. Neural Information Processing Sytems, 2006.
  * [4] Y. Duan, X. Chen, J. Schulman, and P. Abbeel, “Benchmarking Deep Reinforcement Learning for Continuous Control,” arXiv, 2016.
  * [5] J. García and F. Fernández, “A Comprehensive Survey on Safe Reinforcement Learning,” Journal of Machine Learning Research, 2015.
  * [6] R. Cheng, G. Orosz, R. M. Murray, and J. W. Burdick, “End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks,” in Proc. AAAI, 2019.
  * [7] Y. Lan and I. Mezić, “Linearization in the large of nonlinear systems and Koopman operator spectrum,” Physica D, vol. 242, no. 1, pp. 42–53, 2013.
  * [8] A. Mauroy and J. Goncalves, “Linear identification of nonlinear systems: A lifting technique based on the Koopman operator,” in 2016 IEEE 55th Conference on Decision and Control (CDC), pp. 6500–6505, IEEE, 2016.
  * [9] M. Budišić, R. Mohr, and I. Mezić, “Applied Koopmanism,” Chaos, vol. 22, no. 4, 2012.
  * [10] K. Taira, S. L. Brunton, S. T. M. Dawson, C. W. Rowley, T. Colonius, B. J. Mckeon, O. T. Schmidt, S. Gordeyev, V. Theofilis, and L. S. Ukeiley, “Modal Analysis of Fluid Flows: An Overview,” AIAA Journal, vol. 55, no. 12, pp. 4013–4041, 2017.
  * [11] E. Kaiser, J. N. Kutz, and S. L. Brunton, “Data-driven discovery of Koopman eigenfunctions for control,” no. April, 2017.
  * [12] J. L. Proctor, S. L. Brunton, and J. Nathan Kutz, “Generalizing Koopman Theory to Allow for Inputs and Control *,” vol. 17, no. 1, pp. 909–930, 2018\.
  * [13] R. Mohr and I. Mezić, “Construction of eigenfunctions for scalar-type operators via Laplace averages with connections to the Koopman operator,” pp. 1–25, 2014.
  * [14] M. Korda and I. Mezić, “Learning Koopman eigenfunctions for prediction and control: the transient case,” pp. 1–32, 2018.
  * [15] C. Folkestad, D. Pastor, I. Mezic, R. Mohr, M. Fonoberova, and J. Burdick, “Extended Dynamic Mode Decomposition with Learned Koopman Eigenfunctions for Prediction and Control,” in Proc. American Control Conf., Sep. (submitted) 2020.
  * [16] S. Ross, G. J. Gordon, J. A. Bagnell, and M. Learning, “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning,” Journal of Machine Learning Research, vol. 15, pp. 627–635, 2011.
  * [17] D. Q. Mayne, J. B. Rawlings, C. V. Rao, and P. O. Scokaert, “Constrained model predictive control: Stability and optimality,” Automatica, vol. 36, no. 6, pp. 789–814, 2000.
  * [18] M. N. Zeilinger, Real-time Model Predictive Control.  PhD thesis, ETH Zürich, 2011.
  * [19] C. Liu, H. Lu, and W.-H. Chen, “An explicit mpc for quadrotor trajectory tracking,” in 2015 34th Chinese Control Conference (CCC), pp. 4055–4060, IEEE, 2015.
  * [20] M. Bangura and R. Mahony, “Real-time model predictive control for quadrotors,” IFAC Proceedings Volumes, vol. 47, no. 3, pp. 11773–11780, 2014.
  * [21] M. Abdolhosseini, Y. Zhang, and C. A. Rabbath, “An efficient model predictive control scheme for an unmanned quadrotor helicopter,” Journal of intelligent & robotic systems, vol. 70, no. 1-4, pp. 27–38, 2013.
  * [22] P. Bouffard, A. Aswani, and C. Tomlin, “Learning-based model predictive control on a quadrotor: Onboard implementation and experimental results,” in 2012 IEEE International Conference on Robotics and Automation, pp. 279–284, IEEE, 2012.
  * [23] G. Shi, X. Shi, M. O’Connell, R. Yu, K. Azizzadenesheli, A. Anandkumar, Y. Yue, and S.-J. Chung, “Neural Lander: Stable Drone Landing Control using Learned Dynamics,” International Conference on Robotics and Automation (ICRA), pp. 9784–9790, 2019.
  * [24] E. Kaiser, J. N. Kutz, and S. L. Brunton, “Sparse identification of nonlinear dynamics for model predictive control in the low-data limit,” Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, vol. 474, no. 2219, 2018.
  * [25] M. Korda and I. Mezić, “Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control,” Automatica, vol. 93, pp. 149–160, 2018.
  * [26] B. J. Morris, M. J. Powell, and A. D. Ames, “Continuity and smoothness properties of nonlinear optimization-based feedback controllers,” Proceedings of the IEEE Conference on Decision and Control, vol. 54rd IEEE, no. Cdc, pp. 151–158, 2015.
  * [27] A. Pazske, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, D. A. Lin, Zeming, L. Antiga, and A. Lerer, “Automatic Differentiation in PyTorch,” tech. rep., 2017.
  * [28] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and . Duchesnay, “Scikit-learn: Machine Learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.
  * [29] B. Stellato, G. Banjac, P. Goulart, A. Bemporad, and S. Boyd, “OSQP: An Operator Splitting Solver for Quadratic Programs,” 2018 UKACC 12th International Conference on Control, CONTROL 2018, no. 1, p. 339, 2018.


# Towards Better Object Detection in Scale Variation with
Adaptive Feature Selection

Zehui Gong<EMAIL_ADDRESS>Dong Li  Guangdong University of Technology
<EMAIL_ADDRESS>
###### Abstract

It is a common practice to exploit pyramidal feature representation to tackle
the problem of scale variation in object instances. However, most of them
still predict the objects in a certain range of scales based solely or mainly
on a single-level representation, yielding inferior detection performance. To
this end, we propose a novel adaptive feature selection module (AFSM), to
automatically learn the way to fuse multi-level representations in the channel
dimension, in a data-driven manner. It significantly improves the performance
of the detectors that have a feature pyramid structure, while introducing
nearly free inference overhead. Moreover, a class-aware sampling mechanism
(CASM) is proposed to tackle the class imbalance problem, by re-weighting the
sampling ratio to each of the training images, based on the statistical
characteristics of each class. This is crucial to improve the performance of
the minor classes. Experimental results demonstrate the effectiveness of the
proposed method, with 83.04% mAP at 15.96 FPS on the VOC dataset, and 39.48%
AP on the VisDrone-DET validation subset, respectively, outperforming other
state-of-the-art detectors considerably. The code is available at
https://github.com/ZeHuiGong/AFSM.git.

###### keywords:

object detection , scale variation , multi-scale fusion

††journal: Nuclear Physics B

##  1 Introduction

In recent years, the performance of object detection consecutively improves,
thanks to the carefully designed deep convolutional neural networks (CNNs) [1,
2, 3, 4] and high-quality large-scale benchmarks [5, 6]. However, these
advanced detectors [7, 8, 9] are still struggling to tackle the wide range of
variation in object scale, in real-world applications (see Fig. 1). To solve
this problem, recent researches focus mainly on two possible strategies: image
pyramid [10] and feature pyramid [11, 12, 13].

As for the image pyramid, a set of resized versions for one input image are
sequentially sent to the detector at the testing phase. Therefore, the object
scale will fall into an appropriate range at a specific image pyramid.
However, this will increase the computational cost considerably, making it
unsuitable for real-world applications, especially for mobile devices.
Conversely, another line of research constructs the feature pyramid based on
the inherent multi-scale feature maps on the backbone network, which is memory
efficient and consumes less computation. SSD [11] directly uses the multi-
scale feature maps generated by the bottom-up pathway. FPN [12] constructs the
feature pyramid by sequentially fusing high-level semantic information into
low-level feature maps, with an additional top-down pathway and lateral
connection.

(a) large objects

(b) small objects

Figure 1: The size of objects varies significantly, posing a great challenge
for object detection algorithms. In (a), the large object almost occupies 80%
of an image, while in (b), the small ones can hardly be clearly captured
without zooming.

Although FPN and other similar works [14, 15, 16, 17] achieve satisfactory
performance, they still suffer from the following limitation. Specifically,
each object in a specific range of scale is predicted based solely or mainly
on the feature from a single level, while discarding the information in other
levels, resulting in losing the chance to capture richer information.
Generally, High-level features contain richer context and semantic
information, with large receptive fields. This may be useful for small object
prediction because the context information can be an additional cue for
inferring the class of the small objects. Similarly, low-level features are
spatially finer and, contain more details for high localization accuracy.
Allowing large objects to access them is undoubtedly beneficial. It is hence
achieving inferior detection performance to predict object instance in a
certain range of size, based solely on single-level features. Some models
attempt to solve this problem by enabling multi-level feature fusion. ASFF
[18] fuses multi-level features via spatially point-wise addition. SFAM [19]
re-weights the concatenated features at each scale with a SE module [20].

Another challenge for robust object detection is the problem of class
imbalance, i.e., the number of objects varies significantly from class to
class, resulting in two sets of classes, the major and minor classes. This
posts great challenges for the detection algorithms, since the training
process can be dominated by the major classes, leading to rarely supervisory
signals for the minor ones and consequently, harming the overall performance.
Therefore, it is crucial to devise an effective mechanism to tackle the class
imbalance problem.

In this paper, we propose a novel and effective adaptive feature selection
module (AFSM), to tackle the problem of significant variation in object scale.
The proposed method learns to adaptively select useful information from multi-
level representations in the channel dimension. Specifically, for the feature
maps at a certain level, we first resize the feature maps from other levels
into the same resolution. Then, the multi-level features are integrated by the
corresponding selection weight (a learnable vector). Therefore, at each
channel dimension, features at each level have a corresponding importance
weight, thus automatically enhancing the discriminative clues beneficial to
the following detection. The proposed approach is agnostic to the detector,
i.e., irrespective of one-stage or two-stage, and it can be applied to the
detectors that have multi-level representations. Moreover, we propose a class-
aware sampling mechanism (CASM) to solve the problem of class imbalance, which
automatically assigns the sampling weight to each of the training images,
according to the number of objects in each class. CASM increase the occurrence
frequency of the minor class objects, resulting in more supervisory signals
for these objects, hence improving the performance of minor classes
considerably, as shown in Section 4.5.

The main contributions of this paper are summarized as follows:

  * 1.

We propose an adaptive feature selection module to adaptively select useful
information across multi-scale representations, which enhances the ability of
the model to cope with the variation in object scales.

  * 2.

Noticing that the problem of class imbalance restricts the performance of the
detector, we propose a class-aware sampling mechanism to tackle the class
imbalance problem, which can automatically assign the sampling weight to each
of the images during training, according to the number of objects in each
class.

  * 3.

Comprehensive experiments conducted on the VisDrone and PASCAL-VOC datasets
confirm the effectiveness of our proposed method, which achieves state-of-the-
art accuracy compared to other detectors, while still running in real-time.

In Section 2, we review the related works of object detection and feature
pyramid structure. The details of the proposed algorithm are illustrated in
Section 3. And we describe the implementation details and the experimental
results in Section 4. Finally, a brief conclusion of our work is given in
Section 5.

##  2 Related Work

Object detection Formally, the modern object detection methods can be
categorized into two types: one-stage pipeline [8, 11, 9] and two-stage
pipeline [21, 7, 22]. One-stage detectors achieve real-time performance by
outputting the detection results directly without proposal generation
procedure. Specifically, YOLO [8] regresses bounding boxes and class
probabilities in a single efficient backbone network. SSD [11] covers objects
of various sizes by adjusting the output bounding boxes on the default boxes
over multiple feature maps. RetinaNet [9] settles the extreme foreground-
background class imbalance, thus achieving better performance than two-stage
object detectors. Alternatively, two-stage detectors obtain higher accuracy by
combining the region proposal stage and classification stage. Specifically,
Faster R-CNN [7] introduces a region proposal network (RPN), which
simultaneously predicts bounding boxes and objectness scores to achieved
further speeds-up. Double-Head RCNN [22] handles the classification in a fully
connected head while regressing the bounding in a convolution head. Recently,
some researches [23, 24, 25, 26] provide a different view of objects, by
detecting keypoints of the bounding box.

Feature pyramid Feature pyramid representations, i.e., multi-level feature
maps, is a basic solution to tackle the variation in object scale. Skip
connection [27] and hyper-column [28] are the early attempts to integrate
multi-level feature to generate more representative feature maps. After that,
multi-scale prediction [11, 12, 29, 30] begins to emerge, because of the
lightweight property, higher accuracy, and faster inference compared to image
pyramids. SSD [11] predicts class scores and bounding boxes respectively at
multiple feature levels. FPN [12] introduces a top-down pathway, and a lateral
connection to sequentially merge high-level semantic information into the low-
level features, building multi-level representations for the following
detection head. PANet [29] adds an extra bottom-up pathway on the basis of FPN
to shorten the distance between high-level and low-level features. In BiFPN
[30], multiple PANet-like structures with scale-wise level re-weighting is
proposed, to fuse feature maps from different scales. Another line of works
[31, 32] employs the Neural Architecture Search (NAS) algorithm to
automatically find the powerful architecture for multi-level representations,
improving the performance considerably.

Selection of multi-scale features Besides the generation of multi-level
features, some of the researches [18, 19, 33] make a step forward and, delve
into the mechanism of how to select useful information across various feature
levels. In [33], a learnable gating unit is proposed to control the flow of
information in single-level skip connection, in the U-Net architecture for
visual counting. Zhao et al. [19] propose a scale-wise feature aggregation
module (SFAM) to execute channel-wise attention on the concatenated multi-
scale feature maps, where the attention weights are generated by the squeeze
and excitation (SE) module [20]. In [18], the point-wise attention weights are
used for multi-level feature fusion, where a $1\times 1$ conv is employed to
generate the weight at each level, following by a softmax layer for
normalizing.

The spatially point-wise addition proposed in [18] is not effective enough for
multi-level feature selection and, introduces more computational cost (i.e.,
several $1\times 1$ convs for generating fusion weights). In this paper, by
regarding the selection weight at each level as a learnable vector for
channel-wise multi-level feature fusion, AFSM can automatically learn the
distribution of object size over the whole dataset (as showed in Section 4.5),
which makes the selection process more stable and effective. It seems that
SFAM [19] shares the same spirit of our work. However, we emphasize that SFAM
only executes channel-wise attention at a single level, on the concatenated
feature maps, instead of adaptively selecting useful information across
various levels.

##  3 The Proposed Method

Figure 2: The overall framework of our proposed method. The backbone network
is followed by the feature pyramid network (FPN) [12] for multi-scale feature
generation. Then, the adaptive feature selection module is employed to
adaptively select the most useful information for the subsequent detection
task. Finally, the model outputs the center locations and the corresponding
offsets, as well as the object sizes, to form the final bounding boxes.
$\odot$ is the broadcast element-wise multiplication, and $\oplus$ means the
element-wise addition. For simplicity, we show the procedure of multi-scale
feature fusion in a certain level only.

The overall framework of our approach is shown in Fig. 2. In this section, we
first introduce the baseline network architecture of our proposed method,
which is based on the effective anchor-free detector, i.e., CenterNet [25].
Then, we move on to push the original baseline to a stronger version, by
employing the recently advanced tricks, such as mixup [34] and global context
block (GCB) [35], and by proposing two effective techniques, class aware
sampling mechanism and modified GIOU [36] loss. Finally, we describe our
proposed adaptive feature selection module in detail, which adaptively selects
features from multi-scale representations in the channel dimension.

###  3.1 Network Architecture

We adopt CenterNet [25] as our baseline method, because of its simplicity and
high efficiency. CenterNet presents a new representation for detecting
objects, in terms of the center location of a bounding box. Other object
properties, e.g., object size, and center offset, are regressed directly using
the image features from the center locations. Let
$(ct_{x},ct_{y},w,h,\epsilon_{x},\epsilon_{y})$ be one output that generates
from the model. Then, we construct the bounding box $\mathcal{B}\in R^{4}$ by:

$\mathcal{B}=\left\\{\begin{array}[]{c}x_{1}=ct_{x}+\epsilon_{x}-w/2,\\\
y_{1}=ct_{y}+\epsilon_{y}-h/2,\\\ x_{2}=ct_{x}+\epsilon_{x}+w/2,\\\
y_{2}=ct_{y}+\epsilon_{y}+h/2.\end{array}\right.$ (1)

where $(ct_{x},ct_{y})$ is the center location of an object and,
$(\epsilon_{x},\epsilon_{y})$ is the offset prediction used to slightly adjust
the center point. $(w,h)$ is the object size prediction and, $(x_{1},y_{1})$
and $(x_{2},y_{2})$ are the top-left and bottom-right corners of the bounding
box. The overall training objective of CenterNet is given as follows:

$L=L_{ct}+\lambda_{size}L_{size}+\lambda_{off}L_{off}$ (2)

where $L_{ct}$ is the loss for center location estimation, which is a variant
of the focal loss [9]. $L_{off}$ is the L1 loss for center offset regression.
$L_{size}$ is a modified version of the GIOU loss that fits in the object size
regression of CenterNet, and we will describe it in detail in section 3.2.
$\lambda_{size}$ and $\lambda_{off}$ are hyper-parameters, which are all set
to 1.0 in our experiments, unless otherwise specified.

Figure 3: The analysis of category statistics on the VisDrone-DET training
set. Left: The total number of object instances for each category varies
significantly from class to class. Right: The ratio of images contains the
corresponding class. The ratio of images containing ‘awning-tricycle’ is less
than 20%

###  3.2 Stronger Baseline

CenterNet mainly contains two components: a backbone network (usually the
Hourglass-104 [37]) and a detection head with three parallel branches to
predict the center location, offset, and object size (i.e., $w$ and $h$) of a
bounding box. In this paper, we employ ResNet [1] as the backbone network and,
use feature pyramid network (FPN) to generate multi-scale feature maps.
Moreover, we employ the recently advanced training tricks [34, 35], and
propose two effective approaches, to further improve the performance of the
baseline method.

Advanced techniques In [38], mixup algorithm [34] is demonstrated to be an
effective data augmentation method, to improve the robustness of the model by
a stochastic linear interpolation between two training examples. Let $I$ be
one input image, and
$B=\mathrm{\\{}x_{1}^{i},y_{1}^{i},x_{2}^{i},y_{2}^{i},c^{i}\mathrm{\\}}_{i=1}^{n}$
denote the corresponding bounding box annotation of that image, and $n$ is the
number of the boxes. Then, the mixup operation can be formulated as follows:

$\lambda_{1}\sim\mathrm{Beta}(\eta,\eta),$ (3)
$\lambda_{2}=\mathrm{min}(\mathrm{max}(0,\lambda_{1}),1),$ (4)
$I=\lambda_{2}I_{1}+(1-\lambda_{2})I_{2},$ (5)
$B=\mathrm{concact}(B_{1},B_{2}).$ (6)

Algorithm 1 Detailed procedures of class-aware sampling mechanism

0: The list of class labels for each of the training images,
$G=\\{G_{i}\\}_{i=1}^{N}$, where $G_{i}=\\{c_{j}\\}_{j=1}^{n}$, $N$ is the
number of training images, $n$ the number of objects in each image. The number
of training classes, $K$

0: The list of sampling ratios for each image, $S=\\{s_{1},...,s_{N}\\}$, used
for training sampling.

1: $S\leftarrow\emptyset$, $M\leftarrow\emptyset$

2: for each $k\in[1,K]$ do

3: $M(k)\leftarrow 0$ $//$ Initialize the number of objects in each class to
0.

4: end for

5: for $G_{i}\in G$ do

6: $//$ Counting the number of object instance $t$ in class $k$, in image $i$

7: for $k$, $t\in$ counter($G_{i}$) do

8: $M(k)\leftarrow M(k)+t$

9: end for

10: end for

11: for each $i\in[1,N]$ do

12: $s\leftarrow 0$

13: $//$ Enumerate the set of unique classes in image $i$

14: for $k\in\mathrm{unique}(G_{i})$ do

15: $s\leftarrow s+\frac{1}{M(k)}$

16: end for

17: $S\leftarrow S~{}\cup~{}\\{s\\}$

18: end for

19: $//$ Normalize the sampling ratios

20: $S\leftarrow S/\mathrm{sum}(S)$

21: return $S$

Where $\mathrm{concact}(\cdot)$ means the concatenation operation of the
bounding box annotation between two mixup images.

Besides, the long-range pixels in the image may be still correlated. For
example, an object is less likely to be a car when the background is the sea.
Therefore, it is important to model the long-range relations between pixels.
We adopt the global context block (GCB) [35], which is an effective module to
model the global context and, is much lightweight compared to NLNet [39].

Class-aware training sampling In real-world applications, imbalanced data is
ubiquitous, where the number of an object instance in each class varies
significantly (see Fig. 3). This posts a great challenge for the detector,
since the training process can be dominated by the major classes, leading to
rarely supervisory signals for the minor ones. To address this issue, we
propose a novel re-weighting mechanism, i.e., class-aware sampling mechanism
(CASM), to automatically assigns the sampling ratio for each image in the
training set, instead of the uniform sampling. Alg. 1 shows the detailed
procedures of CASM, where the sampling ratio for each of the images is
computed automatically, according to the occurrence frequency of each class in
the dataset.

Modified GIOU CenterNet uses L1 loss for object size regression, which is
sensitive to the variation in object size and, separately regresses the width
and height of an object, hence overlooking the correlation between properties
of the same object. Conversely, GIOU loss [36] considers the optimization as a
whole, and is irrelevant to the object size. However, GIOU loss is first
proposed for the anchor-based methods, which can not be applied to the
CenterNet directly. Therefore, we make some modifications to the GIOU loss.
The entire workflow of computing the modified GIOU (M-GIOU) loss is
illustrated in Alg. 2, where an additional mask $F\in\\{0,1\\}^{N}$ is
introduced to filter out the background object during back-propagation.

Algorithm 2 Workflow of the modified GIOU loss

0: $O^{p}=\\{r_{i}^{p},...,r_{N}^{p}\\}$,
$O^{g}=\\{r_{i}^{g},...,r_{N}^{g}\\}$, and $F\in\\{0,1\\}^{N}$
$O^{p}=\\{r_{i}^{p},...,r_{N}^{p}\\}$ is the set of predicted object size
$O^{g}=\\{r_{i}^{g},...,r_{N}^{g}\\}$ is the set of ground-truth object size
$r_{i}^{p}=(w_{i}^{p},h_{i}^{p})$, $r_{i}^{g}=(w_{i}^{g},h_{i}^{g})$ $N$ is
the maximum number of training objects in one image, e.g., $N=128$ $F$ is a
mask that each element indicates the object is foreground or background.

0: $L_{size}$

1: initialize $L_{size}\leftarrow 0$.

2: for each $i\in[1,N]$ do

3: $w_{i}^{p}\leftarrow\mathrm{exp}(w_{i}^{p})$,
$h_{i}^{p}\leftarrow\mathrm{exp}(h_{i}^{p})$.

4: Getting intersection box size:
$t_{w}\leftarrow\mathrm{min}(w_{i}^{p},w_{i}^{g})$,
$t_{h}\leftarrow\mathrm{min}(h_{i}^{p},h_{i}^{g})$.

5: Getting the smallest enclosing box size:
$c_{w}\leftarrow\mathrm{max}(w_{i}^{p},w_{i}^{g})$,
$c_{h}\leftarrow\mathrm{max}(h_{i}^{p},h_{i}^{g})$.

6: Calcalating area of $r_{i}^{p}$: $A^{p}\leftarrow w_{i}^{p}\times
h_{i}^{p}$.

7: Calculating area of $r_{i}^{g}$: $A^{g}\leftarrow w_{i}^{g}\times
h_{i}^{g}$.

8: Calculating area of smallest enclosing box: $A^{c}\leftarrow c_{w}\times
c_{h}$.

9: Calculating area of intersection box: $A^{t}\leftarrow t_{w}\times t_{h}$.

10: $IoU\leftarrow\frac{A^{t}}{\mathcal{U}}$, where
$\mathcal{U}=(A^{p}+A^{g}-A^{t})$.

11: $GIoU\leftarrow IoU-\frac{A^{c}-\mathcal{U}}{A^{c}}$.

12: $//$ Multiply by the mask $F(i)$, where the loss for background object is
0.

13: $L_{size}\leftarrow L_{size}+(1-GIoU)\times F(i)$

14: end for

15: return $L_{size}$.

###  3.3 Adaptive Feature Selection Module

The object scales vary significantly in real-world scenarios, posing great
challenges for detection algorithms. It is hence vital to develop an approach,
to cope with a wide range of variation in object scales. Therefore, we propose
an adaptive feature selection module (AFSM) to adaptively learn the importance
weights, for fusing feature maps from different feature levels. As shown in
Fig. 2, the pipeline is consists of two steps: feature resizing and adaptive
feature selection.

Feature resizing. Formally, a list of multi-scale features is given as
$X=\\{X^{1},...,X^{l},...\\}$, where $X^{l}$ is the feature maps at level $l$,
whose resolution is $1/2^{l}$ of the input image. For feature maps at level
$l$, we first resize it with a scale factor of $2^{|l-k|}$, to the same
resolution of the feature maps at level $k$, where $k$ is the target level,
e.g., $k=3$. As the output stride is of $4$ in CenterNet, we resize the
feature maps after feature selection, to ensure the resolution is of $1/4$ of
the input image. Specifically, the nearest interpolation operation is adopted
to resize the features respectively at each scale.

Adaptive feature selection. Let
$\hat{X}=\\{\hat{X}^{1},...,\hat{X}^{l},...\\}$ be the feature maps after
resizing. We propose to fuse features across various resolutions as follows:

$Y_{i}=\sum_{l}\hat{X}_{i}^{l}\cdot\alpha_{i}^{l},$ (7)

where $Y_{i}$ implies the $i$-th channel of the output feature maps among
spatial positions. $\alpha_{i}^{l}$ refers to the channel importance weight
that controls the contribution of feature maps at level $l$ to the prediction
of the bounding box. Note that $\alpha_{i}^{l}$ is a simple scalar variable,
sharing across all the spatial positions. Softmax operation is employed to
ensure $\sum_{l}\alpha_{i}^{l}=1$, as follows:

w/o blur w/ blur mixup

Figure 4: Visualization of the images that use random blurring and mixup data
augmentation. By applying these techniques, the distribution of the training
samples is significantly expanded, thus leading to stronger generalization
ability of the model.

$\alpha_{i}^{l}=\frac{\mathrm{exp}(\beta_{i}^{l})}{\sum_{k}\mathrm{exp}(\beta_{i}^{k})}$
(8)

where $\beta_{i}^{l}$ is a learnable scalar variable, which is initialized to
one and, can be adaptively learned by the network through standard back-
propagation.

With our proposed feature selection module, the features at different levels
are aggregated adaptively. The output $Y$ is then used for the subsequent
object detection following the same pipeline of CenterNet.

##  4 Experiments

Table 1: The Comparison with state-of-the-arts on the VisDrone-DET validation subset. All models are trained on VisDrone-DET train set. ${\dagger}$ implies that the multi-scale inference is employed. Method | Backbone | AP | AP50 | AP75 | AR1 | AR10 | AR100 | AR500
---|---|---|---|---|---|---|---|---
Cascade R-CNN [21] | ResNet50 | 24.10 | 42.90 | 23.60 | 0.40 | 2.30 | 21.00 | 35.20
Faster R-CNN [7] | ResNet50 | 23.50 | 43.70 | 22.20 | 0.34 | 2.20 | 18.30 | 35.70
RetinaNet [9] | ResNet50 | 15.10 | 27.70 | 14.30 | 0.17 | 1.30 | 24.60 | 25.80
FCOS [41] | ResNet50 | 16.60 | 28.80 | 16.70 | 0.38 | 2.20 | 24.40 | 24.40
HFEA [42] | ResNeXt101 | 27.10 | - | - |  |  |  |
HFEA [42] | ResNeXt152 | 30.30 | - | - |  |  |  |
ClusDet [43] | ResNeXt101 | 32.40 | 56.20 | 31.60 |  |  |  |
CPEN+FPEN [44] | Hourglass104 | 36.70 | 53.20 | 39.50 |  |  |  |
DMNet+DSHNet [45] | ResNet50 | 30.30 | 51.80 | 30.90 |  |  |  |
DSOD [46] | ResNet50 | 28.80 | 47.10 | 29.30 |  |  |  |
HRDNet [47] | ResNet18+101 | 28.33 | 49.25 | 28.16 | 0.47 | 3.34 | 36.91 | 36.91
HRDNet${\dagger}$ [47] | ResNeXt50+101 | 35.51 | 62.00 | 35.13 | 0.39 | 3.38 | 30.91 | 46.62
Ours | CBResNet50 | 33.95 | 60.46 | 32.69 | 0.66 | 7.25 | 40.28 | 49.84
Ours${\dagger}$ | CBResNet50 | 37.62 | 65.41 | 37.06 | 0.62 | 7.21 | 43.21 | 56.26
Ours | CBResNet50+DCN | 35.43 | 61.88 | 34.60 | 0.80 | 7.77 | 41.77 | 51.48
Ours${\dagger}$ | CBResNet50+DCN | 39.48 | 66.98 | 39.45 | 0.82 | 7.94 | 44.76 | 58.46

###  4.1 Datasets

We select two challenging datasets to validate the effectiveness of our
proposed algorithm, VisDrone [40] and PASCAL-VOC [6].

VisDrone VisDrone is a very challenging dataset, with a wide range of
variation in scale, lighting condition, scenario, viewpoint, and blurring.
This dataset contains a total of 10209 images for object detection on drones,
which are captured by various drone platforms, and from 14 different cities in
China, including various scenarios. Ten object categories, e.g., car and van,
are considered for evaluation. Specifically, 10209 images are split into four
subsets, including 6471 images in the training subset, 548 in the validation
subset, 1610 in the test-dev subset, and 1580 in the test-challenge subset.

PASCAL-VOC VOC is a popular dataset for object detection. The union set of
2007 and 2012 train&val is used for training, including 16551 images. The
evaluation is performed on the VOC 2007 test set, with 4962 testing images,
and a total of 20 categories.

###  4.2 Evaluation Metrics

For the VisDrone dataset, we use the same evaluation protocol proposed in
[40], which is almost the same as COCO [5], except for reporting an additional
$\mathrm{AR^{max=500}}$ score. The mean average precision (mAP) over
thresholds of $\mathrm{IoU}=0.5:0.05:0.95$ is reported. Moreover, we report
the $\mathrm{AP^{IoU=0.5}}$, $\mathrm{AP^{IoU=0.75}}$, $\mathrm{AR^{max=1}}$,
$\mathrm{AR^{max=10}}$, and $\mathrm{AR^{max=100}}$ scores. For the VOC
dataset, we use mAP at IOU threshold 0.5 as the evaluation metric.

Table 2: Comparison results of various algorithms on the VisDrone-DET test-challenge subset. ${\dagger}$ implies that the multi-scale inference is employed. Method | $\mathrm{AP}$ | $\mathrm{AP}^{50}$ | $\mathrm{AP}^{75}$ | $\mathrm{AR}^{1}$ | $\mathrm{AR}^{10}$ | $\mathrm{AR}^{100}$ | $\mathrm{AR}^{500}$
---|---|---|---|---|---|---|---
DPNet-ensemble | 29.62 | 54.00 | 28.70 | 0.58 | 3.69 | 17.10 | 42.37
RRNet | 29.13 | 55.82 | 27.23 | 1.02 | 8.50 | 35.19 | 46.05
ACM-OD | 29.13 | 54.07 | 27.38 | 0.32 | 1.48 | 9.46 | 44.53
S+D | 28.59 | 50.97 | 28.29 | 0.50 | 3.38 | 15.95 | 42.72
BetterFPN | 28.55 | 53.63 | 26.68 | 0.86 | 7.56 | 33.81 | 44.02
HRDet+ | 28.39 | 54.53 | 26.06 | 0.11 | 0.94 | 12.95 | 43.34
CN-DhVaSa | 27.83 | 50.73 | 26.77 | 0.00 | 0.18 | 7.78 | 46.81
MSC-CenterNet | 31.13 | 54.13 | 31.41 | 0.27 | 1.85 | 6.12 | 50.48
CenterNet+ | 30.94 | 52.82 | 31.13 | 0.27 | 1.84 | 5.67 | 50.93
PG-YOLO | 26.05 | 49.63 | 24.15 | 1.45 | 9.20 | 33.65 | 42.63
Ours${\dagger}$ | 32.34 | 56.46 | 32.39 | 1.20 | 9.45 | 38.55 | 51.61

###  4.3 Implementation Details

Training We implement our model on the deep-learning framework of PaddlePaddle
[48]. For the VisDrone dataset, the input resolution of the network is set to
$1023\times 767$ during training, which results in an output resolution of
$256\times 192$. For the VOC dataset, the training resolution is fixed at
$512\times 512$. We use extensive data augmentation techniques to reduce the
risk of overfitting. For standard data augmentation, we adopt random cropping,
random scaling, random flipping, random blurring, and random color jittering.
Moreover, we use the mixup algorithm [34] to improve the generability of the
model. Some example images of our data augmentation techniques are showed in
Fig. 4.

Adam [49] algorithm is adopted to optimize the overall training objective. We
train our model on a V100 GPU with a batch size of 8. Note that for ablation
study, we train our model for 50k iterations, with an input resolution of
$511\times 511$, to save computational resources. When comparing to other
state-of-the-arts, we train the model for a total of 120k iterations. The
learning rate starts from 2e-4 and, is decayed by cosine annealing strategy.

Inference Following the setting of CenterNet, we use three levels of test
augmentation, i.e., no augmentation, flipping, and multi-scale (1.0, 1.25,
1.5, and 1.8) test augmentation. Specifically, for flipping test augmentation,
the network predictions of the flipped and non-flipped inputs are average
before decoding. For multi-scale test augmentation, non-maximum suppression
(NMS) is further applied to filter out the duplicate predictions.

Table 3: The performance comparison on VOC 2007 test set, in terms of mAP (%). Method | Backbone | Input size | mAP | FPS
---|---|---|---|---
Faster R-CNN [7] | ResNet101 | 1000$\times$600 | 76.4 | 5
R-FCN [50] | ResNet101 | 1000$\times$600 | 80.5 | 9
OHEM [51] | VGG-16 | 1000$\times$600 | 74.6 | -
R-FCN [50] | ResNet101+DCN | 1000$\times$600 | 82.6 | -
CoupleNet [52] | ResNet101 | 1000$\times$600 | 82.7 | 8.2
DeNet(wide) [53] | ResNet101 | 512$\times$512 | 77.1 | 33
FPN-Reconfig [54] | ResNet101 | 1000$\times$600 | 82.4 | -
Yolov2 [55] | DarkNet19 | 544$\times$544 | 78.6 | 40
SSD [11] | VGG-16 | 513$\times$513 | 78.9 | 19
DSSD [14] | VGG-16 | 513$\times$513 | 81.5 | 5.5
RefineDet [16] | VGG-16 | 512$\times$512 | 81.8 | 24
CenterNet [25] | ResNet101 | 512$\times$512 | 78.7 | 30
CenterNet [25] | DLA | 512$\times$512 | 80.7 | 33
HRDNet [47] | ResNeXt50+101 | 2000$\times$1200 | 82.4 | -
Ours | CBResNet50 | 511$\times$511 | 78.2 | 25.08
Ours | CBResNet50+DCN | 511$\times$511 | 81.05 | 21.98
Ours | ResNet101+DCN | 511$\times$511 | 83.04 | 15.96

###  4.4 Comparison with state-of-the-arts

We first demonstrate the advancement of our proposed method on the very
challenging VisDrone dataset. Then we move on to prove the strong generality
of our model on the VOC dataset.

VisDrone The comparison results with other state-of-the-art methods on
VisDrone2019 DET test-challenge and validation subset is showed in Table 2 and
Table 1, respectively. Our model achieves the best performance among other
detectors, with an AP score of 32.34% on the test-challenge subset, and 39.48%
AP on the validation subset, respectively. This outperforms other algorithms
by a large margin, with an improvement of 7.08% AP over ClusDet [43],
demonstrating the effectiveness of our proposed model. It is worth noting that
the average recall (AR) obtained by the proposed model also outperforms other
methods at all levels (from AR1 to AR500), which shows that our model can
better cope with the variation in object scale (find out more object
instance), with the help of adaptive feature selection module.

PASCAL-VOC Table 3 tabulates the results of various state-of-the-art methods
on VOC dataset. Note that flipping test augmentation is used by default. Our
method achieves the best performance, with an mAP score of 83.04%, while still
running in real-time (15.96 FPS). DSSD yields 81.5% mAP, which is comparable
to our method using CBResNet50 [56] as the backbone, whereas the inference
speed is approximately one third of ours (5.5 FPS vs. 21.98 FPS). Therefore,
our method performs well on the balance of speed and accuracy.

Figure 5: Qualitative visualization of the detection results on VisDrone-DET
test-challenge subset. Our model is capable of dealing with objects with a
wide range of variation in shape, viewpoint, lighting condition, and blurring,
etc.

###  4.5 Ablation Studies

Strong baseline The contribution of each advanced components to our baseline
detector is tabulated in Table 4, where GCB denotes the global context block
[35], CASM is the class-aware sampling mechanism, and M-GIOU denotes the
modified GIOU loss for object size regression. As shown in Table 4, all the
techniques boost the performance, e.g., adding CASM to the baseline method
improves the AP from 25.92% to 26.71%, with an increase of 0.79%. Thanks to
these techniques, we produce a stronger baseline that delivers an AP score of
28.34%, an improvement of 2.42% over the CenterNet baseline. It is worth
noting that the improvement is almost cost-free, as mixup, CASM, and M-GIOU do
not introduce any extra computational cost, and GCB owns the light-weight
property. Therefore, the final baseline still runs in an FPS of 20.76 on a
single GPU of NVIDIA Tesla V100.

Influence of the regression weight $\lambda_{size}$ Further on we study the
influence of the regression weight on the detector accuracy, as shown in Table
5. $\lambda_{size}=1.0$ yields good results, with an AP score of 26.85%. For
larger ($\lambda_{size}=5.0$) or smaller ($\lambda_{size}=0.2$) values, the AP
degrades a little bit, with 0.23% and 0.57% AP, respectively. This shows that
M-GIOU loss is not much sensitive to the variation of $\lambda_{size}$.

Channel fusion vs. Spatial fusion To analyze the effectiveness of the proposed
AFSM, we further report the results of ASFF [18] based on the CenterNet
baseline. As shown in Table 6, applying ASFF to the baseline improves the
performance by 3.23% AP, showing that fusing multi-level representations via
point-wise spatial addition helps the model to tackle the variation in object
scale. However, adding AFSM to the baseline boosts the performance by 4.02%
AP, with an increase of 0.79% AP compared to ASFF. This implies that fusing
features across feature levels by re-weighting the importance distribution
over the channel dimension may be more suitable and effective than the spatial
dimension. Moreover, instead of using several $1\times 1$ convs to generate
fusion weights for a specific image, we regard the fusion weight as a
learnable vector. This enables AFSM to learn the distribution of object size
over the whole dataset, which makes the selection process more stable and
effective.

Table 4: Effect of each advanced techniques on the baseline, in terms of APs (%). All models employ ResNet50 [1] as backbone network. FPS are reported on VisDrone2019 DET validation subset, keeping the original resolution for testing. CenterNet | GCB | mixup | CASM | M-GIOU | AP | AP50 | AP75 | FPS
---|---|---|---|---|---|---|---|---
✓ |  |  |  |  | 25.92 | 46.81 | 24.75 | 23.20
✓ | ✓ |  |  |  | 27.51 | 49.11 | 26.66 | 20.76
✓ |  | ✓ |  |  | 26.94 | 47.87 | 26.14 | 22.58
✓ |  |  | ✓ |  | 26.71 | 47.76 | 25.71 | 22.58
✓ |  |  |  | ✓ | 26.85 | 48.04 | 26.00 | 22.58
✓ | ✓ | ✓ | ✓ | ✓ | 28.34 | 49.92 | 27.92 | 20.76
Table 5: The influence of $\lambda_{size}$ on our model. $\lambda_{size}$ | AP | AP50 | AP75 | AR1 | AR10 | AR100 | AR500
---|---|---|---|---|---|---|---
0.2 | 26.28 | 47.74 | 25.21 | 0.62 | 7.01 | 35.76 | 35.76
1.0 | 26.85 | 48.04 | 26.00 | 0.72 | 7.03 | 36.23 | 36.23
5.0 | 26.62 | 47.18 | 25.94 | 0.58 | 6.75 | 36.05 | 36.05

How to obtain adaptive feature selection weights For obtaining the weights
$\alpha^{l}\in\mathbb{R}^{d}$ to select representative features across multi-
scale levels, we evaluate the performance based on three kinds of methods,
where $d$ is the number of channels in feature maps $X^{l}$. The difference
between these methods is the ways to obtain $\beta^{l}$ in Eq. 8. The first is
a learnable vector, which is initialized to one and, can be updated by the
standard back-propagation algorithm. For the second version, we first pool the
feature maps $X^{l}$ using global average pooling (GAP) operation, obtaining
$X_{p}^{l}\in\mathbb{R}^{d}$. Then each of the pooled features is followed by
a fully connected (FC) layer, to obtain $\beta^{l}$, which is given as
follows:

$X_{p}^{l}=\mathrm{GAP}(X^{l})$ (9) $\beta^{l}=\mathrm{FC}(X_{p}^{l})$ (10)

The third version uses GAP to pool the feature maps $X^{l}$. Then we
concatenate the features from all levels, denoted as
$X_{p}=\mathrm{[}...;(X_{p}^{l})^{\intercal};...;(X_{p}^{m})^{\intercal}\mathrm{]}^{\intercal}\in\mathbb{R}^{md}$,
following by a FC layer to obtain
$\gamma=\mathrm{[}...;(\beta^{l})^{\intercal};...;(\beta^{m})^{\intercal}\mathrm{]}^{\intercal}\in\mathbb{R}^{md}$,
where $m$ is the number of levels. For convenience, we denote the three kinds
of methods as V1, V2, V3, respectively, and the results of these methods are
showed in Table 7. These three methods achieve comparable performance, with
29.94%, 29.57%, and 29.79% AP for V1, V2, V3, respectively. However, V1 enjoys
the advantages of the fastest inference among the three versions, with an FPS
of 21.71. This shows that it is not the ways of obtaining $\beta^{l}$ that
brings the performance gain, but the mechanism of adaptive feature selection
proposed in this paper. The advantages of AFSM are two-fold. On the one hand,
it aggregates representations across various feature levels, thus making full
use of information on different scales. On the other hand, by applying an
importance weight $\alpha^{l}$ when aggregating representation at level $l$,
the model can obtain useful information while relegating the useless parts for
the subsequent detection task.

(a) PASCAL-VOC

(b) VisDrone

Figure 6: Visualization of the learned weights $\alpha$ at each level (from level3 to level6). The horizontal axis is the channel dimension, with 256 channels. Table 6: Comparison of AFSM and ASFF [18]. APs are reported on the VisDrone-DET 2019 validation subset. Method | AP | AP50 | AP75 | FPS
---|---|---|---|---
Baseline | 25.92 | 46.81 | 24.75 | 23.20
\+ ASFF | 29.15 | 54.04 | 27.34 | 22.34
\+ AFSM | 29.94 | 55.40 | 28.28 | 22.76
Table 7: Different ways of obtaining feature selection weights. The APs are reported on the VisDrone-DET validation subset. Baseline is CenterNet with ResNet50 as backbone network, and uses the feature maps at level 3 for the following detection head. Three kinds of methods yield comparable performance, while V1 achieving the fastest inference. Method | AP | AP50 | AP75 | FPS
---|---|---|---|---
Baseline | 25.92 | 46.81 | 24.75 | 23.20
\+ V1 | 29.94 | 55.4 | 28.28 | 22.76
\+ V2 | 29.57 | 55.3 | 27.58 | 17.77
\+ V3 | 29.79 | 55.15 | 28.04 | 18.02

Visualization of feature selection weights In order to analyze whether the
proposed adaptive feature selection module learns to aggregate useful
information from different feature levels, we visualize the learned weights
$\alpha$ at each level as a heatmap. As shown in Fig. 6, the selection weight
$\alpha^{l}\in\mathbb{R}^{d}$ at level $l$ is exhibited horizontally, where
$d=256$ in this paper. It can be observed from Fig. 6 that, AFSM successfully
captures the scale distribution of the specific dataset, and learns to
adaptively fuse features across multi-scale levels. Specifically, for the VOC
dataset, the object scale is relatively uniform and, there is not so much
small object. Therefore, the semantic information at high scale level, e.g.,
level 5, plays a more important role for predicting the large objects, as
shown in Fig. 6. On the other hand, the object scale varies significantly on
the VisDrone dataset and, most of them are small objects. It is hence the
spatially finer texture information is much beneficial for detecting the vast
majority of small objects (see Fig. 6).

Figure 7: Effect of the class-aware sampling mechanism (CASM). CASM boosts the
performance for each of classes, especially for ‘awning tricycle’, with an
improvement of 2.12% AP.

Effectiveness of the class-aware sampling mechanism To analyze the effect of
the proposed class-aware sampling mechanism, we plot the AP (%) of each class
on the VisDrone-DET validation subset, with and without CASM. Fig. 7 clearly
shows that CASM boosts the performance for each of the classes, especially for
the minor classes, e.g., with an improvement of 2.16% and 2.12% AP on ‘truck’
and ‘awning tricycle’, respectively. This demonstrates the effectiveness of
CASM, which uplifts the occurrence for minor classes during the training
phase, resulting in more supervisory signals and consequently, achieving
higher accuracy.

Evaluation on other detection algorithms We first instantiate the proposed
adaptive feature selection module into the CenterNet detection framework, to
evaluate its contribution. However, we emphasize that AFSM is a plug and play
module, which can also be applied to other detection algorithms, irrespective
of one-stage or two-stage. Therefore, more experiments are carried out on two
representative detectors, Faster R-CNN [7] and CornerNet [23]. The former is a
state-of-the-art two-stage detector and, the latter is the first to attempt to
detect objects with the keypoint estimation method.

The experimental results are showed in Table 8. AFSM consistently increases
the performance of two selected detection algorithms, with ResNet50 and
ResNet101 as the backbone network. However, it is worth to note that the
contribution to CornerNet is more significant, which improves from 72.13% to
73.60%, an increase of 1.47% mAP, when applying AFSM to CornerNet. This is due
to the fact that, CornerNet uses a single-scale feature (i.e., level 3) for
the subsequent detection head, and AFSM facilitates the detection branch
considerably, with the ability to adaptively selecting useful information from
various feature scales. On the other hand, Faster R-CNN has already employed
FPN for multi-scale object detection, while AFSM can still improve the
performance by 0.43% mAP, demonstrating the effectiveness of AFSM.

Table 8: Contribution of AFSM to CornerNet [23] and Faster R-CNN [7]. mAP (%) is reported on VOC 2007 test set. $(800,1333)$ implies that the shorter size of image is resize to 800, while keeping the longer size within 1333. $\dagger$ means that ResNet50-D [57] is used as backbone network. Method | Backbone | Resolution | AFSM | mAP | FPS
---|---|---|---|---|---
CornerNet | ResNet50$\dagger$ | 511$\times$511 |  | 72.13 | 31.5
CornerNet | ResNet50$\dagger$ | 511$\times$511 | ✓ | 73.60 | 30.77
CornerNet | ResNet101 | 511$\times$511 |  | 70.67 | 29.2
CornerNet | ResNet101 | 511$\times$511 | ✓ | 71.44 | 27.9
Faster R-CNN | ResNet50$\dagger$ | (800,1333) |  | 81.63 | 25.23
Faster R-CNN | ResNet50$\dagger$ | (800,1333) | ✓ | 82.06 | 21.86

##  5 Conclusion

This paper aims at tackling two fundamental challenges that limit the
performance of object detection algorithms, i.e., the wide range of variation
in object scale and the class imbalance problem. To address these challenges,
a novel and effective adaptive feature selection module (AFSM) is first
proposed, which adaptively learns the selection weight at each scale in the
channel dimension, to enhance the important clues at a certain feature level.
Secondly, we propose a class-aware sampling mechanism (CASM) to automatically
assign the sampling weight to each of the images during training, according to
the number of objects in each class. Each of the novel designs can improve the
performance of the detector considerably. Finally, we uplift the baseline to a
new state-of-the-art performance, with tiny inference overhead, thanks to the
novel proposed methods.

## Acknowledgment

## References

  * [1] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [2] G. Huang, Z. Liu, L. Van Der Maaten, K. Q. Weinberger, Densely connected convolutional networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700–4708.
  * [3] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556.
  * [4] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1–9.
  * [5] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, C. L. Zitnick, Microsoft coco: Common objects in context, in: European conference on computer vision, Springer, 2014, pp. 740–755.
  * [6] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, A. Zisserman, The pascal visual object classes (voc) challenge, International Journal of Computer Vision 88 (2) (2010) 303–338.
  * [7] S. Ren, K. He, R. Girshick, J. Sun, Faster r-cnn: Towards real-time object detection with region proposal networks, in: Advances in neural information processing systems, 2015, pp. 91–99.
  * [8] J. Redmon, S. Divvala, R. Girshick, A. Farhadi, You only look once: Unified, real-time object detection, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 779–788.
  * [9] T.-Y. Lin, P. Goyal, R. Girshick, K. He, P. Dollár, Focal loss for dense object detection, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2980–2988.
  * [10] B. Singh, L. S. Davis, An analysis of scale invariance in object detection snip, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3578–3587.
  * [11] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, A. C. Berg, Ssd: Single shot multibox detector, in: European conference on computer vision, Springer, 2016, pp. 21–37.
  * [12] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, S. Belongie, Feature pyramid networks for object detection, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2117–2125.
  * [13] K. He, G. Gkioxari, P. Dollár, R. Girshick, Mask r-cnn, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961–2969.
  * [14] C.-Y. Fu, W. Liu, A. Ranga, A. Tyagi, A. C. Berg, Dssd: Deconvolutional single shot detector, arXiv preprint arXiv:1701.06659.
  * [15] S. Woo, S. Hwang, I. S. Kweon, Stairnet: Top-down semantic aggregation for accurate one shot detection, in: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE, 2018, pp. 1093–1102.
  * [16] S. Zhang, L. Wen, X. Bian, Z. Lei, S. Z. Li, Single-shot refinement neural network for object detection, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4203–4212.
  * [17] J. Redmon, A. Farhadi, Yolov3: An incremental improvement, arXiv preprint arXiv:1804.02767.
  * [18] S. Liu, D. Huang, Y. Wang, Learning spatial fusion for single-shot object detection, arXiv preprint arXiv:1911.09516.
  * [19] Q. Zhao, T. Sheng, Y. Wang, Z. Tang, Y. Chen, L. Cai, H. Ling, M2det: A single-shot object detector based on multi-level feature pyramid network, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, 2019, pp. 9259–9266.
  * [20] J. Hu, L. Shen, G. Sun, Squeeze-and-excitation networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132–7141.
  * [21] Z. Cai, N. Vasconcelos, Cascade r-cnn: Delving into high quality object detection, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 6154–6162.
  * [22] Y. Wu, Y. Chen, L. Yuan, Z. Liu, L. Wang, H. Li, Y. Fu, Rethinking classification and localization for object detection, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 10186–10195.
  * [23] H. Law, J. Deng, Cornernet: Detecting objects as paired keypoints, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 734–750.
  * [24] K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, Q. Tian, Centernet: Keypoint triplets for object detection, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 6569–6578.
  * [25] X. Zhou, D. Wang, P. Krähenbühl, Objects as points, in: arXiv preprint arXiv:1904.07850, 2019.
  * [26] X. Zhou, J. Zhuo, P. Krähenbühl, Bottom-up object detection by grouping extreme and center points, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019) 850–859.
  * [27] J. Long, E. Shelhamer, T. Darrell, Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3431–3440.
  * [28] B. Hariharan, P. Arbeláez, R. Girshick, J. Malik, Hypercolumns for object segmentation and fine-grained localization, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 447–456.
  * [29] S. Liu, L. Qi, H. Qin, J. Shi, J. Jia, Path aggregation network for instance segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8759–8768.
  * [30] M. Tan, R. Pang, Q. V. Le, Efficientdet: Scalable and efficient object detection, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 10781–10790.
  * [31] B. Chen, G. Ghiasi, H. Liu, T.-Y. Lin, D. Kalenichenko, H. Adam, Q. V. Le, Mnasfpn: Learning latency-aware pyramid architecture for object detection on mobile devices, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 13607–13616.
  * [32] G. Ghiasi, T.-Y. Lin, Q. V. Le, Nas-fpn: Learning scalable feature pyramid architecture for object detection, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2019, pp. 7036–7045.
  * [33] D. Oñoro-Rubio, M. Niepert, R. J. López-Sastre, Learning short-cut connections for object counting, arXiv preprint arXiv:1805.02919.
  * [34] H. Zhang, M. Cisse, Y. N. Dauphin, D. Lopez-Paz, mixup: Beyond empirical risk minimization, arXiv preprint arXiv:1710.09412.
  * [35] Y. Cao, J. Xu, S. Lin, F. Wei, H. Hu, Gcnet: Non-local networks meet squeeze-excitation networks and beyond, in: Proceedings of the IEEE International Conference on Computer Vision Workshops, 2019, pp. 0–0.
  * [36] H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, S. Savarese, Generalized intersection over union: A metric and a loss for bounding box regression, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 658–666.
  * [37] A. Newell, K. Yang, J. Deng, Stacked hourglass networks for human pose estimation, in: European conference on computer vision, Springer, 2016, pp. 483–499.
  * [38] Z. Zhang, T. He, H. Zhang, Z. Zhang, J. Xie, M. Li, Bag of freebies for training object detection neural networks, arXiv preprint arXiv:1902.04103.
  * [39] X. Wang, R. Girshick, A. Gupta, K. He, Non-local neural networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7794–7803.
  * [40] P. Zhu, L. Wen, D. Du, X. Bian, Q. Hu, H. Ling, Vision meets drones: Past, present and future, arXiv preprint arXiv:2001.06303.
  * [41] Z. Tian, C. Shen, H. Chen, T. He, Fcos: Fully convolutional one-stage object detection, in: Proceedings of the IEEE international conference on computer vision, 2019, pp. 9627–9636.
  * [42] J. Zhang, J. Huang, X. Chen, D. Zhang, How to fully exploit the abilities of aerial image detectors, in: Proceedings of the IEEE International Conference on Computer Vision Workshops, 2019, pp. 0–0.
  * [43] F. Yang, H. Fan, P. Chu, E. Blasch, H. Ling, Clustered object detection in aerial images, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 8311–8320.
  * [44] Z. Tang, X. Liu, G. Shen, B. Yang, Penet: Object detection using points estimation in aerial images, arXiv preprint arXiv:2001.08247.
  * [45] W. Yu, T. Yang, C. Chen, Towards resolving the challenge of long-tail distribution in uav images for object detection, arXiv preprint arXiv:2011.03822.
  * [46] X. Zhang, E. Izquierdo, K. Chandramouli, Dense and small object detection in uav vision based on cascade network, in: Proceedings of the IEEE International Conference on Computer Vision Workshops, 2019, pp. 0–0.
  * [47] Z. Liu, G. Gao, L. Sun, Z. Fang, Hrdnet: High-resolution detection network for small objects, arXiv preprint arXiv:2006.07607.
  * [48] Y. Ma, D. Yu, T. Wu, H. Wang, Paddlepaddle: An open-source deep learning platform from industrial practice, Frontiers of Data and Domputing 1 (1) (2019) 105–115.
  * [49] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980.
  * [50] J. Dai, Y. Li, K. He, J. Sun, R-fcn: Object detection via region-based fully convolutional networks, in: Advances in neural information processing systems, 2016, pp. 379–387.
  * [51] A. Shrivastava, A. Gupta, R. Girshick, Training region-based object detectors with online hard example mining, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 761–769.
  * [52] Y. Zhu, C. Zhao, J. Wang, X. Zhao, Y. Wu, H. Lu, Couplenet: Coupling global structure with local parts for object detection, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 4126–4134.
  * [53] A. Ghodrati, A. Diba, M. Pedersoli, T. Tuytelaars, L. Van Gool, Deepproposal: Hunting objects by cascading deep convolutional layers, in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 2578–2586.
  * [54] T. Kong, F. Sun, C. Tan, H. Liu, W. Huang, Deep feature pyramid reconfiguration for object detection, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 169–185.
  * [55] J. Redmon, A. Farhadi, Yolo9000: better, faster, stronger, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7263–7271.
  * [56] Y. Liu, Y. Wang, S. Wang, T. Liang, Q. Zhao, Z. Tang, H. Ling, Cbnet: A novel composite backbone network architecture for object detection., in: AAAI, 2020, pp. 11653–11660.
  * [57] T. He, Z. Zhang, H. Zhang, Z. Zhang, J. Xie, M. Li, Bag of tricks for image classification with convolutional neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 558–567.