# Word Frequency Does Not Predict Grammatical Knowledge in Language Models

Charles Yu, Ryan Sie11footnotemark: 1, Nico Tedeschi, and Leon Bergen
<EMAIL_ADDRESS>
University of California, San Diego
These authors contributed equally.

###### Abstract

Neural language models learn, to varying degrees of accuracy, the grammatical
properties of natural languages. In this work, we investigate whether there
are systematic sources of variation in the language models’ accuracy. Focusing
on subject-verb agreement and reflexive anaphora, we find that certain nouns
are systematically understood better than others, an effect which is robust
across grammatical tasks and different language models. Surprisingly, we find
that across four orders of magnitude, corpus frequency is unrelated to a
noun’s performance on grammatical tasks. Finally, we find that a novel noun’s
grammatical properties can be few-shot learned from various types of training
data. The results present a paradox: there should be less variation in
grammatical performance than is actually observed.

##  1 Introduction

Neural language models (Howard and Ruder, 2018; Devlin et al., 2019; Dai et
al., 2019; Yang et al., 2019; Radford et al., 2019) have achieved success in
both text prediction and downstream tasks such as question-answering, text
classification, and natural language inference. The strong performance of
these models raises scientific questions about the knowledge they have
acquired, in particular, about the abstractness and generality of their
linguistic representations.

Previous work has investigated the linguistic representations of neural
language models in several domains, and found varying evidence for how
linguistically adequate these representations are (Lau et al., 2017; Marvin
and Linzen, 2018; Goldberg, 2019; Futrell et al., 2019). This work has
employed psycholinguistic methodology in order to elicit grammatical judgments
from these models, inferring the models’ underlying representations from the
patterns of judgments.

In the current work, we focus on the variation in grammatical knowledge that
potentially exists within a neural language model. Just as in human
psycholinguistic tasks, previous work on neural LMs has observed variability
in grammatical judgments between different sentences; not all violations of a
grammatical constraint are judged to be equally bad. It is not clear, however,
whether there are systematic sources of variation in these judgments, and if
so, what the sources are.

We will focus on variation among lexical items, using English subject-verb
agreement and reflexive anaphora as a case study. We first ask whether
language models learn the grammatical properties of some nouns more accurately
than for others. We do this by measuring the accuracy of language models when
making grammatical judgments involving different nouns. We find systematic
variation among nouns: nouns that perform well on one task or language model
are more likely to perform well on other tasks or other language models. We
then consider possible sources of the observed variation between nouns,
finding that the grammatical properties of nouns are paradoxically easy to
learn; our results suggest that there should be much less variation than is
actually observed.111All code and experimental materials are available at
https://github.com/CharlesYu2000/lm-variation

### Related work

A number of other studies have investigated the linguistic representations of
neural models, both language models specifically and networks trained using
other objectives. Linzen et al. (2016); Gulordava et al. (2018); Kuncoro et
al. (2018) probe the ability of LSTMs to learn hierarchical structures.
Warstadt et al. (2019b) introduces a large-scale corpus of grammatical
acceptability judgements, trains RNNs to predict these judgments, and
concludes that the models outperform unsupervised baselines, but fall far
short of human performance. Lepori et al. (2020) finds that tree-based RNNs
outperform sequential RNNs on number prediction tasks, but that fine-tuning on
an artificially-generated augmentation set can bring the models closer to
parity.

Other work has focused on probing whether neural language models have acquired
adequate representations of specific linguistic phenomena. Marvin and Linzen
(2018) and Goldberg (2019) use a minimal pair methodology to assess the
grammatical knowledge of RNNs and BERT, looking at subject-verb number
agreement, reflexive anaphora, and negative polarity items. Wilcox et al.
(2018) examines whether RNN language models exhibit wh-licensing interactions
on surprisal associated with gaps, concluding they can represent long-distance
filler-gap dependencies and learn certain island constraints. Futrell et al.
(2019) studies whether neural language models show evidence for incremental
syntactic state representations using psycholinguistic methodology. Warstadt
et al. (2019a) studies BERT’s knowledge of NPI’s, focusing on differences
between tasks: boolean classification (e.g. Linzen et al. 2016 and Warstadt et
al. 2019b), minimal pair comparisons (e.g. Marvin and Linzen 2018 and Wilcox
et al. 2019), and probing tasks (e.g. Giulianelli et al. 2018).

##  2 Approach

We use the minimal pair methodology of Marvin and Linzen (2018) in order to
investigate the grammatical judgments of neural language models. Given a
minimal pair of sentences, i.e. a pair that differ from each other in their
acceptability due to a difference in just one grammatical property. If the
model understands the grammatical phenomenon being studied, it should assign
higher probability to the grammatical sentence than to the ungrammatical
sentence.

###  2.1 Grammatical tasks

Table 1 shows the 10 grammatical tasks (Marvin and Linzen, 2018) and the
templates used for generating minimal pairs. The tasks fall into two general
categories: subject-verb agreement (SVA) and reflexive anaphora (RA). The
first SVA task, SVA Simple, probes whether the model understands that subject
number must agree with the number of third-person present verbs:
[*=?*]¡judgements¿ The cat walks. * The cat walk. The other SVA tasks probe
whether the models have more sophisticated representations of number
agreement. For example, the SVA PP task measures whether the model is able to
ignore distractors (“boys”) which occur between the head of the subject and
the verb: [*=?*]¡judgements¿ The cat next to the boys jumps. * The cat next to
the boys jump. The object relative clause tasks probe whether the model
accurately maintains the head’s number in the presence of an embedded clause.
Marvin and Linzen (2018) provide extensive discussion of the linguistic
motivation for these tasks.

The RA tasks measure whether the language model understands the structural
conditions on the binding of reflexive pronouns. The tasks make use of the
following property of English reflexives: a reflexive pronoun needs to agree
in number with its antecedent. The RA Sent.Comp task evaluates whether the
model understands that reflexives must be in the same clause as their
antecedents: [*=?*]¡judgements¿ The lawyers said the defendant incriminated
himself. * The lawyers said the defendant incriminated themselves. The RA
tasks involving object relative clauses evaluate whether the models understand
that reflexive anaphora do not bind to the noun in an embedded clause but
rather to the head noun.

Task | Template
---|---
SVA Simple | The $\langle$TargetNoun$\rangle$ $\langle$Verb$\rangle$.
SVA Subj.Rel.Clause | The $\langle$TargetNoun$\rangle$ that liked the $\langle$Noun$\rangle$ $\langle$Verb$\rangle$.
SVA Sent.Comp. | The $\langle$Noun$\rangle$ said the $\langle$TargetNoun$\rangle$ $\langle$Verb$\rangle$.
SVA PP | The $\langle$TargetNoun$\rangle$ next to the $\langle$Noun$\rangle$ $\langle$Verb$\rangle$.
SVA Obj.Rel.Clause.That | The $\langle$TargetNoun$\rangle$ that the $\langle$Noun$\rangle$ liked $\langle$Verb$\rangle$.
SVA Obj.Rel.Clause.NoThat | The $\langle$TargetNoun$\rangle$ the $\langle$Noun$\rangle$ liked $\langle$Verb$\rangle$.
RA Simple | The $\langle$TargetNoun$\rangle$ $\langle$PastTransVerb$\rangle$ $\langle$himself/themselves$\rangle$.
RA Sent.Comp. | The $\langle$NonGenderedNoun$\rangle$ said the $\langle$TargetNoun$\rangle$ $\langle$PastTransVerb$\rangle$ $\langle$himself/themselves$\rangle$.
RA Obj.Rel.Clause.That | The $\langle$TargetNoun$\rangle$ that the $\langle$NonGenderedNoun$\rangle$ liked $\langle$PastTransVerb$\rangle$ $\langle$himself/themselves$\rangle$.
RA Obj.Rel.Clause.NoThat | The $\langle$TargetNoun$\rangle$ the $\langle$NonGenderedNoun$\rangle$ liked $\langle$PastTransVerb$\rangle$ $\langle$himself/themselves$\rangle$.

Table 1: Templates used for sentence generation. TargetNoun indicates the
position of the target noun whose performance score is being calculated.

###  2.2 Measuring the performance of a noun

We use these tasks in order to measure how well the model understands the
grammatical properties of a particular _target noun_. Given a specific target
noun, it is substituted as the TargetNoun in each of the task templates shown
in Table 1. This gives a partially specified template. For example,
substituting the target noun “zombie” in the SVA Simple template results in:
[*=?*]¡judgements¿ The zombie $\langle$Verb$\rangle$. Given each of these
partially specified templates, 500 minimal pairs are randomly sampled by
filling in the remaining lexical items. Finally, the model’s grammatical
judgments on the 500 minimal pairs are computed (by taking the difference in
scores between the grammatical and ungrammatical variants) and averaged,
resulting in a task performance score for the noun.

###  2.3 Limitations

These analyses are limited in several respects. First, only two grammatical
tasks are used. By using a wider range of tasks, it will be possible to
investigate a larger set of grammatical phenomena outside of number agreement.

Second, while the study focuses on the grammatical information carried by
nouns, other lexical types such as verbs are likely to carry this information
as well. Future work can determine whether the approach generalizes to verbs
and other lexical types.

Finally, while the study uses acceptability judgments in order to determine
the models’ grammatical knowledge, other probing tasks exist and may produce
different results (Warstadt et al., 2019a). We use acceptability judgments
because, to the best of our knowledge, feature probing has not been
extensively studied for GPT-2 or Transformer-XL. Different probing
architectures may produce different results for these models. It would be
desirable to understand the robustness of the current results to the choice of
experimental readout.

##  3 Methods

In this section we describe the process of calculating a target noun’s task
performance score in more detail.

###  3.1 Sentence generation

Using WordNet (Fellbaum, 1998) and VerbNet (Schuler, 2005), we compiled a list
of lexical items as shown in Table 2. The target nouns were drawn from the
Noun list, which consisted of animate nouns. Only nouns with distinct singular
and plural forms were included. All verbs in the Verb set have an intransitive
reading. For each pair of task template and target noun, 500 sentences were
randomly sampled by choosing lexical items from the appropriate word lists.

For each sampled sentence, 2*2 or 2*2*2 versions were generated (depending on
the template). These versions varied the grammaticality of the sentence and
the plurality of the target noun and any distractor nouns. For example, for
the SVA Simple task, 2*2 versions are generated for every sampled sentence:
Singular-Grammatical: The horse walks. Singular-Ungrammatical: * The horse
walk. Plural-Grammatical: The horses walk. Plural-Ungrammatical: * The horses
walks.

###  3.2 Models

Our experiments use three models, Transformer-XL (Dai et al., 2019), GPT-2
(Radford et al., 2019), and BERT (Devlin et al., 2019). We use the Hugging
Face implementations (Wolf et al., 2019) with the pre-trained models transfo-
xl-wt103, which is trained on the WikiText-103 dataset, gpt2-xl, which is
trained on the WebText dataset, and bert-base-uncased, which is trained on
BookCorpus and English Wikipedia.

Set Name | Transformer-XL | GPT-2 | BERT
---|---|---|---
Noun | 916 | 723 | 704
Verb | 615 | 228 | 406
NonGenderedNoun | 870 | 679 | 663
PastTransVerb | 1298 | 1034 | 1298

Table 2: Size of word sets for each model.

###  3.3 Sentence scoring

We now describe how a score was calculated for a particular sampled sentence.
For each of the sentence variants (e.g. Example 3.1), the model computes a
score. In the case of Transformer-XL and GPT-2, this score is simply the the
log probability of the string. For example, for Transformer-XL:

$\text{Score}_{\text{string}}(s)=\log P_{\text{TXL}}(s)$ (1)

where $P_{\text{TXL}}$ is the Transformer-XL language model probability
distribution.

For BERT, given its masked language model architecture, we follow the approach
of Goldberg (2019). For the SVA tasks, we compute the log conditional
probability of the verb whose number must agree with the target noun. For the
RA tasks, we compute the log conditional probability of the reflexive pronoun.
Both conditional probabilities are computed conditional on the left and right
contexts.

Given the scores for a sentence’s variants, we compute an overall score for
the sentence, which captures how much the model prefers the grammatical
variants to the ungrammatical variants. For each sampled sentence $S$, there
are either 2 or 4 minimal pairs among its variants. In Example 3.1, a. and b.
is a minimal pair, and c. and d. is a minimal pair. Letting $s_{a},...,s_{d}$
denote these variants, the overall score for the sentence is given by:

$\displaystyle\text{Score}_{\text{sent}}(S)=\frac{1}{2}($
$\displaystyle\text{Score}_{\text{string}}(s_{a})-\text{Score}_{\text{string}}(s_{b})$
$\displaystyle+$
$\displaystyle\text{Score}_{\text{string}}(s_{c})-\text{Score}_{\text{string}}(s_{d}))$

The formula when there are four minimal pairs is similar.

###  3.4 Noun scoring

We next compute an overall score for the target noun. As described in Section
3.1, for a specific target noun $n$ and task, we sample 500 sentences
$S_{1},...,S_{500}$. The noun’s score for this task is then given by:

$\text{Score}_{\text{noun}}(n)=\frac{1}{500}\sum_{i=1}^{500}\text{Score}_{\text{sent}}(S_{i})$
(2)

###  3.5 Word filtering and tokenization

Words were removed from a particular model if either their singular or plural
form was tokenized to _unk_ , or if their singular and plural forms were
assigned different numbers of tokens.222The latter constraint was used in
order to simplify batching. For BERT, words in the Verb set were removed if
they were assigned more than one token, as BERT does not model the joint
distribution over multiple masked tokens.

For Transformer-XL, we add a padding text333https://tinyurl.com/y9kjuj5q and a
start-of-sentence-token ($\langle$SOS$\rangle$) to the beginning of the
sentence and an end-of-sentence token ($\langle$EOS$\rangle$) to the end of
the sentence. For GPT-2, we make no modifications to the generated sentence
(although prefix spaces are added to the strings for tokenization purposes).
For BERT, since it is a masked language model, we replace the Verb (for SVA)
or reflexive pronoun (for RA) with a [MASK] token after tokenization. Thus,
each sentence will have a single mask token corresponding to the word that
should agree with the target noun.

##  4 Results

###  4.1 Noun performance is correlated across tasks

We first examine how each noun’s performance varies across the grammatical
tasks. For each noun-task pair, we measure the average performance of the noun
on that task, as described above. This gives 10 features per noun,
corresponding to the 10 grammatical tasks.

Figure 1 shows the pairwise comparisons between performance on the different
tasks for Transformer-XL. Results for BERT and GPT-2 are similar and are shown
in the appendix. The figure shows that performance is correlated across the
tasks; for many pairs of tasks, nouns which have higher performance on one
task are likely to have higher performance on the other.

Using principal component analysis, we found that a single principal component
explains 47% of task variance for Transformer-XL, and two principal components
explain 73%. Results are similar for BERT and GPT-2, and are shown in the
appendix. The first PC primarily measures performance on the four reflexive
anaphora tasks, while the second PC measures performance on the subject-verb
agreement across relative clause tasks. This suggests that there is a
dimension that characterizes whether the model understands how reflexive
binding constraints operate for a noun, and a dimension for whether the model
understands subject-verb agreement for the noun. Note that Figure 1
additionally demonstrates correlations between the reflexive tasks and the
subject-verb agreement tasks.

These results provide evidence that language models’ variation in performance
on the grammatical tasks is, in part, explained by properties of the nouns
which are stable across tasks. The models understand number agreement better
for some nouns, and worse for others.

Figure 1: Pairwise comparisons between tasks with Transformer-XL. Rows and
columns represent tasks, and one point represents a single noun’s performance
on a pair of tasks. The four tasks on the lower right, with strongest
correlations, all involve reflexive anaphora.

###  4.2 Noun performance is correlated across models

We next investigate whether nouns exhibit stable behavior across different
neural language models. For each pair of the three language models, we
measured how well a noun’s task performance in one language model predicted
its task performance in the other language model.

Figure 2 shows comparisons between pairs of language models on the 10
grammatical tasks. Of the 30 comparisons, 24 show significant positive
correlations between the pairs of language models. 22 of the correlations
remain significant after Bonferroni correction.

GPT-2 and Transformer-XL show the strongest correlation in performance. It is
possible that this is due to methodological differences between the task setup
for GPT-2 and Transformer-XL compared to BERT: GPT-2 and Transformer-XL are
performing a language modeling task in which the probability of a full
sentence is queried, while BERT performs masked language modeling on a single
target word. The difference may also be due to corresponding training
differences between BERT and the autoregressive language models.

The results provide evidence that nouns exhibit stable task performance across
language models. The source of the correlation across language models must
come from features of the training data. Properties of the natural text
distribution of nouns lead some of these nouns to be better understood than
others.

Figure 2: Pairwise comparisons between GPT-2, Transformer-XL, and BERT on the
10 grammatical tasks. Each row corresponds to a pair of language models, and
each column is a single task. One point represents the performance of a noun
on a single task.

###  4.3 Effect of frequency on task performance

Figure 3: Relationship between corpus frequency and task performance for
Transformer-XL, BERT, and GPT-2. Performance scores are z-normalized. Colors
indicate the ten grammatical tasks and singular/plural form of the noun (_s_
indicates singular, _p_ indicates plural). Each point represents task
performance for a single noun.

In Sections 4.1 and 4.2, we found evidence that nouns exhibit stable
performance across different grammatical tasks and language models. One
obvious explanation of these results is that nouns vary in their frequency in
natural text, and language models learn more accurate grammatical
representations for more frequent nouns.

In order to investigate this, we measured the frequency of each noun in two
corpora: WikiText-103, a 103 million token subset of Wikipedia, which was used
for training Transformer-XL; and Open WebText (Gokaslan and Cohen, 2019), an
open-source implementation of the web corpus used to train GPT-2.444BERT was
trained on a mix of Wikipedia text and BookCorpus. Because, as of this
writing, BookCorpus is no longer distributed, WikiText-103 was used as a proxy
for BERT training frequencies. Word frequencies were measured separately for
singular and plural noun forms. Figure 3 shows the relationship between
frequency and task performance on each of the ten grammatical tasks. The
appendix shows the results broken down by task type.

The results show no clear relationship between noun frequency and task
performance. Frequency explains no more than $0.1\%$ of the variation in
performance. This holds true over more than four orders of magnitude in
frequency. This provides evidence that 1) differences in corpus frequency do
not explain the systematic differences observed between nouns, and 2)
relatively few observations suffice for transformer language models to learn
correct number agreement behavior for a noun. In the next section, we
investigate this finding further.

##  5 Few-shot learning for novel lexical items

The results in the previous section provide evidence that nouns systematically
vary in their performance on grammatical tasks; some nouns perform better than
others across tasks and language models. However, this variation is not
explained by frequency of occurrence in natural text. Nouns that occur on the
order of 100 times in a corpus do not have systematically worse performance
than nouns that occur $10^{6}$ times.

The results raise a question: if frequency does not influence how well a noun
is understood, what does? If low frequency nouns are understood as well as
higher frequency nouns, then this suggests that language models few-shot learn
the grammatical properties of nouns. We suggest that by studying what makes a
noun learnable in a few-shot setting, it may be psosible to better understand
the sources of the observed variation.

We use a few-shot learning paradigm, introducing a new lexical item into the
vocabulary of the language model, either “wug” (intended as a new singular
noun), or “wuz” (intended as a plural). We then fine-tune the language model
using several example sentences containing this word. Note that this paradigm
is distinct from nearly all of the few-shot learning experiments performed in
Radford et al. (2019); Brown et al. (2020), which operate on a known
vocabulary.555Brown et al. (2020) perform several experiments on novel
vocabulary items.

###  5.1 Learning agreement from syntactic data

We first look at whether training data containing explicit syntactic markers
of number agreement is sufficient for few-shot learning. Table 3 describes the
types of training data we examine. The three types of training data use
different syntactic markers of plurality to indicate whether the new noun is
singular or plural.

The language models are fine-tuned with 5 sentences drawn from a single
training data type. GPT-2 was fine-tuned for 2 epochs, and BERT was fine-tuned
for 4 epochs.666Prior to more systematic experiments, we informally optimized
the number of fine-tuning epochs. Transformer-XL was not used for the fine-
tuning experiments, due to issues with introducing new vocabulary items given
Transformer-XL’s adaptive weight embedding.

After fine-tuning, each model was evaluated on the 10 grammatical tasks in
Table 1. For each grammatical task, 500 sentences were sampled from the task
template, and a performance score was calculated by averaging scores of the
samples, as described in Section 3.4.

Figure 4 shows results for fine-tuning on the three types of syntactic data.
Compared to model performance on real lexical items (shown in the leftmost
column), both BERT and GPT-2 achieve qualitatively similar performance given
the Pred-adj and Reflexive training data, but worse performance given the
Simple training data. Performance is weakest on subject-verb agreement (SV-
agreement) tasks involving relative clauses. When trained on data containing
reflexive anaphora, both models achieve notably higher performance on the
grammatical tasks involving reflexive anaphora.

The results provide evidence that small amounts of syntactic training data
support learning the agreement properties of novel nouns. They also provide
evidence of heterogeneity among different types of training data. Training
from bare present tense verbs is least effective, and training from sentences
containing reflexives leads to improved performance on tasks which require
understanding of the conditions on reflexive binding.

Training data type | Template
---|---
Simple | The wug/wuz $\langle$PresentTenseVerb$\rangle$.
Pred-adj | The wug/wuz is/are $\langle$Adj$\rangle$.
Reflexive | The wug/wuz $\langle$Verb$\rangle$ himself/themselves.

Table 3: The three types of training data used for syntactic fine-tuning.
Figure 4: Few-shot learning from syntactic examples (averaging over plural and
singular results). Columns show different types of training data, and rows
show the 10 grammatical tasks. The bert-base and gpt2-xl columns indicate
model performance on known lexical items, i.e. summarizing results from
Section 4. The baseline columns indicate performance of non-fine-tuned models
on the novel wug/wuz lexical items. Scores are differences of log-
probabilities between grammatical and ungrammatical. The 95% confidence
interval around each point estimate is always smaller than $\displaystyle\pm
0.25$.

###  5.2 Learning agreement from semantic data

We next examine whether purely semantic indicators of plurality are sufficient
for learning a noun’s number agreement properties. We look at several types of
constructions which provide information about the plurality of a noun, but
using predicates with past tense verbs that don’t inflect for number so that
there is no grammatical number agreement. In particular, we note the different
possible readings with reference to the distributive and collective
distinction described in the semantics literature (Lønning, 1997; Lasersohn,
2011; Champollion, 2015). For documentation of predicates that require a
collective NP subject, see Levin (1993).

We use the fine-tuning method from Section 5.1.

#### Singular constructions

In order to induce singular noun interpretations, we use the singular-biased
constructions shown at the top of Table 4. For example, if a wug worked all
alone or came unaccompanied, it is likely that “wug” is both semantically and
grammatically singular. However, these constructions do not gramatically
require the head noun to be singular: they are compatible with distributive
readings where the predicate individually applies to members of a group (e.g.
“the lawyers worked all alone” means each lawyer worked alone).

| Training data type | Example
---|---|---
Singular | all-alone | The wug worked all alone.
unaccompanied | The wug came unaccompanied.
separated-entire | The wug became separated from the entire group.
personally | The wug personally thanked me.
Plural | unison | The wuz nodded in unison.
together | The wuz ate together.
simultaneously | The wuz jumped simultaneously.
outnumbered | The wuz outnumbered the cats.
constituted | The wuz constituted a majority of the team.
gathered | The wuz gathered quietly.

Table 4: Types of training data used for semantic fine-tuning.

BERT and GPT-2 were fine-tuned on 5 examples of each of the singular
constructions. Figure 5 shows the results. None of the constructions
consistently induced correct performance on the grammatical tasks across both
models. Three of the constructions — _all-alone_ , _unaccompanied_ , and
_personally_ — led to strong performance on the reflexive anaphora tasks
(stronger than the average performance calculated in Section 4). The
_separated-entire_ construction consistently decreased performance on the
tasks relative to baseline.

Figure 5: Few-shot learning from singular semantic examples. The bert-base and
gpt2-xl columns indicate model performance on known lexical items, i.e.
summarizing results from Section 4. The baseline columns indicate performance
of non-fine-tuned models on the novel wug token.

#### Plural constructions

In order to provide the models with data indicating that a novel noun is
plural, we use constructions which force either collective or distributive
readings. For example, in Table 4, if the wuz constituted the majority of the
team, then the word “wuz” must be semantically plural. The construction
_constituted a majority_ is collective because it must apply to the group as a
whole: The doctors constituted a majority of the team. *Distributive reading:
each of the doctors constituted a majority. Collective reading: the doctors as
a group constituted a majority. While the argument of a collective predicate
must be semantically plural, it is not necessarily grammatically plural. For
example, the singular “the group” could constitute the majority of the team.

Three of the constructions in Table 4 are collective: _outnumbered_ ,
_constituted_ , and _gathered_. The other three are distributive phrasal
predicates, which force distributive readings: The architects nodded in
unison. Distributive reading: each of the architects nodded. *Collective
reading: the group of architects itself nodded.

Figure 6: Few-shot learning from plural semantic examples. The baseline
columns indicate performance of non-fine-tuned models on the novel wuz token.

Figure 6 shows the plural learning results. The 6 types of training data
perform comparably on the subject-verb agreement tasks (and similar to the
baseline model, which represents performance prior to fine-tuning). The three
distributive phrasal constructions perform better on the reflexive anaphora
tasks than the three collective constructions, though all constructions
improve relative to the baseline.

##  6 Discussion

We have investigated the sources of variation in neural language models’
grammatical judgments. We found that there are systematic differences between
nouns: when a language model exhibits knowledge of a noun’s grammatical
properties in one task, it is more likely to do so in other tasks. Moreover,
when one language model exhibits this knowledge, other language models are
more likely to as well. The study found two latent dimensions of variation
between nouns: one corresponding to how well the models understood its
behavior with reflexive pronouns, and the other corresponding to subject-verb
agreement.

Subsequent analyses demonstrate a pair of empirical phenomena:

  1. 1.

It is relatively easy to learn the number agreement properties of a noun. The
models learn the agreement properties of a novel noun from just a few samples,
and the data supporting few-shot learning appears to be densely distributed;
nearly all types of syntactic and semantic data examined lead to improvements
on the reflexive pronoun or subject-verb agreement tasks.

  2. 2.

Nouns that occur more frequently during training are not learned more
accurately. Many nouns that occur with high frequency are not learned
accurately.

These results suggest that nouns should vary less in their grammatical
performance than is actually observed; the study finds _excess variation_ in
grammatical performance. If number agreement can be correctly learned from a
few samples (FSL samples), then one would expect model performance to either
a) improve with more data, as more FSL samples are observed, or b) improve
with more data up to some threshold, and then asymptote after learning has
saturated. In either case, for high frequency nouns, a sufficient number of
FSL samples should be observed for these nouns to be learned very accurately.

A potential explanation of the results is that they are caused by catastrophic
forgetting (Ratcliff, 1990; French, 1999): although a sufficient number of FSL
samples are observed for a noun, these samples are forgotten during training,
causing the performance of the noun to degrade. This explanation is
implausible. If catastrophic forgetting is occurring, then the problem should
be more severe for infrequent nouns than for frequent nouns, as the interval
between training samples will be longer for infrequent nouns. This would
predict better performance for frequent nouns.

## Acknowledgements

We thank the Google Cloud Platform research program for support. The Titan V
used for this research was donated by the NVIDIA Corporation.

## References

  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_.
  * Champollion (2015) Lucas Champollion. 2015.  Distributivity, collectivity and cumulativity.  In L. Matthewson, C. Meier, H. Rullmann, and T. E. Zimmermann, editors, _Companion to Semantics_. Wiley-Blackwell.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019.  Transformer-XL: Attentive language models beyond a fixed-length context.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2978–2988, Florence, Italy. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Fellbaum (1998) C. Fellbaum, editor. 1998.  _WordNet: An electronic lexical database_.  MIT Press.
  * French (1999) Robert M French. 1999.  Catastrophic forgetting in connectionist networks.  _Trends in cognitive sciences_ , 3(4):128–135.
  * Futrell et al. (2019) Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019.  Neural language models as psycholinguistic subjects: Representations of syntactic state.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 32–42, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Giulianelli et al. (2018) Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018.  Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 240–248, Brussels, Belgium. Association for Computational Linguistics.
  * Gokaslan and Cohen (2019) Aaron Gokaslan and Vanya Cohen. 2019.  Openwebtext corpus.  http://Skylion007.github.io/OpenWebTextCorpus.
  * Goldberg (2019) Yoav Goldberg. 2019.  Assessing BERT’s syntactic abilities.  _arXiv preprint arXiv:1901.05287_.
  * Gulordava et al. (2018) Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018.  Colorless green recurrent networks dream hierarchically.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1195–1205, New Orleans, Louisiana. Association for Computational Linguistics.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 328–339, Melbourne, Australia. Association for Computational Linguistics.
  * Kuncoro et al. (2018) Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. 2018.  LSTMs can learn syntax-sensitive dependencies well, but modeling structure makes them better.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1426–1436, Melbourne, Australia. Association for Computational Linguistics.
  * Lasersohn (2011) Peter Nathan Lasersohn. 2011.  Mass nouns and plurals.  In _Semantics: An International Handbook of Natural Language Meaning_ , pages 1131–1153. De Gruyter.
  * Lau et al. (2017) Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017.  Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge.  _Cognitive Science_ , 41(5):1202–1241.
  * Lepori et al. (2020) Michael A Lepori, Tal Linzen, and R Thomas McCoy. 2020.  Representations of syntax [MASK] useful: Effects of constituency and dependency structure in recursive LSTMs.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , Seattle, Washington. Association for Computational Linguistics.
  * Levin (1993) Beth Levin. 1993.  _English verb classes and alternations: A preliminary investigation_.  University of Chicago press.
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016.  Assessing the ability of LSTMs to learn syntax-sensitive dependencies.  _Transactions of the Association for Computational Linguistics_ , 4:521–535.
  * Lønning (1997) Jan Tore Lønning. 1997.  Plurals and collectivity.  In _Handbook of logic and language_ , pages 1009–1053. Elsevier.
  * Marvin and Linzen (2018) Rebecca Marvin and Tal Linzen. 2018.  Targeted syntactic evaluation of language models.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Ratcliff (1990) Roger Ratcliff. 1990.  Connectionist models of recognition memory: constraints imposed by learning and forgetting functions.  _Psychological review_ , 97(2):285.
  * Schuler (2005) Karin Kipper Schuler. 2005.  Verbnet: A broad-coverage, comprehensive verb lexicon.
  * Warstadt et al. (2019a) Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretic, and Samuel R. Bowman. 2019a.  Investigating BERT’s knowledge of language: Five analysis methods with NPIs.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2877–2887, Hong Kong, China. Association for Computational Linguistics.
  * Warstadt et al. (2019b) Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019b.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641.
  * Wilcox et al. (2018) Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018.  What do RNN language models learn about filler–gap dependencies? In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 211–221, Brussels, Belgium. Association for Computational Linguistics.
  * Wilcox et al. (2019) Ethan Wilcox, Peng Qian, Richard Futrell, Miguel Ballesteros, and Roger Levy. 2019\.  Structural supervision improves learning of non-local grammatical dependencies.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3302–3312, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _ArXiv_ , abs/1910.03771.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in neural information processing systems_ , pages 5754–5764.

##  Appendix A Further analyses

This section contains several additional analyses: principal components for
task performance among the three language models (Tables 5-8); pairwise
comparison between task performance for BERT and GPT-2 (Figures 7 and 8); and
more fine-grained comparisons between word frequency and model performance
(Figures 9 and 10).

PC Number | Transformer-XL | BERT | GPT-2
---|---|---|---
1 | 0.4663 | 0.3865 | 0.4146
2 | 0.7299 | 0.5619 | 0.6873
3 | 0.8511 | 0.7073 | 0.8059
4 | 0.9083 | 0.8034 | 0.8720
5 | 0.9499 | 0.8919 | 0.9175
Table 5: Cumulative proportion of variance explained by the top (of 10) PCs
for each model as detailed in Section 4.1.

Contributor by Rank | PC 1 | PC 2 | PC 3
---|---|---|---
1 | RA ObjRelClauseNoThat - 0.386980 | SV SubjRelClause - 0.449504 | SV SentComp - 0.534710
2 | RA ObjRelClauseThat - 0.376354 | SV ObjRelClauseNoThat - 0.442445 | SV Simple - 0.516031
3 | RA SentComp - 0.359096 | SV ObjRelClauseThat - 0.439015 | RA ObjRelClauseThat - 0.402452
4 | RA Simple - 0.347978 | RA Simple - 0.376192 | SV ObjRelClauseThat - 0.312466

Table 6: Top contributors (tasks) to top few (of 10) PCs for Transformer-XL’s
noun performance as detailed in Section 4.1. Cells contain the task name
followed by their (absolute) component value in the eigenvector.

Contributor | PC 1 | PC 2 | PC 3
---|---|---|---
1 | RA ObjRelClauseNoThat - 0.456096 | SV ObjRelClauseThat - 0.577452 | SV SentComp - 0.686402
2 | RA ObjRelClauseThat - 0.444272 | SV ObjRelClauseNoThat - 0.576572 | SV Simple - 0.499513
3 | RA Simple - 0.383953 | SV PP - 0.353213 | SV SubjRelClause - 0.346437
4 | RA SentComp - 0.383866 | RA Simple - 0.288091 | SV PP - 0.248977

Table 7: Top contributors (tasks) to top few (of 10) PCs for BERT’s noun
performance as detailed in Section 4.1. Cells contain the task name followed
by their (absolute) component value in the eigenvector.

Contributor | PC 1 | PC 2 | PC 3
---|---|---|---
1 | RA ObjRelClauseNoThat - 0.454492 | SV ObjRelClauseNoThat - 0.477923 | SV SentComp - 0.549969
2 | RA Simple - 0.447148 | SV SubjRelClause - 0.444648 | SV Simple - 0.525072
3 | RA SentComp - 0.441366 | SV ObjRelClauseThat - 0.426385 | SV ObjRelClauseThat - 0.478782
4 | RA ObjRelClauseThat - 0.425359 | SV SentComp - 0.385486 | SV ObjRelClauseNoThat - 0.362165

Table 8: Top contributors (tasks) to top few (of 10) PCs for GPT-2’s noun
performance as detailed in Section 4.1. Cells contain the task name followed
by their (absolute) component value in the eigenvector.

Figure 7: BERT: Pairwise comparisons between tasks.

Figure 8: GPT-2: Pairwise comparisons between tasks.

Figure 9: Noun Frequency vs. Model Performance on Subject-Verb Agreement Tasks

Figure 10: Noun Frequency vs. Model Performance on Reflexive Anaphora Tasks


# Dogfight: Detecting Drones from Drones Videos

Muhammad Waseem Ashraf1 Waqas Sultani1 Mubarak Shah2
1Intelligent Machines Lab  Information Technology University  Pakistan
2Center for Research in Computer Vision  University of Central Florida  USA
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

As airborne vehicles are becoming more autonomous and ubiquitous, it has
become vital to develop the capability to detect the objects in their
surroundings. This paper attempts to address the problem of drones detection
from other flying drones. The erratic movement of the source and target
drones, small size, arbitrary shape, large intensity variations, and occlusion
make this problem quite challenging. In this scenario, region-proposal based
methods are not able to capture sufficient discriminative foreground-
background information. Also, due to the extremely small size and complex
motion of the source and target drones, feature aggregation based methods are
unable to perform well. To handle this, instead of using region-proposal based
methods, we propose to use a two-stage segmentation-based approach employing
spatio-temporal attention cues. During the first stage, given the overlapping
frame regions, detailed contextual information is captured over convolution
feature maps using pyramid pooling. After that pixel and channel-wise
attention is enforced on the feature maps to ensure accurate drone
localization. In the second stage, first stage detections are verified and new
probable drone locations are explored. To discover new drone locations, motion
boundaries are used. This is followed by tracking candidate drone detections
for a few frames, cuboid formation, extraction of the 3D convolution feature
map, and drones detection within each cuboid. The proposed approach is
evaluated on two publicly available drone detection datasets and outperforms
several competitive baselines.

Figure 1: A comparison of our approach (PR) with state-of-the-art object
detectors: FCOS (FC) [36], Mask-RCNN (MR) [13], MEGA (ME) [7], SLSA (SL) [39],
and SCRDet (SC) [42]. In this frame (1080$\times$1920), there are four drones
of sizes: 10$\times$15, 11$\times$22, 12$\times$20, 6$\times$17\. The green
bounding box represents the ground truth and output of detectors are shown in
black colors. For the clarity, we have not shown all false positives. The
proposed approach provides better drone localization, reduces false positives
and improves recall.

##  1 Introduction

Drones are actively being used in several daily life applications such as
agriculture [8], wildfire fighting [34], inventory applications [19],
cinematography [16] and surveillance [47, 28]. Due to a large-scale
application of drones, recently computer vision researchers have put forward
several new techniques for object detection [10], tracking [28], agriculture
monitoring [8] and human action recognition [1, 2] in the imagery obtained
through drones. In addition to detecting different objects from a drone video,
it is also important to detect the drone itself from a video captured by
another drone to avoid drone attacks [3], drone collisions [33] and safe
multi-drone flights [24, 32].

Detection of ground objects and aerial drones from drone videos is a very
challenging problem due to a large and abrupt camera motion, arbitrary drone
shape and view changes, occlusion, and more importantly small object size.
Although a lot of recent research has been conducted to detect and track
ground objects and to detect human action using drones [10, 47, 28, 1, 2, 35],
limited work is being done to detect drones from drone videos [24, 33]. To
tackle the problem of drone detection, Li et al., [24] proposed a new drone to
drone detection dataset and employed handcrafted features for background
estimation and foreground moving objects detection. Similarly, Rozantsev et
al., [33] introduced a new challenging dataset of drones and air-crafts. They
employed regression-based approaches to achieve object-centric stabilization
and perform cuboid classification for detection purposes.

Usually flying drones occupy a few pixels in the video frames. For instance,
the average drone size respectively is 0.05% and 0.07% of the average frame
size in drone detection datasets proposed in [24] and [33]. Note that this is
much smaller than that of PASCAL VOC (22.62%) and ImageNet (19.94%). Small
objects including drones usually appear in the cluttered background and are
oriented in different directions which makes its detection quite difficult.
This issue was also pointed out by Huang et al., in [17], where they
demonstrated that the mean Average Precision (mAP) of small objects is much
lower than that of larger objects. Furthermore, the object detection
performance further worsens in the videos [7]. To address this, Noh et al.,
[29] proposed a feature-level super-resolution-based approach that utilizes
high-resolution target features for supervising a low-resolution model.
However, this would require the availability of both low and high-resolution
drone images which are difficult to obtain in drone videos where the drone is
already flying at a far distance. Similarly, Yang et al., [42] employed a
region proposal-based multi-level feature fusion approach to detect small
objects and introduce a new loss function to handle rotated objects. However,
due to very small size objects, less salient, and cluttered backgrounds i.e.,
clouds, buildings, etc., it is difficult to obtain well-localized region
proposals, specifically in drone detection datasets. Through adversarial
learning, Wu et al., [40] proposed to learn domain-specific features employing
metadata (flying altitudes, weather, and view angles). Given the recent low
prices of drones, it is more useful to use an RGB camera for detection and
collision avoidance purposes instead of relying on the expensive hardware for
metadata collection. Authors in [48, 27, 39] proposed to use convolution
feature aggregations across video frames to achieve improved video object
detection. Our experimental results and analysis reveal that although feature
aggregations [48, 27, 39] techniques work well for large video object
detection, for drone detection explicit motion information is more useful.

In this paper, we propose a two-stage segmentation-based approach to detect
drones in cluttered backgrounds. The first stage uses only the appearance cues
while the second stage exploits spatio-temporal cues. Given a video frame, we
divide it into overlapping frame regions. Each frame region is passed through
deep residual networks [14] to obtain the convolution feature maps which are
then followed by pyramid pooling layers [44] to embed the contextual
information. After that pixel-wise and channel-wise attention is employed on
the convolution feature maps to discriminate drone boundaries from the
background and achieve improved drone localization. The purpose of the second
stage is to discover missing detections, remove false detections, and confirm
the true positive detections by employing motion information. To discover the
missing drones, we employ motion boundaries to find probable drone locations.
Given the detections from the first stage and motion boundaries locations, we
track each location forward and backward for a few (eight) frames. After that,
cuboids are extracted across those tracks and are fed to a 3D convolutional
neural network [5] for spatio-temporal feature extraction. This is followed by
pyramid pooling layers. Similar to the first stage, we employ pixel and
channel-wise attention in the second stage as well to get the improved
localization. The proposed approach significantly outperforms several
competitive baselines. In the experimental section, we validate the efficacy
of each step of the proposed approach. The rest of the paper is organized as
follows. Section 2 provides a brief overview of the related developments in
small object detection including drones in video and images. Section 3 deals
with our proposed methodology and Section 4 covers experimental results.
Finally, Section 5 concludes the paper.

Figure 2: Our pipeline is divided into two stages. Stage-1 extracts Resnet50*
features from the overlapping regions of each frame followed by pyramid
pooling to retain global and local contextual information. Channel-wise and
pixel-wise attention help in learning better localization of drones. Resnet50*
refers to the modifications that we have applied (ref Section 3.1). Stage-2
combines spatial information with temporal data of the videos. Detections from
stage-1 along with candidate regions discovered using motion boundaries are
used as candidate regions where UAV can exist. All the proposals are tracked
for 8 frames in a forward and backward manner to generate cuboids of size
224$\times$224$\times$8\. Each cuboid is passed through the I3D network
followed by the attention network to accurately locate drones within each
cuboid. In figure MD, TP, FP, and MB corresponds to missed detection, true
positive, false positive, motion boundaries respectively.

##  2 Related work

Object Detection: Recent years have witnessed tremendous strides in improving
object recognition accuracy [13, 30, 31, 26, 4, 22] on complex benchmark
datasets. Law et al. [22] proposed a single-stage network where they detect an
object bounding box as a pair of key points. Also, the authors introduced a
novel corner pooling scheme for better corners localization. To address the
foreground-background class imbalance in object detection datasets, Lin et al.
[26] presented a modified cross-entropy loss that reduces the weights of the
loss assigned to well-classified examples. Similarly, Li et al., [25] pointed
out the underperformance of object detectors for long-tail distribution
datasets. They proposed a novel balanced group-softmax to modulate the
training process and ensure that the classifier is sufficiently trained for
all the classes. Cai et al., [4] put forward an idea of multi-stage object
detection where a sequence of detectors trained with increasing intersection
over union (IoU) thresholds to achieve improved object localization. To
further improve object detection accuracy, authors in [12] introduced a new
feature fusion technique through multi-scale features summation in a feature
pyramid. Although impressive detection results, most of these detectors have
low detection and localization accuracy for small objects.

Small Object Detection: To address the problem of small, cluttered, and
oriented objects, Li et al., [23] proposed to use generative adversarial
networks to decrease the gap of feature representation of small and large
objects. To further improve feature representation for small objects, authors
in [29] proposed to jointly use the low and high-resolution feature maps of
the same images during the training by matching the relative receptive fields
between the pairs. Yang et al., [42] proposed a multi-layer feature fusion
technique along with attention networks to achieve improved object detection
for cluttered and rotated objects. However, most of these methods are region
proposal based which are unable to capture sufficient foreground-background
information specifically when the objects are extremely small (such as 0.05%
or 0.07% of the average image size as in the case of drone detection). A
related problem to ours is the detection of ground objects from drone images
such as [41] which proposed to detect clustered objects in aerial images.
However, while detecting drones from a drone, target objects (drones) undergo
fast, abrupt, and highly unpredictable movements as compared to objects (car,
human) on the ground. Target objects can have arbitrary changing shapes due to
their rotation around any axis as compared to the restricted movements of
ground objects. Furthermore, as compared to ground objects, drones have much
smaller sizes, and the target objects appear/disappear more frequently due to
clouds.

Video Object Detection: Historically, taking a page from object detection,
several video-based object detectors were introduced, including [48, 27, 39,
7]. The object detection accuracy degrades in the video due to several factors
including motion blur, occlusion, and out of focus, etc,. To address this Zhu
et al., [48] proposed to include the feature representation of each frame by
aggregation of nearby frames features using optical flow. Authors in [27]
improve feature aggregation speed using the Bottleneck-LSTM layer. Instead of
aggregating features from nearby frames, Wu et al., [39] proposed to use whole
sequence-level features aggregation for improved video object detection.
Recently, Chen et al., [7] utilized both the global semantic and local
localization information employing memory aggregation network and have shown
impressive results on several video object detection benchmarks. Although
impressive results, most of these methods are tested on standard video object
detection datasets where the object covers a significant portion of the video
frame. Furthermore, motion is much more complex in drone videos as compared to
standard video object detection datasets.

Drones Detection: Recent low prices of drones will bring more and more UAVs to
the sky. To keep the prices and weight of UAVs small, it is useful to detect
other flying drones using simple RGB cameras instead of expensive radar
systems. Therefore researchers have addressed the problem of drone detection
for different applications. After obtaining a large number of spatio-temporal
(s-t) tubes at different spatial resolutions, Rozantsev et al., [33] employs
two CNN models to obtain coarse and fine motion stabilization in each s-t tube
respectively. Next, they obtain UAVs detection by classifying each s-t tube
using a third CNN. There are several differences between our method and [33].
1) Instead of using computationally expensive region-based sliding windows at
several scales, we employ an efficient fully convolutional segmentation-based
approach, 2) as compared to [33], our approach does not need perfectly drone-
centered cuboids and learn rich spatio-temporal information using I3D, and 3)
we employ attention networks to improve feature representation for improved
localization. Similarly, authors in [24, 43] detected moving drones by
subtracting background images and then identified UAVs using deeply learned
classifiers. Furthermore, they use Kalman filtering to get improved detection.
Their method uses a lot of parameters and thresholds which make it
irreproducible. Also, their reliance on background subtraction for moving
object detection produces a lot of false positives. Instead of using RGB
images, some researchers have also proposed to employ depth maps to achieve 3D
localization [6, 38]. However, depth maps are quite expensive to obtain in
real-world scenarios and add the extra payload to UAVs.

##  3 Proposed method

Our goal is to detect and localize drones in the video frames which are
captured by other flying drones. Our proposed approach to tackle this
challenging problem is based on the three observations: (1) due to very small
drone size, region proposal based method may not be able to capture enough
discriminative foreground-background information, therefore the bottom-up
segmentation based approach which classifies each pixel is preferable; (2) the
model should learn subtle visual differences between a drone and the
background (clouds, etc); (3) due to large abrupt motion of target and source
drones, the feature aggregation methods may not be sufficient and we need to
use explicitly optical flow information as has been successfully employed in
several action recognition works [11]. In the following, we first discuss
details of the segmentation network (Sec. 3.1), followed by the attention
networks used to get improved localization (Sec 3.2). Finally, we discuss how
we use motion information to discover missing detection and thus improve
recall.

###  3.1 Stage-1: Exploiting Spatial Cues

We start with appearance-based pixel classification to accurately localize
drones. For spatial feature computation, we resort to deep residual networks
[14]. However, given the extremely small size of drones (0.05% or 0.07% of
image size), obtaining good discriminative features by utilizing the whole
images is not possible. The standard 2D CNN networks such as Resnet50 require
a fixed size input image (473$\times$473). Therefore image resizing from high-
resolution image to low resolution (1080$\times$1920 to 473$\times$473) for
feature computation can further decrease the spatial resolution of the drone
to one or two pixels. Secondly, as the network goes deeper, we lose local
information. To address this, we use two steps: 1) To avoid resizing the
image, we divide each frame into overlapping regions, 2) We modify Resnet50 to
keep local information intact as we go deep in the network. Specifically, we
extract features from all four blocks of Resnet50 [14] and concatenate them
together after spatial resizing of the first block to avoid dimension
mismatch. Finally, we use 1$\times$1 convolution to get back the original
dimension. We call modified Resnet50 as Resnet50*. Inspired by the use of
pyramid pooling [45] in several applications we employ pyramid pooling in our
framework. Specifically, after obtaining features from Resnet50*, we apply the
pyramid pooling using four different kernel sizes and concatenate those multi-
scale features after up-sampling.

In experiments, we observe that although the above network provides decent
drone detection, in several cases, it is unable to accurately detect and
localize the drones. Therefore, to make feature maps more focused on the
foreground, we use pixel and channel-wise attentions. Pixel and channel-wise
attention networks are described in the next section.

###  3.2 Attention networks

Assuming the drone of size 16$\times$11 (the average drone size in [24]), by
missing only a few pixels from both sides, the intersection over union (IOU)
drops to below 0.5. Therefore, it is crucial to get accurate localization for
true drone detection. To achieve this, we introduce detailed pixel-wise and
channel-wise attention on the convolution feature maps. Recently, several
attention networks [37, 15, 42, 9] have been introduced for different computer
vision applications.

Channel-wise attention: Inspired by [15], we use a channel-wise attention
network to automatically learn to give more weights to informative feature
channels and suppress less informative ones. The architectural details of
channel-wise attention networks are given in Figure 3 (a). This attention is
achieved through channel-wise multiplication of attention vector with
convolution feature maps.

Figure 3: Architectural details of (a) channel-wise and (b) pixel-wise
attention network, where ‘FC’ represents fully connected layer with number of
units in (a), ‘C’ and ‘F’ represent convolution and number of filters
respectively in (b).

Pixel-wise attention: Similar to channel-wise attention vector, we generate
pixel-wise attention matrix to assign more weights to spatial location which
corresponds to drones and less weight to non-drone regions similar to [9, 42].
The architectural details of the pixel attention network are given in Figure 3
(b). To suppress background information, we perform element-wise
multiplication of pixel attention mask with all convolution maps channels.
This is followed by the addition of attention masks to give high weights to
regions containing useful information.

In the experiments, we observe that attention networks significantly help in
achieving better drone localization. Note that the complete stage-1 is trained
end-to-end where attention is learned automatically through network
architecture, training data, and losses. Figure 4 demonstrates the difference
in the network outputs which training with and without attention networks.

Figure 4: Role of attention. (a) Input image. (b) without and (c) with
attention networks. Top two rows shows the examples where attention networks
helps the network to learn to give more weights to the pixels associated with
drones and the last row represents the case where attention network suppress
the non-drone pixels.

###  3.3 Losses

Drone detection datasets have two major challenges: There exists a large drone
versus non-drone class imbalance i.e., the majority of pixels belong to the
background and only a few pixels (if any) occupy the drone. Secondly, due to
extremely small size drones, the only difference of even 1 or 2 pixels between
the detected box and ground truth brings down the IoU score to less than 0.5.
Therefore, we use multiple losses to train our network. Specifically, to
address the class imbalance, focal loss [26] is used. and Distance-IOU loss
[46] is employed to achieve better IOU localization. Distance-IOU not only
minimizes the IOU between the ground truth and detected bounding boxes but
also reduces the distance between centers of two boxes. Finally, we use
smooth-L1 loss [31] to jointly train pixel-wise attention network as shown in
Figure 3 (b).

###  3.4 Stage2: Exploiting Spatiotemporal Cues

The purpose of this stage is to confirm true detections, reject the false
detections and discover the missing detections of the stage-1. To find out the
new probable drone locations, we use motion gradients which is explained
below.

Motion boundaries: Drone can be characterized by locations undergoing motion.
However, since drone detection datasets involve moving camera, simple optical
flow magnitude cannot be much useful. Therefore, we propose to use optical
flow gradients to capture the change in motion. Specifically, given every
three frames of a video, we first stabilize them using key-points detection
and then compute forward and backward optical flow. After that maximum motion
gradients across all three frames are computed as follows:

$\displaystyle G=max(\sqrt{u_{x}^{2}+u_{y}^{2}},\sqrt{v_{x}^{2}+v_{y}^{2}}),$
(1) $\displaystyle M=max(G_{0\rightarrow 1},G_{1\rightarrow 2},G_{2\rightarrow
1},G_{1\rightarrow 0}),$ (2)

where $u_{x}$, $v_{x}$, $u_{y}$, and $v_{y}$ are optical flow gradients along
x and y axis respectively, $M$ show motion boundaries and $G_{0\rightarrow
1},G_{1\rightarrow 2},G_{2\rightarrow 1},G_{1\rightarrow 0}$ represents motion
gradients between frames, 0→1, 1→2, 2→1, 1→0 respectively.

There are two limitations of motion boundaries: The motion boundaries provide
high magnitude across the boundaries of drones and in most cases, do not
completely cover drones. Secondly, due to the underlying approximation of
optical flow calculation, usually, the maximum of optical flow gradient
magnitude does not match exactly with the moving drone. To address these
issues, we dilate the motion boundaries and then apply conditional random
field [20] to get the better localization of candidate drone regions.

Cuboids formation: Given detections from stage-1 and newly discovered
locations obtained using motion boundaries, our next step is to extract
spatio-temporal features from all candidate drone locations. To this end, we
initialize the correlation tracker at each candidate location (including
stage-1 detections and newly discovered locations). Due to small drone and
complex camera motions, trajectories tend to drift away from their initial
location within a few frames, therefore, we restrict trajectory length to
eight frames. Specifically, given the candidate drone location, tracking is
done three frames forward and four frames backward. Note that tracking is done
after the motion stabilization of the corresponding eight frames. To capture
contextual information across candidate location and to compensate for the
trajectory drifting, N $\times$ N patches are extracted from video frames
across each track, resulting in a cuboid of size N$\times$N$\times$8\.
Finally, to extract spatio-temporal features from each cuboid, we employ the
Inflated-3D (I3D) network [5]. We choose I3D due to its fast speed, small
memory consumption, and excellent capability of capturing detailed spatio-
temporal characteristics. To make the size of the cuboid consistent with that
of the standard I3D network input dimensions, we use bilinear interpolation on
each patch to resize the cuboid from N$\times$N$\times$8 to
224$\times$224$\times$8\. 3D Convolution features are extracted from the third
last layer of the I3D network which has dimensions of
14$\times$14$\times$480\. To have consistency with the stage-1, we use
bilinear interpolation to resize the feature maps to 60
$\times$60$\times$480\. This is followed by 2D convolution layers to convert
60$\times$60$\times$480 to 60$\times$60$\times$2048 feature maps.
Experimentally, we have also tried patch superresolution and feature map
superresolution instead of resizing using bilinear interpolation, however, we
did not observe any performance improvement.

Finally, spatio-temporal convolution feature maps of each cuboid are
aggregated across different scales using spatial pyramid pooling. This is
followed by attention networks and network losses as discussed in Section 3.2

Figure 5: Samples frames from NPS-drone [24] dataset. The green boxes enclose
drones.

##  4 Experiments

The main goal of our experiments is to verify that the proposed approach
accurately localize drone in moving camera videos. To this end, we perform
extensive experiments on two challenging drone datasets and compare our
approach with several competitive approaches, and analyze different components
of the proposed approach.

###  4.1 Implementation details

In this section, we provide implementation details of our approach. During
stage-1, we divide each frame into nine overlapping patches for NPS-Drones
dataset [24] and into four overlapping patches for FL-Drones dataset [33]. For
both datasets, stage-1 is trained end to end from scratch, while for stage-2,
a pre-trained I3D network with frozen weights is used for feature extractions.
For the NPS-Drones dataset, a fixed 100$\times$100 patch size is extracted
across the drone to make a cuboid while in the FL-Drones dataset size of the
patch corresponds to the size of the drone. In all the experiments, we use
Adam [18] optimizer with an initial learning rate of 0.001 without the decay
parameter.

A simple correlational tracker is employed during cuboid formation. We remove
candidate locations (provided by motion boundaries), which are smaller or
bigger than the maximum and minimum size of drones in the training data
respectively. To improve training, we have also used hard-negative mining. In
a post-processing step, we remove detections that only appear for a single
frame.

Evaluation metrics: Following the related works, we evaluate the performance
of our approach and baselines using precision, recall, F1-score, and average
precision (AP) where every frame is treated as an individual image for
evaluation. We evaluate all the methods on every 4th frame of the testing
data.

Figure 6: Samples frames from FL-drone [33] dataset. The green boxes enclose
drones.

###  4.2 Datasets

We evaluate our approach on two drones datasets. Each of them is briefly
introduced below.

NPS-Drones [24]: This dataset is published by Naval Postgraduate School (NPS)
and is publicly
available111https://engineering.purdue.edu/~bouman/UAV_Dataset/. The dataset
contains 50 videos that were recorded at HD resolutions (1920$\times$1080 and
1280$\times$760) using GoPro-3 cameras mounted on a custom delta-wing
airframe. The minimum, average and maximum size of drones are 10 $\times$ 8,
16.2 $\times$ 11.6, and 65 $\times$ 21, respectively. The total number of
frames in the dataset are 70250. In the experiments, the first 40 videos are
used for training and validation and the last 10 videos are used for the
testing.

Figure 7: This figure shows the variability of drone shape and size in two
datasets: NPS-drone [24] (first two rows) and FL-drone [33] (last two rows)
datasets. The green boxes present the ground truth bounding boxes.

FL-Drones [33]: The second drone dataset used in the paper was introduced by
Rozantsev et al. [33]. This dataset is quite challenging due to extreme
illumination, pose, and size changes. This dataset contains indoor and outdoor
samples and the flying drones have variable shapes, and their shape is barely
retained even in consecutive frames. Drones get mixed up in the background due
to small size and intense lighting conditions coupled with complex background.
The minimum, average, and maximum sizes of drones are: 9$\times$9,
25.5$\times$16.4, and 259$\times$197, respectively and frames resolutions are
640$\times$480 and 752$\times$480\. This dataset contains 14 videos with a
total of 38948 frames. As suggested by the authors [33], half of the data is
used for the training, and the other half is used for the testing.

Sample frames and drones from both datasets are shown in Figure 5, Figure 6
and, Figure 7. Note that there are several frames in both datasets without any
drone. It is interesting to know that the majority of original annotations
released by the authors of [33, 24] are not precise and bounding boxes are
much bigger than that of actual drones. To address this, we have re-annotated
both datasets again. Given that number of frames in both datasets are more
than 100K, re-annotation took significant time. The examples of improved
annotations are shown in Figure 8. These improved annotations will be
released.

Figure 8: In the figure, we show the examples of our improved annotations. The
blue boxes show the originally released annotations by the authors and the red
boxes show our annotations done. The first three columns show the images from
the NPS-drone dataset [24] and the last three columns are images from the FL-
drone dataset [33]. Under each image, we show the IOU between new and old
annotations.

Method |  Precision |  Recall |  F1 score |  AP
---|---|---|---|---
SCRDet-H [42] | 0.81 | 0.74 | 0.77 | 0.65
SCRDet-R [42] | 0.79 | 0.71 | 0.75 | 0.61
FCOS [36] | 0.88 | 0.84 | 0.86 | 0.83
Mask-RCNN [13] | 0.66 | 0.91 | 0.76 | 0.89
MEGA [7] | 0.88 | 0.82 | 0.85 | 0.83
SLSA [39] | 0.47 | 0.67 | 0.55 | 0.46
Proposed | 0.92 | 0.91 | 0.92 | 0.89

Table 1: Quantitative comparison of the proposed approach with several state
of the art approaches on the NPS dataset [24].

.

Method |  Precision |  Recall |  F1 score |  AP
---|---|---|---|---
SCRDet-H [42] | 0.54 | 0.62 | 0.58 | 0.52
SCRDet-R [42] | 0.55 | 0.62 | 0.58 | 0.52
FCOS [36] | 0.69 | 0.70 | 0.69 | 0.62
Mask-RCNN [13] | 0.76 | 0.68 | 0.72 | 0.68
MEGA [7] | 0.71 | 0.72 | 0.71 | 0.65
SLSA [39] | 0.57 | 0.72 | 0.64 | 0.61
Proposed | 0.84 | 0.76 | 0.80 | 0.72

Table 2: Quantitative comparison of the proposed approach with several state
of the art approaches on the FL-Drones dataset [33].

.

###  4.3 Comparison with the State-of-the-art

We compare the proposed approach with recently proposed approaches such as
fully convolutional one-stage object detector [36], cluttered and rotated
small object detector [42], instance segmentation [13], and video object
detectors [7, 39]. For comparison purposes, all the baseline methods are
finetuned using the pre-trained weights available with public codes.
Specifically, SLSA [39], FCOS [36], MEGA [7], and SCRDet [42] are finetuned
using corresponding papers’ pretrained weights and MASK-RCNN [13] is fine-
tuned using Resnet50 weights trained on ImageNet. I3D has the weights obtained
through training on Kinetics. Due to computational power limitations, all of
the methods that we compared with are trained for iterations ranging from 80,
000 to 100, 000 using a single Nvidia 1080Ti GPU.

The quantitative comparison of proposed approach with the baselines over both
drone detection datasets are shown in Table 1 and Table 2. The results
demonstrate that our approach significantly outperforms the recent baselines
on different evaluation metrics.

###  4.4 Ablation studies

Component analysis: We analyze different components of the proposed approach
to verify their effectiveness. The experimental results in Table 3 enforce
that each component of our approach is important and contributes toward final
accuracy.

Method |  Precision |  Recall |  F1 score
---|---|---|---
CE | 0.87 | 0.82 | 0.85
$\mathcal{L}$ | 0.88 | 0.84 | 0.86
$\mathcal{L}$\+ $\mathcal{CA}$ | 0.88 | 0.86 | 0.87
$\mathcal{L}$\+ $\mathcal{PA}$ | 0.89 | 0.85 | 0.87
Stage-1 | 0.92 | 0.88 | 0.90
Stage-1+Stage-2 | 0.92 | 0.91 | 0.92
Table 3: Ablation study of different components of our method on NPS dataset.
The first row shows the proposed approach employing only cross-entropy loss
(CE). The second row represents results incorporating focal loss and Distance-
IOU loss ($\mathcal{L}$). The third row demonstrates results employing
channel-wise attention ($\mathcal{L}$\+ $\mathcal{CA}$). After that we show
results using pixel-wise attention ($\mathcal{L}$\+ $\mathcal{PA}$). Finally,
the last two rows show results after stage-1 and stage-2 respectively.

Two Stage Approach: As shown in Table 3 and Fig 9, we observe that two-stage
approach produce much better detection results as compared to 1-stage approach
by discovering and classifying difficult drone locations. We have also tried
multiple frames in 1-stage similar to [21] but achieved very low detection
accuracy (below F1-score=40). We believe that due to the large and complex
target and source drone motion, the 1-stage multiple frames approach is unable
to learn to detect drones.

Failure cases: Since the proposed approach tries to detect drones based on
appearance and motion cues, the drones with slow motion and indistinguishable
shape are hard to detect. Figure 10 show failure cases of our approach.

Figure 9: Qualitative comparison of one stage versus two stage detection
results. (a) shows the detection results for stage-1 and (b) represents the
detection results of two stage approach. Red boxes represent detection and
blues boxes are just for better visualizations. The first stage misses one
drone in each example. Figure 10: Failure cases of our method. Miss detections
and correct detections are shown by green and red boxes respectively. Outer
boxes are just for better visualizations. The drones with indistinguishable
shape and motion are hard to detect. (a) and (b) represents samples from NPS-
Drones [24] and FL-Drones [33] datasets respectively.

##  5 Conclusion

One of the most important challenges associated with drone use is collision
avoidance or safe multi-drone flights. Therefore, it is crucial to develop
robust computer vision methods that can detect and enable collision avoidance
using inexpensive cameras. We have presented a two-stage approach for drone
detection from other flying drones employing spatio-temporal cues. Instead of
relying on region proposal-based methods, we have used a segmentation-based
approach for accurate drone detection using pixel and channel-wise attention.
In addition to using appearance information, we have also exploited motion
information between frames to get a better recall. We observe that for drones
to drones videos, the two-stage approach performs better than the one-stage
approach. Thorough comparisons with the state of the arts and detailed
ablation studies validate the framework and the ideas proposed in this work.

Acknowledgements: We would like to thank Mohsen Ali and Sarfaraz Hussein for
their insightful discussions.

## References

  * [1] D. H. Arjun Nagendran and M. Shah.  Ucf-arg data set.
  * [2] M. Barekatain, M. Martí, H. Shih, S. Murray, K. Nakayama, Y. Matsuo, and H. Prendinger.  Okutama-action: An aerial view video dataset for concurrent human action detection.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2017, Honolulu, HI, USA, July 21-26, 2017, 2017.
  * [3] K. L. Best, J. Schmid, S. Tierney, J. Awan, N. M. Beyene, M. A. Holliday, R. Khan, and K. Lee.  How to Analyze the Cyber Threat from Drones: Background, Analysis Frameworks, and Analysis Tools.  RAND Corporation, Santa Monica, CA, 2020.
  * [4] Z. Cai and N. Vasconcelos.  Cascade r-cnn: Delving into high quality object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6154–6162, 2018.
  * [5] J. Carreira and A. Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017.
  * [6] A. Carrio, S. Vemprala, A. Ripoll, S. Saripalli, and P. Campoy.  Drone detection using depth maps.  In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1034–1037. IEEE, 2018.
  * [7] Y. Chen, Y. Cao, H. Hu, and L. Wang.  Memory enhanced global-local aggregation for video object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10337–10346, 2020.
  * [8] M. T. Chiu, X. Xu, Y. Wei, Z. Huang, A. G. Schwing, R. Brunner, H. Khachatrian, H. Karapetyan, I. Dozier, G. Rose, D. Wilson, A. Tudor, N. Hovakimyan, T. S. Huang, and H. Shi.  Agriculture-vision: A large aerial image database for agricultural pattern analysis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [9] J. Choe and H. Shim.  Attention-based dropout layer for weakly supervised object localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [10] D. Du and P. Z. et al.  Visdrone-det2019: The vision meets drone object detection in image challenge results.  In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), 2019.
  * [11] C. Feichtenhofer, A. Pinz, R. P. Wildes, and A. Zisserman.  What have we learned from deep representations for action recognition?  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [12] C. Guo, B. Fan, Q. Zhang, S. Xiang, and C. Pan.  Augfpn: Improving multi-scale feature learning for object detection, 2019\.
  * [13] K. He, G. Gkioxari, P. Dollár, and R. Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [14] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [15] J. Hu, L. Shen, and G. Sun.  Squeeze-and-excitation networks.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7132–7141, 2018.
  * [16] C. Huang, F. Gao, J. Pan, Z. Yang, W. Qiu, P. Chen, X. Yang, S. Shen, and K. Cheng.  Act: An autonomous drone cinematography system for action scenes.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7039–7046, 2018.
  * [17] J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama, and K. Murphy.  Speed/accuracy trade-offs for modern convolutional object detectors.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [18] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [19] W. Kwon, J. H. Park, M. Lee, J. Her, S. H. Kim, and J. W. Seo.  Robust autonomous navigation of unmanned aerial vehicles (uavs) for warehouses’ inventory application.  IEEE Robotics and Automation Letters, 2020.
  * [20] J. Lafferty, A. McCallum, and F. Pereira.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.  In Proc. 18th International Conf. on Machine Learning, 2001.
  * [21] R. LaLonde, D. Zhang, and M. Shah.  Clusternet: Detecting small objects in large scenes by exploiting spatio-temporal information.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4003–4012, 2018.
  * [22] H. Law and J. Deng.  Cornernet: Detecting objects as paired keypoints.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 734–750, 2018.
  * [23] J. Li, X. Liang, Y. Wei, T. Xu, J. Feng, and S. Yan.  Perceptual generative adversarial networks for small object detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [24] J. Li, D. H. Ye, T. Chung, M. Kolsch, J. Wachs, and C. Bouman.  Multi-target detection and tracking from a single camera in unmanned aerial vehicles (uavs).  In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016.
  * [25] Y. Li, T. Wang, B. Kang, S. Tang, C. Wang, J. Li, and J. Feng.  Overcoming classifier imbalance for long-tail object detection with balanced group softmax.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [26] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [27] M. Liu and M. Zhu.  Mobile video object detection with temporally-aware feature maps.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5686–5695, 2018.
  * [28] M. Mueller, N. Smith, and B. Ghanem.  A benchmark and simulator for uav tracking.  In B. Leibe, J. Matas, N. Sebe, and M. Welling, editors, Computer Vision – ECCV 2016. Springer International Publishing, 2016.
  * [29] J. Noh, W. Bae, W. Lee, J. Seo, and G. Kim.  Better to follow, follow to be better: Towards precise supervision of feature super-resolution for small object detection.  In Proceedings of the IEEE International Conference on Computer Vision, pages 9725–9734, 2019.
  * [30] J. Redmon and A. Farhadi.  Yolov3: An incremental improvement.  arXiv preprint arXiv:1804.02767, 2018.
  * [31] S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Advances in neural information processing systems, pages 91–99, 2015.
  * [32] A. Rozantsev, V. Lepetit, and P. Fua.  Flying objects detection from a single moving camera.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [33] A. Rozantsev, V. Lepetit, and P. Fua.  Detecting flying objects using a single moving camera.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(5):879–892, 2017.
  * [34] D. Saikin, T. Báča, M. Gurtner, and M. Saska.  Wildfire fighting by unmanned aerial system exploiting its time-varying mass.  IEEE Robotics and Automation Letters, PP:1–1, 02 2020.
  * [35] W. Sultani and M. Shah.  Human action recognition in drone videos using a few aerial training examples, 2020.
  * [36] Z. Tian, C. Shen, H. Chen, and T. He.  Fcos: Fully convolutional one-stage object detection.  In Proceedings of the IEEE international conference on computer vision, pages 9627–9636, 2019.
  * [37] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, 2017.
  * [38] M. Vrba, D. Heřt, and M. Saska.  Onboard marker-less detection and localization of non-cooperating drones for their safe interception by an autonomous aerial system.  IEEE Robotics and Automation Letters, 4(4):3402–3409, 2019.
  * [39] H. Wu, Y. Chen, N. Wang, and Z. Zhang.  Sequence level semantics aggregation for video object detection.  In Proceedings of the IEEE International Conference on Computer Vision, pages 9217–9225, 2019.
  * [40] Z. Wu, K. Suresh, P. Narayanan, H. Xu, H. Kwon, and Z. Wang.  Delving into robust object detection from unmanned aerial vehicles: A deep nuisance disentanglement approach.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
  * [41] F. Yang, H. Fan, P. Chu, E. Blasch, and H. Ling.  Clustered object detection in aerial images.  In The IEEE International Conference on Computer Vision (ICCV), October 2019.
  * [42] X. Yang, J. Yang, J. Yan, Y. Zhang, T. Zhang, Z. Guo, X. Sun, and K. Fu.  Scrdet: Towards more robust detection for small, cluttered and rotated objects.  In Proceedings of the IEEE International Conference on Computer Vision, pages 8232–8241, 2019.
  * [43] D. H. Ye, J. Li, Q. Chen, J. Wachs, and C. Bouman.  Deep learning for moving object detection and tracking from a single camera in unmanned aerial vehicles (uavs).  Electronic Imaging, 2018:4661–4666, 01 2018.
  * [44] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia.  Pyramid scene parsing network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881–2890, 2017.
  * [45] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia.  Pyramid scene parsing network.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6230–6239, 2017.
  * [46] Z. Zheng, P. Wang, W. Liu, J. Li, R. Ye, and D. Ren.  Distance-iou loss: Faster and better learning for bounding box regression.  volume 34, pages 12993–13000, 02 2020.
  * [47] P. Zhu, L. Wen, X. Bian, H. Ling, and Q. Hu.  Vision meets drones: A challenge.  2018\.
  * [48] X. Zhu, Y. Wang, J. Dai, L. Yuan, and Y. Wei.  Flow-guided feature aggregation for video object detection.  In Proceedings of the IEEE International Conference on Computer Vision, pages 408–417, 2017.