# Mapping county-level mobility pattern changes in the United States in
response to COVID-19

Song Gao 1 Corresponding Email<EMAIL_ADDRESS>Jinmeng Rao1 Yuhao Kang1
Yunlei Liang1 Jake Kruse1
1GeoDS Lab  Department of Geography  University of Wisconsin-Madison  WI 53706
USA

###### Abstract

To contain the COVID-19 epidemic, one of the non-pharmacological epidemic
control measures is reducing the transmission rate of SARS-COV-2 in the
population through social distancing. An interactive web-based mapping
platform that provides timely quantitative information on how people in
different counties and states reacted to the social distancing guidelines was
developed by the GeoDS Lab @UW-Madison with the support of the National
Science Foundation RAPID program. The web portal integrates geographic
information systems (GIS) and daily updated human mobility statistical
patterns (median travel distance and stay-at-home dwell time) derived from
large-scale anonymized and aggregated smartphone location big data at the
county-level in United States. It aims to increase risk awareness of the
public, support data-driven public health and governmental decision-making,
and help enhance community responses to the COVID-19 pandemic.

##  1 Introduction

The coronavirus disease (COVID-19) pandemic is a global threat with escalating
health, economic and social challenges. As of May 12, 2020, there had been
1,342,594 total confirmed cases and 80,820 total deaths in the U.S. according
to the reports of the U.S. Centers for Disease Control and Prevention (CDC)
111https://www.cdc.gov/coronavirus/2019-ncov/cases-updates/cases-in-us.html.
To contain the COVID-19 epidemic, one of the non-pharmacological epidemic
control measures is reducing the transmission rate of SARS-COV-2 in the
population through (physical) social distancing. An interactive web-based
mapping platform222https://geods.geography.wisc.edu/covid19/physical-
distancing/ (as shown in Fig. 1) that provides timely quantitative information
on how people in different counties and states reacted to the social
distancing guidelines and followed stay-at-home mandates was developed by the
GeoDS Lab at the University of Wisconsin-Madison with the support of the
National Science Foundation. It integrates geographic information systems
(GIS) and daily updated human mobility statistical patterns (median travel
distance and stay-at-home dwell time) derived from large-scale anonymized and
aggregated smartphone location big data at the county-level in United States
[23, 21, 18, 14]. This mobility dashboard aims to increase risk awareness of
the public, support data-driven decision-making, and help enhance community
responses to the COVID-19 pandemic.

Figure 1: The web interface of the mapping platform.

With the rapid development of information, communication, and technologies,
new data acquisition and assessment methods are available to help evaluate the
risk awareness of epidemic transmission and geographic spreading from the
community perspective. The online mobility mapping platform can help
effectively monitor not only the spatiotemporal changes of travel distance and
stay-at-home time of people as well as the effect of social distancing
policies on human movement behaviors. It can also inform the analysis of
potential influential factors on the compliance with COVID-19 social
distancing orders such as self-report beliefs, social disparities, political
and geographic contexts in risk communication and public-health interventions
[11, 2, 17, 12]. Recent studies have utilized various sources (e.g., Google
mobility reports) of mobile location big data to quantify mobility change
patterns and its relation to COVID-19 spread, air pollution, and economic
impacts [3, 7, 22, 10, 13, 9]. In the following, we first introduce the data
sources and the design principles and techniques to develop the mobility
tracking Web portal in detail. Then, we describe the observed human mobility
changes in the U.S. in three periods: before stay-at-home orders, after stay-
at-home mandates in place, and the economy partial reopening period.

##  2 Methods

###  2.1 Data Sources

In response to the rapid spread of the SARS-COV-2 virus, as of April 7, 2020,
42 states and Washington D.C. in the U.S. have issued the statewide “stay-at-
home” or “safer-at-home”
orders333https://www.nytimes.com/interactive/2020/us/coronavirus-stay-at-home-
order.html. People generally tend to change their travel behavior to slow down
the spread of the novel coronavirus but geographic and temporal heterogeneity
still exists. To quantitatively understand how people reacted to the stay-at-
home orders imposed during the COVID-19 outbreak, human mobility changes are
considered in terms of changes in median travel distance and stay-at-home
dwell time.

We first use the U.S. mobility data released by the Descartes Labs [21] to map
the daily travel changes. The individual mobility is measured by the maximum
travel distance (km) to a location from the initial location of the day given
a unique mobile device (i.e., individual max-distance mobility). To
investigate the mobility changes in a geographic region, a baseline was first
determined [21], which was defined as the median of the max-distance mobility
on the weekdays between 2/17/2020 and 3/7/2020 in the specified region (i.e.,
by county). By comparing daily mobility with baseline mobility, we can measure
how people in each county react to COVID-19 by reducing their daily travel
distance:

Median Travel Distance: The median of the individual daily maximum travel
distance for all location samples in the specified region.

Percent Change in Mobility: The percentage change in the daily median mobility
from the baseline.

In addition, we use the SafeGraph “Social Distancing Metrics”
data444https://docs.safegraph.com/docs/social-distancing-metrics to compute
the Median of Stay-at-Home Dwell Time for understanding people’s compliance to
the stay-at-home orders in each county across the nation. Fig. 2 shows the
percent of the population staying home (sheltering in place) given a
particular date. Data preprocessing is needed to determine the common
nighttime location (i.e., home location) of each mobile device over a 6-week
period to a Geohash-7 granularity (153m x 153m). The devices are aggregated by
home census block group and the metrics are further computed to the county
level. SafeGraph data provide unique and valuable insights into the foot-
traffic changes to large-scale businesses and consumer POIs across the nation
[20]. To measure stay-at-home dwell time, the home place for each individual
is identified and the hours/minutes for all sampled devices staying at that
home place across the day are summed up. To measure dynamic changes in home
dwell time, the median home dwell time for all observed devices is computed in
different geographical units (e.g., census block groups, counties, and
states).

Figure 2: The stay-at-home dwell time data provided by SafeGraph.

###  2.2 System Design

Figure 3: The mobility tracking system design diagram.

The interactive web mapping platform is designed and developed using the
ArcGIS Operational Dashboards555https://doc.arcgis.com/en/dashboards/. It
integrates data streams with maps, time series plots, gauges, and other visual
elements to represent the dynamic status of human mobility patterns
comprehensively. The system design diagram for the mobility tracking dashboard
is shown in Fig. 3.

Our mobility tracking dashboard consists of three layers: Data and Feature
layer, Methodology layer, and Application layer. At the Data and Feature
layer, we first acquire the abovementioned travel distance and home dwell time
mobility data from the two data providers and preprocess them (e.g., data
cleaning, data quality check). Then we aggregate the processed datasets to the
county level and match them to the U.S. county map (using the spatial join
operator). Mobility indicators such as the median travel distance mobility
index and the median home dwell time are calculated during this process.
Finally, these datasets are formatted and uploaded to ArcGIS Online Content as
web items, which are then organized as web maps, feature layers, and time
series tables for publication. The whole process is iterating regularly to
keep the data up to date.

At the Methodology layer, we apply the functionality design (e.g., map
visualization, data query, time-series data plotting) and the interface design
(e.g., component design, interaction design). These two parts are internally
associated to help users better access, understand, and interact with the
data. At the Application layer, the system can further support various
mobility-related applications such as mobility pattern recognition,
monitoring, analysis, modeling, prediction, and decision support. For
instance, the mobility change data product has been used in epidemic modeling
[5, 9] and in air pollution association analysis [13]. Furthermore, to
optimize the mobile user interface and improve user experience [19], we
employed another design tool – the ArcGIS Experience
Builder666https://www.esri.com/en-us/arcgis/products/arcgis-experience-
builder/overview to construct mapcentric apps and display them on a scrolling
and multi-panel screen for mobile devices.

##  3 Mobility Changes and Insights

On the mobility tracking map, the red color counties represent an INCREASE in
daily mobility compared with the abovementioned baseline statistic, while blue
color counties represent a DECREASE in mobility. The color saturation reflects
the degree of changes. The darker the more mobility changes in a county. In
the following, we discuss the observed mobility patterns in three periods.

Before stay-at-home orders

As shown in Fig. 4, the median travel distance in most counties (except some
counties such as the King County, Washington, one of the known epidemic
outbreak centers in the U.S. [15]) across the nation largely increased on
March 6, 2020 as expected on a normal spring travel date as usual if this
pandemic didn’t come out.

Figure 4: The mobility change patterns in the U.S. on March 6, 2020.

During stay-at-home mandates

However, as of March 15, 2020, two days after the U.S. Federal Government
announced the national emergency, people in most states in the Pacific Coast,
Midwest, and East Coast had reacted actively to the social distancing
guidelines and reduced their daily mobility (in Fig. 1). But the adherence to
the stay-at-home orders is geographically heterogeneous across the nation. By
zooming into the New York state (in Fig. 5), its median max-distance reduced
to less than 0.1 km and decreased about 73% of that compared to the baseline.
Despite warnings from health experts that drastic control measures are needed
to slow the spread of the virus, people in the Franklin County (Florida) and
the Monroe County (Florida) as shown in Fig. 6, and many other counties in
Arizona, New Mexico, Colorado, Utah and Wyoming still have increased daily
mobility patterns (Fig. 1) since there was no statewide lock-down order yet.
As of March 31, 2020, most states in the U.S. had issued fully shelter-in-
place or partially safer-at-home and no large-group gatherings orders [1],
asking residents to stay at home and go out only for essential services, such
as grocery shopping and medical cares. Thus, the spatial distribution of the
mobility changes was dominantly in blue (decreased mobility) on that day
except for some counties (as shown in Fig. 7), including Sutton County
(Texas), Carbon County (Wyoming), Big Horn County (Montana), Millard County
(Utah), Yuma, Prowers, and Lincoln counties (Colorado), and Pocahontas and
Decatur counties in Iowa. The county/state mobility patterns may change over
time if there will be some inevitable gathering events (e.g., in-person voting
in presidential primary elections and protests). For example, when casting a
ballot in Wisconsin’s April 7 election, crowds lined up at polling places
across the state — including thousands funneled to five sites in Milwaukee. A
small increase of median travel distance from 1.1 kilometers on Monday to 1.9
km in the Waukesha County (within the metropolitan area of Milwaukee), and
longer-distance travels in Lafayette County (from to 2.7 km April 6 to 4.5 km
April 7) and in Rock County (2 km to 2.66 km) were observed on that day (in
Fig. 8). Later, 71 people who went to the primary polls on April 7 were tested
positive for the SARS-COV-2 virus777http://madison.com/wsj/news/local/health-
med-fit/71-people-who-went-to-the-polls-on-april-7-got-covid-19-tie-
to/article_ef5ab183-8e29-579a-a52b-1de069c320c7.html. A study found a
statistically significant association between in-person voting and the spread
of COVID-19 cases two to three weeks after the election [6].

Reopening period

Since early May, 2020, when several states began to lift their stay-at-home
and non-essential business
restrictions888https://www.nytimes.com/interactive/2020/us/states-reopen-map-
coronavirus.html, more and more people returned to public life and the median
travel distance bounced up (in Fig. 9) after keeping at the minimum value for
a while during the stay-at-home period. Such an upward mobility trend is
expected to continue as more states partially reopen their economy.

In addition, by examining the state-specific stay-at-home dwell time temporal
patterns (in Fig. 10), we found that the three-day moving averages of the
median stay-at-home dwell time generally are higher for people residing in the
top 5 most infected states (New York, New Jersey, Illinois, Massachusetts,
California)999https://www.cdc.gov/covid-data-tracker/ than that in the least
infected states (Alaska, Montana, Wyoming, Vermont). In Hawaii, people
followed the stay-at-home order since March 25 and stayed longer time at home
until April 20 when a significant drop was identified. The temporal curve
further bounced up in late April but dropped again since May 4 when retail
stores in certain parts of the state reopened.

Figure 5: The mobility change patterns in New York on March 15, 2020. Figure
6: The mobility change patterns in Florida on March 15, 2020. Figure 7: The
U.S. mobility change patterns on March 31, 2020. Figure 8: The mobility change
patterns in Wisconsin on April 7, 2020. Figure 9: The U.S. mobility change
patterns on May 1, 2020. Figure 10: The three-day moving average of the median
stay-at-home dwell time changes for the five most and the five least infected
states (Data Source: SafeGraph).

##  4 Conclusion and Discussion

In this work, we utilized large-scale smartphone location-derived aggregated
mobility data with user-friendly design principles to develop a human mobility
tracking web portal in response to the COVID-19 epidemic. We also analyzed the
mobility changes across the nation regarding the statewide stay-at-home
orders. High-level of adherence was found at the beginning of the orders in
place although geographic variation existed. Some inevitable gathering events
(e.g., in-person voting in the presidential election) were associated with
mobility changes on that day and the lagged increasing COVID-19 cases two or
three weeks later. It demonstrated the value of tracking human mobility
patterns in the infectious disease research and in public health policy making
[4, 16].

It is worth noting that reduced mobility doesn’t necessarily ensure the
(physical) social distancing in practice according to CDC’s
definition101010https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-
sick/social-distancing.html: ”Stay at least 6 feet (2 meters) from other
people”. Due to the mobile phone GPS horizontal error and uncertainty [8],
such physical distancing patterns cannot be identified from the used
aggregated mobility data; it requires other wearable sensors or trackers
(e.g., Bluetooth). However, it will involve another important issue about
personal data privacy and ethical concerns. It is still an ongoing challenge
to find a ”sweet spot” to use such data to derive effective analyses for
saving lives while protecting the individual geoprivacy. We welcome user
feedback from different domains for further enhancement of our web portal to
inform public health decision-making practices.

## Acknowledgements

We would like to thank the Descartes Labs and the SafeGraph Inc. for providing
the anonymous and aggregated human mobility and place visit data. We would
also like to thank all individuals and organizations for collecting and
updating the COVID-19 epidemiological data and reports. Funding: S.G.
acknowledges the funding support provided by the National Science Foundation
(Award No. BCS-2027375). Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the author(s) and do
not necessarily reflect the views of the National Science Foundation. Author
contributions: Research design and conceptualization: S.G., J.M.R.; Data
collection and processing: S.G., Y.H.K., J.M.R.; Result analysis: S.G.,
J.M.R., Y.L.L; Visualization: J.M.R., J.K.; Project administration: S.G.;
Writing: all authors.

## References

  * [1] About 95% of Americans have been ordered to stay at home. This map shows which cities and states are under lockdown, available at https://www.businessinsider.com/us-map-stay-at-home-orders-lockdowns-2020-3.
  * [2] H. Allcott, L. Boxell, J. Conway, M. Gentzkow, M. Thaler, and D. Y. Yang.  Polarization and public health: Partisan differences in social distancing during the coronavirus pandemic.  NBER Working Paper, (w26946), 2020.
  * [3] M. Andersen.  Early evidence on social distancing in response to covid-19 in the United States.  Available at SSRN 3569368, 2020.
  * [4] C. O. Buckee, S. Balsari, J. Chan, M. Crosas, F. Dominici, U. Gasser, Y. H. Grad, B. Grenfell, M. E. Halloran, M. U. Kraemer, et al.  Aggregated mobility data could help fight COVID-19.  Science (New York, NY), 2020.
  * [5] S. Chen, Q. Li, S. Gao, Y. Kang, and X. Shi.  Mitigating covid-19 outbreak via high testing capacity and strong transmission-intervention in the United States.  medRxiv, 2020.
  * [6] C. D. Cotti, B. Engelhardt, J. Foster, E. Nesson, and P. Niekamp.  The relationship between in-person voting, consolidated polling locations, and absentee voting on covid-19: Evidence from the wisconsin primary.  Consolidated Polling Locations, and Absentee Voting on COVID-19: Evidence from the Wisconsin Primary (May 10, 2020), 2020.
  * [7] N. Dasgupta, M. J. Funk, A. Lazard, B. E. White, and S. W. Marshall.  Quantifying the social distancing privilege gap: a longitudinal study of smartphone movement.  medRxiv, 2020.
  * [8] S. Gao and G. Mai.  Mobile GIS and location-based services.  Comprehensive Geographic Information Systems, pages 384–397, 2017\.
  * [9] S. Gao, J. Rao, Y. Kang, Y. Liang, J. Kruse, D. Doepfer, A. K. Sethi, J. F. M. Reyes, J. Patz, and B. S. Yandell.  Mobile phone location data reveal the effect and geographic variation of social distancing on the spread of the covid-19 epidemic.  arXiv preprint arXiv:2004.11430, 2020.
  * [10] S. Ghader, J. Zhao, M. Lee, W. Zhou, G. Zhao, and L. Zhang.  Observed mobility behavior data reveal social distancing inertia.  arXiv preprint arXiv:2004.14748, 2020.
  * [11] A. Gollwitzer, C. Martel, J. Marshall, J. M. Höhs, and J. A. Bargh.  Connecting self-reported social distancing to real-world behavior at the individual and US state level.  PsyArXiv: 10.31234/osf.io/kvnwp, 2020.
  * [12] N. M. Kavanagh, R. R. Goel, and A. S. Venkataramani.  Association of county-level socioeconomic and political characteristics with engagement in social distancing for covid-19.  medRxiv, 2020.
  * [13] D. Liang, L. Shi, J. Zhao, P. Liu, J. Schwartz, S. Gao, J. A. Sarnat, Y. Liu, S. T. Ebelt, N. C. Scovronick, et al.  Urban air pollution may enhance covid-19 case-fatality and mortality rates in the United States.  medRxiv, 2020.
  * [14] Y. Liang, S. Gao, Y. Cai, N. Z. Foutz, and L. Wu.  Calibrating the dynamic huff model for business analysis using location big data.  Transactions in GIS, 2020.
  * [15] T. McMichael.  Covid-19 in a long-term care facility, King County, Washington, February 27-March 9, 2020.  Morbidity and Mortality Weekly Report, 69, 2020.
  * [16] N. Oliver, B. Lepri, H. Sterly, R. Lambiotte, S. Delataille, M. De Nadai, E. Letouzé, A. A. Salah, R. Benjamins, C. Cattuto, et al.  Mobile phone data for informing public health actions across the covid-19 pandemic life cycle.  Science Advances, 2020.
  * [17] M. Painter and T. Qiu.  Political beliefs affect compliance with covid-19 social distancing orders.  Available at SSRN 3569098, 2020.
  * [18] T. Prestby, J. App, Y. Kang, and S. Gao.  Understanding neighborhood isolation through spatial interaction network analysis using location big data.  Environment and Planning A: Economy and Space, page 0308518X19891911, 2019.
  * [19] R. Roth.  Cv-13-user interface and user experience (UI/UX) design.  University Consortium for Geographic Information Science, 2017.
  * [20] SafeGraph.  The impact of coronavirus (COVID-19) on foot traffic.  U.S. Consumer Activity During COVID-19 Pandemic, 2020.
  * [21] M. S. Warren and S. W. Skillman.  Mobility changes in response to COVID-19.  arXiv preprint arXiv:2003.14228, 2020.
  * [22] L. Zhang, S. Ghader, M. L. Pack, C. Xiong, A. Darzi, M. Yang, Q. Sun, A. Kabiri, and S. Hu.  An interactive covid-19 mobility impact and social distancing analysis platform.  medRxiv, 2020.
  * [23] C. Zhou, F. Su, T. Pei, A. Zhang, Y. Du, B. Luo, Z. Cao, J. Wang, W. Yuan, Y. Zhu, et al.  COVID-19: Challenges to GIS with big data.  Geography and Sustainability, 2020.


# Experience feedback using Representation Learning for Few-Shot Object
Detection on Aerial Images

Pierre Le Jeune
COSE & LIPN & L2TI
Université Sorbonne Paris Nord
<EMAIL_ADDRESS>
Mustapha Lebbah
LIPN
Université Sorbonne Paris Nord
<EMAIL_ADDRESS>
Anissa Mokraoui
L2TI
Université Sorbonne Paris Nord
<EMAIL_ADDRESS>
Hanene Azzag
LIPN
Université Sorbonne Paris Nord
<EMAIL_ADDRESS>

###### Abstract

This paper proposes a few-shot method based on Faster R-CNN and representation
learning for object detection in aerial images. The two classification
branches of Faster R-CNN are replaced by prototypical networks for online
adaptation to new classes. These networks produce embeddings vectors for each
generated box, which are then compared with class prototypes. The distance
between an embedding and a prototype determines the corresponding
classification score. The resulting networks are trained in an episodic
manner. A new detection task is randomly sampled at each epoch, consisting in
detecting only a subset of the classes annotated in the dataset. This training
strategy encourages the network to adapt to new classes as it would at test
time. In addition, several ideas are explored to improve the proposed method
such as a hard negative examples mining strategy and self-supervised
clustering for background objects. The performance of our method is assessed
on DOTA, a large-scale remote sensing images dataset. The experiments
conducted provide a broader understanding of the capabilities of
representation learning. It highlights in particular some intrinsic weaknesses
for the few-shot object detection task. Finally, some suggestions and
perspectives are formulated according to these insights.

##  1 Introduction

Object detection is a key problem in computer vision. It consists in finding
all occurrences of objects belonging to a predefined set of classes in an
image and classify them. Its applications range from medical diagnosis to
aerial intelligence through autonomous vehicles. Object detection methods
automate repetitive and time-consuming tasks performed by human operators
until now. In the context of Remote Sensing Images (RSI), detection is used
for a wide variety of tasks such as environmental surveillance, urban
planning, crops and flock monitoring or traffic analysis.

Deep learning and especially convolutional neural networks (CNNs) outperform
previous methods on most computer vision tasks and object detection is no
exception. Plenty of methods have been introduced to address this challenge.
Among them, Faster R-CNN [1] and YOLO [2] may be the most well-known and
studied. Even though this problem is far from being solved, detection
algorithms perform well when provided with sufficient annotated data. However,
this is often not available in practice and the creation of large dataset for
detection requires both time and expertise preventing the deployment of such
methods for many use cases. Another limitation to the widespread deployment of
detection techniques is the lack of adaptability. Once fully trained, it is
hard to modify a model to adapt to new objects. This is critical for some
applications which need to detect different objects from one usage to another.
Aerial intelligence is an example of such application: each mission may have
its specific objects of interest and therefore a detection model must be
adaptable (literally) on the fly. The overall objective of this work is to be
deployed on vertical aerial images. Yet, large-scale dataset of such images,
annotated for object detection, are rare. RSI are a convenient alternative and
provide an accurate estimation of performance in deployment.

Few-Shot Learning (FSL) techniques have been introduced to address these
issues and deal with limited data. This has been extensively studied for
classification [3], [4]. Its principle is to learn general knowledge from a
large dataset so that it can generalize efficiently (i.e. quickly and from
limited data) on new classes. There exist different approaches for this task.
Representation learning tries to learn abstract embeddings of images so that
the representation space is semantically organized in a way that makes
classification relatively easy (e.g. with a linear classifier). Meta-learning
based methods learn a model (teacher) that helps another model (student) to
perform well based on a limited amount of data. This is often done by training
both network on multiple low-data tasks (e.g. by changing classes between
epochs). Transfer learning is also a valid approach for FSL. It consists in
training a model on a large dataset and then adapt it to perform well on
another smaller one. This requires a supplementary training step and is often
subject to catastrophic forgetting [5], and overfitting. It needs advanced
tricks to prevent these undesirable effects.

This performs relatively well for classification but the more challenging
detection task still lacks few-shot alternatives. Though, recent work focused
on Few-Shot Object Detection (FSOD) applying ideas from FSL literature to
object detection. The first approaches were mostly oriented toward transfer
learning and fine-tuning [6, 7] disregarding the others FSL practices. Some
other work took inspiration from meta-learning [8] and representation learning
[9]. This is mostly applied to natural images, yet applications on remote
sensing images are scarce [10, 11]. Even if object detection is hard, applying
it on remote sensing images is even harder: object’s sizes can vary greatly
plus they can be arbitrarily oriented and densely packed. These supplementary
difficulties might explain why this specific topic remains mainly untouched.

This work introduces a new few-shot learning method for object detection and
evaluates its performance on aerial images. It detects objects from only a few
examples of a class and without any fine-tuning. The main idea is inspired
from prototypical networks [3] which learns an embedding function that maps
images into a representation space. Prototypes are computed from the few
examples available for each class and classification scores are attributed to
each input image according to the distances between its embedding and the
prototypes. The classic Faster R-CNN framework is modified to perform few-shot
detection based on this idea. Both classification branch in Region Proposal
Network (RPN) and in Fast R-CNN are replaced by prototypical networks to allow
fast online adaptation. In addition, a few improvements are introduced on the
prototypical baseline in order to fix its weaknesses.

This paper begins with a brief overview of literature on object detection,
few-shot learning and the intersection of these two topics. Then the
prototypical Faster R-CNN architecture is presented in detail alongside with
several improvements on our baseline. Next, the potential of the proposed
modifications throughout a series of experiences is demonstrated. Finally, the
proposed approach is discussed with a critical eye, and it is asked whether
representation learning is suitable for object detection.

##  2 Related work

###  2.1 Object detection

During the last decade, deep learning and especially CNN have made impressive
progresses in most computer vision tasks. Object detection is no exception and
plenty of CNN based methods were proposed to address this problem. Among the
most common, YOLO [2] is a one-stage method with a trade-off on speed. Faster
R-CNN [1] is another well-known object detection technique that focuses more
on accuracy with two separate stages. Most subsequent methods are more or less
inspired from these. A common feature between these techniques is the
generation of anchor boxes. These are bounding boxes chosen with different
aspect ratios and sizes, uniformly distributed on the image, that the networks
take as reference for the regression. Recently, some works tend to reduce the
accuracy gap between one and two stages methods. Especially, FCOS [12] drops
completely the anchors to build a fully convolutional one stage object
detection network that matches the performance of Faster R-CNN.

As our work is mainly based on Faster R-CNN, we will describe its functioning
in details. As mentioned above, it is made of two stages: a Region Proposal
Network (RPN) and a prediction head described in [13]. A third component must
also be mentioned, the backbone. This is a large CNN that extracts features
from images. It is often chosen as a ResNet [14] with a Feature Pyramidal
Network (FPN) [15] on top. This extracts features from different levels and is
very helpful for detecting objects of various sizes. Once the features are
extracted, they are fed into the RPN. This network is fully convolutional and
will output an objectness score $o_{a}$ for each generated box $a$ (i.e.
generally 3 boxes per positions in the multi-level feature map). This score
represents the likeliness of having an object within the corresponding patch
in the image. In addition, the RPN outputs box regressions $b_{a}^{\text{R}}$.
The regressions, combined with the anchors sizes and positions, give the
actual boxes coordinates in the image. Then, the best scoring boxes are
selected to be fed to the prediction head, where boxes are refined and
classified. For each box, the corresponding features are selected through a
pooling operation (RoI Align, proposed by [16]). These are flattened and
passed to the second stage. It computes refining coordinates shifts
$b_{j}^{\text{H}}$ and classes scores $c_{j}$ for each box. Following this, a
post-processing step filters out small, low-scoring and redundant boxes.

The training of this method is straightforward, each network has two losses,
one for the regression branch and one for the classification as described
below:

$\displaystyle\mathcal{L}_{reg}^{\text{R}}(b_{i}^{\text{R}},\hat{b}_{i}^{\text{R}})$
$\displaystyle=\text{SmoothL1Loss}(b_{i}^{\text{R}},\hat{b}_{i}^{\text{R}}),$
(1) $\displaystyle\mathcal{L}_{obj}^{\text{R}}(o_{i},\hat{o}_{i})$
$\displaystyle=\hat{o}_{i}\log(o_{i})+(1-\hat{o}_{i})\log(1-o_{i}),$ (2)
$\displaystyle\mathcal{L}_{reg}^{\text{H}}(b_{j}^{\text{H}},\hat{b}_{j}^{\text{H}})$
$\displaystyle=\text{SmoothL1Loss}(b_{j}^{\text{H}},\hat{b}_{j}^{\text{H}}),$
(3) $\displaystyle\mathcal{L}_{cls}^{\text{H}}(c_{j},\hat{c}_{j})$
$\displaystyle=-\log(c_{j}),$ (4)

where SmoothL1loss is a slight modification to L1 error function. Below a
certain threshold, the error is computed according to L2 loss, while above
that threshold L1 is used. This penalizes the network less for small errors in
boxes regressions. The variables with hat correspond to ground truth values.
These losses are combined in an overall objective function that is optimized
using stochastic gradient descent:

$\displaystyle\mathcal{L}=\mathcal{L}_{reg}^{\text{R}}+\mathcal{L}_{obj}^{\text{R}}+\mathcal{L}_{reg}^{\text{H}}+\mathcal{L}_{cls}^{\text{H}}.$
(5)

During training, not all boxes are selected for computing losses. First, the
generated boxes are separated into two groups: positive examples, i.e. boxes
with an overlap of at least 0.7 with a ground truth annotation, and negative
examples which represent the background class. Positive and negatives examples
are included in classification losses computation while only positives are for
regression.

###  2.2 Few-shot learning

Few-shot learning corresponds to learning a task in a limited data setting.
Specifically, a task is defined as $K$-shots, $N$-ways learning when the
training set only contains $K$ examples for each of its $N$ classes. In FSL
literature, it is common to introduce the query and support sets for a given
task. The support contains the available examples: $K$ images for each of the
$N$ classes of the task. It can be seen as the training set for that task. The
query set contains images from the same classes as the support and is used for
assessing the performance of the model, like a test set.

There exists different techniques to tackle low data regime. Transfer learning
is one of them. It consists in training a network on a large-scale dataset
(source domain) and then fine-tuning it on the few examples (target domain)
available for the actual task. This kind of methods require re-training each
time a new class is added. In the case of aerial surveillance, this is not
suitable as the adaptability must be almost immediate. In addition, these
techniques are often inclined to catastrophic forgetting [5], a recurrent
problem in continual learning. Classes previously learned are forgotten when
learning to classify new classes. Therefore, we will only discuss in this
paper methods based on meta learning or representation learning.

Meta-learning, in the context of FSL, refers to methods that train a model
called the meta-learner whose task will be to help another model to train
efficiently (in terms of computation and data) for the actual problem. This
can be pictured as training a teacher model whose goal is to make its student
learn better. That way, the student can quickly learn new tasks and perform
well in low-data regime. Meta-learning methods often share the same training
strategy with two nested optimization loops. The inner one updates the weights
of the student net, while the outer one deals with the meta-learner. Between
each iteration of the outer loop, a new task is chosen for the inner network
to learn. The student is trained on the support set while the teacher’s
objective is to maximize the student performance. Many ways of teaching were
introduced. For instance, the teacher network can be trained to directly
output weight updates of the student as described in [4]. Another approach is
to output only initial weights for the student as in [17]. The starting point
is hopefully better than random, thus allowing faster and better optimization.
While these techniques are promising, they do not scale very well for large
networks as the teacher must be substantially larger than the learner. In
order to assess generalization performance, classes are split into two
categories: base and novel classes (or equivalently seen and unseen classes).
During meta-training, tasks are sampled using only base classes. At test time,
however they can be sampled either from novel classes or both base and novel.

Another drawback is the necessary fine-tuning. Even if it should be quicker
than regular transfer learning, most meta-learning methods still require this
additional step. Some exceptions are based on representation learning or
metric learning. The principle is to train a network to output an abstract
representation from an input image. This is first introduced for signature
verification with siamese networks in [18]. Prototypical networks [3] is a
pioneer work in using this for FSL. An embedding network is trained to produce
such representations. Before inference, prototypes for each class are computed
using the embedding net and the support images. During inference, the query
images are embedded and the distance between their representations and the
prototypes determine the classification scores. In [3], this is done with a
linear classifier, but other choices are possible. Relation networks [19]
proposes another network to compute the class scores from the image
representation and the prototypes. Similarly, matching networks [20] used two
separate networks to embed the image and the prototypes. These methods are
usually trained by randomly sampling tasks at each epoch, just as other meta-
learning methods. This succession of new tasks helps the network to generalize
well and therefore improves its accuracy on unseen classes.

Figure 1: Architectural view of prototypical Faster R-CNN. Embedding vectors
for each of the possible box locations is generated by the RPN. These are
compared with class prototypes to output objectness scores. The same process
occurs in the second stage to produce classes scores.

###  2.3 Few shot object detection

The focus of FSL was previously on classification tasks. Detection is a harder
problem and so is FSL. That explains why the combination of both was only
studied recently. One early work on this problem is Low-Shot Transfer Detector
[6]. It applies few-shot transfer learning in order to refine a pre-trained
detection network on a small dataset. To do so, they introduce two
regularization losses during the fine-tuning phase. One forces the network to
focus on new objects by reducing the magnitude of the feature maps on
background areas. The other loss prevents forgetting knowledge from the source
domain. It penalizes the refined network to have dissimilar activation in the
pre-softmax layer compared to the network trained on base classes only.
Similarly, [7] proposes to first pre-train a Faster R-CNN on a base dataset
and then fine-tune only the last classification and regression layers with the
new classes. This is quite similar to [6], but even simpler as it does not
need any new loss functions.

Even if RepMet [9] focuses mostly on few-shot classification, authors showed
that their method can also be applied for detection. Their approach is mainly
based on metric learning. During base training, they learn alongside the
network’s weights a set of representatives for different classes (several per
class). These high-dimensional vectors define the centers of Gaussian
distributions inside the embedding space. The class probabilities are assigned
from the distances between the embedding vectors and the representatives in an
analogous way as in [3]. At test time, the encoding network generates
representatives from the few examples in dataset, which are then fine-tuned
for a few iterations. For detection, they replace the second stage of Faster
R-CNN with their classification module. Although this method is astride on
meta-learning and representation learning it does not leverage an episodic
task training strategy. Instead, it relies on fine-tuning: it trains first on
base classes and fine-tune on novel ones. To our knowledge, this is the first
attempt to solve FSOD with metric learning.

Most recent work focuses on meta-learning in order to solve FSOD. For
instance, [21] trains a one stage detector along with a meta features
extractor. This extractor is a CNN that computes a reweighting vector for each
class from the support set. When a query image is passed through the detector,
the features maps output by the backbone are channel-wise multiplied with the
reweighting vectors to produce class-specific maps. These maps are then passed
to the detector’s head. Each map is responsible for detecting objects of the
corresponding class. Closely related, [8] incorporates a similar reweighting
scheme in Faster R-CNN. A major difference though is that the features and
reweighting vectors are computed with the same network. A similar class-
attention mechanism, but inside the RPN network is proposed by [22]. In the
second stage they use multi-relation heads that combine support set
information and query image features in multiple ways inside the
classification branch. Another attempt to leverage an attention mechanism is
proposed by [23]. Query and support examples are co-adapted to reduce features
discrepancies and improve prediction both in RPN and second stage. Except the
latter, these methods are trained episodically.

Different methods for combining support information with the query features
exist, e.g. [11] creates a graph between the support vectors and the query
embeddings. Then a GRU [24] processes it in order to provide attention between
regions of interest and support set images. Alike this, [25] processes a
graph, whose final nodes are the reweighting vectors, with a graph convolution
network. The graph is initialized with similarities between classes’ names,
embedded with GloVe [26].

With regard to FSOD on remote sensing images, very few methods exist. To the
best of our knowledge, only two works have been published on this topic. [10]
improves on [21] by using an improved version of YOLO. This mainly adds multi-
scale features and predictions. This is especially important to tackle RSI as
object size can vary greatly. The second one [11], makes use of a two-stages
detector and compute reweighting vectors with a GRU relation module. Both of
the methods provide benchmark on VHR-10 dataset [27], yet the number of novel
classes and the different base/novel classes splits are different, making the
comparison difficult.

##  3 Proposed method

To deal with FSOD, prototypical Faster R-CNN is introduced. It is a modified
version of Faster R-CNN based on metric learning. The key idea is to replace
the classification branches in both stages by prototypical networks. This is
related to RepMet [9] that learns representatives only in the second stage.
Yet it fixes one major flaw of RepMet, the RPN. Once trained, the RPN
specializes on classes seen during training. This means that objects from new
classes are filtered out by the RPN, leaving no chance for the second stage to
detect them. It has a low recall on unseen classes. This is especially not
desirable in FSL. Instead, the RPN should be able to adapt to new classes. Our
method is an attempt to fill this shortcoming.

First, a few notations are introduced. Let $\mathcal{C}=[1,N]$ be the set of
all classes. In the case of $n$-ways $k$-shots learning, each task consists in
detecting objects among $n$ classes with $k$ examples per class. At each
episode a new detection task $i$ is randomly sampled:
$C_{i}\subset\mathcal{C}$ with $|C_{i}|=n$. Then, data are sampled from the
whole dataset to form a support and a query sets.
$S_{i}=\\{(x_{1},t_{1}),...,(x_{nk},t_{nk})\\}$ and
$Q_{i}=\\{(x_{1},t_{1}),...,(x_{nk^{\prime}},t_{nk^{\prime}})\\}$. For
detection, each image $x_{j}$ comes with a set of annotations $t_{j}$ which
contains the location and label of all objects in the image. Annotations that
do not belong to the episode classes are discarded. To build the support set,
for each class $c\in C_{i}$, we select images containing objects of class $c$
and disregard all other objects (i.e. their annotations are not included in
the support set but the image is not masked, so they are still visible). If
there are more than one object $c$ in the image, only one is selected randomly
as the annotated example.

###  3.1 Prototypical Faster R-CNN

We propose to change the output dimension of the classification branches in
both the RPN and the head. That way, instead of producing a classification (or
objectness) score per box, these networks output embedding vectors. Each
vector represents the information contained inside the corresponding box. The
computation of the representations is straightforward. Features at several
scales are extracted by the backbone (denoted $f$). These features are then
fed to the RPN that computes representation vectors for each location in the
features maps (i.e. each corresponding to an anchor). A 2-layers CNN (shared
with the regression branch) is applied on each feature maps, then a RoI Align
operation extracts same-size features for each anchor. Finally, a 2-layers MLP
maps these features into the embedding space. The dimension of the
representation is $r=128$ and is kept fixed in all our experiments. We call
the RPN embedding pipeline $g$ such that $(f\circ g)(x_{j})=z_{j}$ is the set
of all anchors’ embeddings for image $x_{j}$.

For the second stage, the best scoring boxes produced by the RPN are selected.
Their corresponding features are cropped with RoI Align and fed to a 4-layers
MLP that outputs embedding vectors as well. Note that the 2 first layers of
the head’s MLP are shared with the regression branch. As for the RPN, let $h$
be the encoding function of the head: $(f\circ h)(x_{j})=\tilde{z}_{j}$. Fig.
1 illustrates the whole representation pipeline and how the different scores
are computed in the network.

Classes prototypes are computed from the support images, with the same
network, except that only the example’s box is used for feature pooling. When
multiple examples are available for a class, their embeddings are averaged to
build one prototype per class: $p_{i,c}$ and $\tilde{p}_{i,c}$ for the RPN and
the head respectively.

Given the embeddings of an image and the prototypes, we compute the likelihood
for each class as follows. This supposes that each class is represented by a
Gaussian distribution centered on its prototype:

$p(x_{j,a}|y_{j,a}=c)=\exp\Big{(}\frac{-d(z_{j,a},p_{c})^{2}}{2\sigma^{2}}\Big{)},$
(6)

where $x_{j,a}$ refers to the crop of image $x_{j}$ with anchor $a$ and
$y_{j,a}$ is its corresponding label ($y_{j,a}\in C_{i}\cup\\{\varnothing\\}$,
with $\varnothing$ representing the background class). $d$ is a distance
measure over the representation space, in our experiments, $d$ is the
Euclidean distance. Note that in our case, the embeddings are normalized after
their computation, therefore Euclidean distance is equivalent to Cosine
Similarity. $\sigma=0.5$ is the variance of the distribution and is fixed.
This likelihood computation is the same for the RPN and the head. However, in
the head, the likelihood of the background class is also computed:

$p(x_{j,a}|\varnothing)=1-\max\limits_{c\in C_{i}}p(x_{j,a}|c).$ (7)

From this, we derive the objectness score in the RPN and the classification
(including background) scores in the head:

$\displaystyle o_{j,a}$ $\displaystyle=\max\limits_{c\in C_{i}}p(x_{j,a}|c),$
(8) $\displaystyle p(c|x_{j,a})$
$\displaystyle=\frac{p(x_{j,a}|c)}{\sum\limits_{c\in
C_{i}\cup\\{\varnothing\\}}p(x_{j,a}|c)}.$ (9)

The training is done episodically, sampling a random subset of classes
$C_{i}\subset\mathcal{C}$ at each epoch. The embedding network is trained
using the same loss functions as Faster R-CNN (see (1)–(5)) and the same
positive/negative examples selection. The loss is computed on the query set
and between each update of the network, the prototypes are recomputed from the
same support set. Once all query images have been seen by the network, a new
task is sampled. In our experiments, the query set contains 5 images for each
of its $n$ classes, this means at least 5 examples for each class, but this
number can be larger as more than one object are present in the images. The
optimization is done with Adam optimizer and a learning rate of $1e-4$. The
backbone network is pretrained on ImageNet and its first layers are kept
frozen during training.

Table 1: Mean average precision over 5 runs on DOTA dataset with 95% confidence interval. Results are given for two different train/test classes split. Split A: $\\{0,1,4\\}$, Split B: $\\{7,11,13\\}$ (only test classes are given).  |  $k$ shots | 1 | 3 | 5 | 10
---|---|---|---|---|---
Split A | Train classes | 0.275 $\pm$ 0.01 | 0.352 $\pm$ 0.02 | 0.390 $\pm$ 0.01 | 0.384 $\pm$ 0.02
Test classes | 0.047 $\pm$ 0.02 | 0.024 $\pm$ 0.01 | 0.038 $\pm$ 0.01 | 0.041 $\pm$ 0.01
Split B | Train classes | 0.415 $\pm$ 0.03 | 0.392 $\pm$ 0.03 | 0.434 $\pm$ 0.02 | 0.414 $\pm$ 0.03
Test classes | 0.08 $\pm$ 0.01 | 0.101 $\pm$ 0.02 | 0.121 $\pm$ 0.01 | 0.101 $\pm$ 0.02

###  3.2 Iterative improvements

In order to improve the performance of our model, a series of improvements are
described on top of the baseline presented above.

####  3.2.1 Hard example mining

One issue encountered with the baseline was the detection of all training
classes regardless of support examples. This is class memorization. Although
this improves performance for training classes, it produces lots of false
positive detections. In order to address this, we propose to sample hard
negative examples to encourage the network to detect support classes only. The
main idea is to take advantage of the annotations for classes not selected in
the current task to find hard negative examples, i.e. classes that the network
could have memorized from previous tasks. With a new task at each epoch, it is
likely that the network still produces detection for objects annotated in one
of the previous epochs if it does not rely on the support information.
Therefore, these annotations can be used to find examples that should be
considered as background for the current task only. These are different from
the background examples that do not contain any class of the dataset, which
are referred as easy negative examples. Hence, this encourages the network to
detect only objects annotated in the support set.

####  3.2.2 Moving average prototypes

Another issue with the baseline is that the prototypes can change abruptly,
either when the network is updated or when the support set changes. This makes
the training unstable. In order to prevent such rapid modification of the
prototypes, an exponential moving average is introduced to smooth the
modification. Hence, $\bar{p}_{c}^{(t+1)}=\alpha
p_{c}+(1-\alpha)\bar{p}_{c}^{(t+1)}$. $\alpha$ is set to $0.1$ in our
experiments. $\bar{p}_{c}^{(t)}$ is the averaged prototype for class $c$ at
iteration $t$, while $p_{c}^{(t)}$ is the prototype computed from the support
set, for class $c$ at iteration $t$ (as described in section 3.1).

####  3.2.3 Background clustering

Lastly, the baseline shows poor separation of unseen classes embeddings. This
leads to poor performance on novel classes at test time. In order to solve
this, an inspiration is drawn from [28]. At each iteration, they fit a K-means
on the learned representations. This gives pseudo-labels to train the network
for classification in a self-supervised manner. Similarly, we propose to fit a
K-means on the negative embeddings (i.e. representing boxes not matched by any
ground truth object). From the resulting pseudo-labels a contrastive loss
function (Triplet Loss [29]) is computed. The triplets are formed with
embeddings that were labeled identically by the K-means. It encourages the
network to organize the embedding space into tight and separated clusters.
This will eventually discover semantic clusters that represent objects unseen
(i.e. non-annotated) during training.

##  4 Results and experiments

In order to assess the performance of our method, one dataset is chosen: DOTA
[30] which contains remote sensing images. It has 16 different classes and
contains around 400k annotated objects. These objects are distributed within
2800 large-scale images and their sizes vary greatly, even within a single
class as the spatial resolutions between two images can be significantly
different.

The experimental protocol is as follows: three classes are reserved for
evaluation and two different splits were randomly selected. The network is
trained episodically with the remaining classes for 30k iterations. Training
more improves the performance on training classes, but the network starts to
overfit and performs far worse on test classes. Hence, early stopping, to
preserve generalization on new classes. The networks are evaluated on both the
base and novel classes to assess both the learning and the generalization
capabilities. For each experiment, mean average precision (mAP) is provided,
computed according to PASCAL VOC [31], with different number of shots: 1, 3, 5
and 10.

###  4.1 Results on DOTA

Table 1 contains the mean average precision reported on DOTA dataset with our
best model, according to section 4.2. The results are reported as the mean
over 5 runs with 95% confidence intervals. We chose two different train/test
classes split randomly. From this, it can be seen that performance on test
classes is far below the train classes. This is expected since no supervision
was available during training for these objects. It is interesting to see that
the more examples (i.e. shots) are given to the network for a class, the
better it performs. This may be explained as more shots provide a better
approximation of the cluster’s center. Yet this pattern is not always
observed, for instance with training classes of split B, the performance is
stable with respect to the number of shots. This could happen when a single
class is represented by two separate clusters. Having multiple representation
for one class increase the chance of sampling representation from different
clusters. Hence, the average can be completely outside both clusters.
Nevertheless, it can be seen from Fig. 2 (which was made with one shot only)
that the prototypes are often not positioned in the center of their cluster
and averaging is often a right strategy to improve performance.

It can also be seen that the increase of performance stagnates with the number
of shots. Important gain is reported between 1, 3 and 5 shots but no
significant improvement from 5 to 10. A relatively low number of examples is
able to approximate correctly the class prototype. Increasing this number does
not improve the positioning of the centers any more and can even be harmful as
discussed above.

More broadly, the performance is quite low both for base and novel classes and
this does not meet our objectives. In comparison, Faster R-CNN trained in a
supervised manner on the complete dataset achieved around 0.7 mAP. It would be
unfair to directly compare this value with the performance of our network as
it was mainly designed for adaptability. Yet the performance loss on training
classes is quite large.

Unfortunately, no other method proposed benchmark on DOTA dataset for FSOD and
the very few works [11, 10] in this field did not provide their code, thus
comparison was not possible.

###  4.2 Ablation study

In order to validate the hypothesis formulated in 3.2, an ablation study is
conducted. The results of this analysis can be found in Table 2. They only
partially validate this hypothesis. On the one hand, the introduction of hard
examples mining and moving average prototypes improves consistently the test
mAP in the one-shot setting. On the other hand, background clustering reduces
greatly the performance on train classes, while achieving similar results on
test classes. It is still unclear why this does not perform better, it will be
investigated as part of future work. According to this analysis, we chose to
fix our architecture with hard example mining and the moving average as it
combines best train and test performance. Yet, those conclusions must be taken
carefully since the variability between different runs is high and the
performance gains are small between the experiments.

Table 2: Ablation study on improvements described in section 3.2. Each row corresponds to the addition on top of the baseline. HEM corresponds to hard example mining, MA to moving average prototypes and BC to background clustering. Once again results are provided for different numbers of shots, both for train and test classes. Bold scores correspond to highest mAP for each $k$, either for train or test classes. |  $k$ shots | 1 | 3 | 5 | 10
---|---|---|---|---|---
Baseline | Train | 0.355 | 0.359 | 0.343 | 0.304
Test | 0.021 | 0.027 | 0.038 | 0.041
\+ HEM | Train | 0.312 | 0.356 | 0.412 | 0.343
Test | 0.04 | 0.023 | 0.033 | 0.026
\+ HEM + MA | Train | 0.265 | 0.339 | 0.37 | 0.351
Test | 0.069 | 0.035 | 0.042 | 0.059
\+ HEM + MA + BC | Train | 0.133 | 0.145 | 0.182 | 0.148
Test | 0.043 | 0.041 | 0.047 | 0.026

##  5 Discussions and perspective

From these results, one question arises: is representation learning a suitable
choice for object detection? Representation learning methods are competitive
with state-of-the-art for few-shot classification, but seem to be
inappropriate for few-shot detection. This may be because detection task
requires distinguishing between more closely related images. When a trained
RPN classifies two overlapping patches of an image, it may produce completely
different outputs whether it contains an object entirely or not. Such quasi-
discontinuities are unlikely to occur with a randomly initialized network.
Thus, the embeddings of two closely overlapping patches will be mapped nearby
in the representation space.

Fig. 2(a), where threadlike structures can be seen, illustrates this well.
These patterns are made of embeddings from close overlapping patches within
the same input image. This prior spatial organization may be the cause of the
low performance of our method. This is especially true as the classifier on
the representation space is equivalent to a linear classifier (see [3] for
details). For few-shot classification, there is no such prior organizing the
representation space as two different images cannot belong to the same larger
image. Hence, the structures and colors are not as similar as those of two
close patches for object detection. Fig. 2 shows that training is able to
overcome this and organizes the space into semantic clusters. Yet this only
happens for training classes, for which strong supervision is available during
training. For test classes, the weak supervision available is not enough to
build a semantically-aware structure: the test classes representations are
mixed together and with negative examples representations (in black in Fig.
2).

Results provided in section 4.1 are computed on a test set, whose images were
not seen during training. Yet, mAP both on base and novel classes is tracked
during training on images already seen by the network. It shows a large
improvement compared to what is reported in table 1 (around 0.65 mAP on train
classes and 0.2 on test classes). This strong overfitting showcases the lack
of generalization of our method. It may be explained by a simple reason.
Objects can only represent a small part of the patch for object detection.
Hence, much information is embedded in the representation alongside the
relevant semantic information. The position of the embedding is partially
controlled by the background and the network can easily learn correlation
between background and semantic, making it easy to detect objects correctly in
previously seen images. This is especially true for small objects. In Fig.
2(b), classes 1, 9 and 10 represent small vehicles and have the largest
clusters. In comparison, classes 5, 6 and 14 respectively represent basketball
court, running track and soccer field. Their clusters are much tighter. Their
embeddings contain much variance compared to other classes and therefore
detection is harder. Of course, this could also occur for classification but
in practice the background variety is far smaller and represent a smaller
portion of the image. Hence, it produces tighter clusters and better class
separation. It can also be seen from Table 1 that the performance on split A
(containing small vehicles) is worse than on split B. Such a large difference
between the splits suggests that the evaluation protocol is not well-suited
for this problem. More splits should be used in order to assess generalization
on all classes and performance metrics should be reported per class.

In order to leverage strong supervision, one could try self-supervised
methods. It has recently been shown that these methods, e.g. [28, 32] can
learn generic representations that generalize for many visual tasks, in
particular in low-data regime. It would be interesting to investigate further
these methods for few-shot object detection, this is planned as future work.
In addition, we plan to try methods based on attention mechanism instead of
representation learning as our experiments highlight some weaknesses of the
latter. Attention mechanisms, have recently shown great performance for plenty
of tasks including few-shot object detection. Finally, a change of the
underlying detection architecture is required as modifications in Faster R-CNN
can be cumbersome (due to its two stages and the generation of anchors boxes).
Instead, FCOS, which is a one-stage and anchor-less detector, is probably
better suited.

In a nutshell, we proposed a novel method for few-shot object detection based
on representation learning. These early results do not meet our expectations
in terms of performance. Yet the insights generated in this study allow to
understand the strength and weaknesses of representation learning for few-shot
object detection task. This will be helpful for future research. Ongoing work
is focusing on improving these results using simpler architecture like FCOS.

(a) Before training

(b) After training

Figure 2: TSNE visualization on the embedding space, before and after
training. Training organizes this space semantically and reduces the
threadlike patterns representing close patches in the input image. Yet this is
not completely solved for unseen classes during training.

## Acknowledgment

The authors would like to thank COSE for their close collaboration and the
funding of this project.

## References

  * [1] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28:91–99, 2015\.
  * [2] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.
  * [3] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [4] S Ravi H Larochelle.  Optimization as a model for few-shot learning.  In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017\.
  * [5] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  * [6] Hao Chen, Yali Wang, Guoyou Wang, and Yu Qiao.  Lstd: A low-shot transfer detector for object detection.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [7] Xin Wang, Thomas E. Huang, Trevor Darrell, Joseph E Gonzalez, and Fisher Yu.  Frustratingly simple few-shot object detection.  July 2020.
  * [8] Xiaopeng Yan, Ziliang Chen, Anni Xu, Xiaoxi Wang, Xiaodan Liang, and Liang Lin.  Meta r-cnn: Towards general solver for instance-level low-shot learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9577–9586, 2019.
  * [9] Leonid Karlinsky, Joseph Shtok, Sivan Harary, Eli Schwartz, Amit Aides, Rogerio Feris, Raja Giryes, and Alex M Bronstein.  Repmet: Representative-based metric learning for classification and few-shot object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5197–5206, 2019.
  * [10] Xiang Li, Jingyu Deng, and Yi Fang.  Few-shot object detection on remote sensing images.  IEEE Transactions on Geoscience and Remote Sensing, pages 1–14, 2021.
  * [11] Zixuan Xiao, Jiahao Qi, Wei Xue, and P. Zhong.  Few-shot object detection with self-adaptive attention network for remote sensing images.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14:4854–4865, 2021.
  * [12] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: Fully convolutional one-stage object detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9627–9636, 2019.
  * [13] Ross Girshick.  Fast r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [15] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [16] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [17] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In International Conference on Machine Learning, pages 1126–1135. PMLR, 2017.
  * [18] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah.  Signature verification using a" siamese" time delay neural network.  Advances in neural information processing systems, 6:737–744, 1993\.
  * [19] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.  Learning to compare: Relation network for few-shot learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199–1208, 2018.
  * [20] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra.  Matching networks for one shot learning.  In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
  * [21] Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell.  Few-shot object detection via feature reweighting.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8420–8429, 2019.
  * [22] Qi Fan, Wei Zhuo, Chi-Keung Tang, and Yu-Wing Tai.  Few-shot object detection with attention-rpn and multi-relation detector.  In CVPR, 2020.
  * [23] Ting-I Hsieh, Yi-Chen Lo, Hwann-Tzong Chen, and Tyng-Luh Liu.  One-shot object detection with co-attention and co-excitation.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [24] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling, 2014.
  * [25] Geonuk Kim, Hong-Gyu Jung, and Seong-Whan Lee.  Few-shot object detection via knowledge transfer.  In 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 3564–3569.
  * [26] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
  * [27] Gong Cheng, Peicheng Zhou, and Junwei Han.  Learning rotation-invariant convolutional neural networks for object detection in vhr optical remote sensing images.  IEEE Transactions on Geoscience and Remote Sensing, 54(12):7405–7415, 2016.
  * [28] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 132–149, 2018.
  * [29] Elad Hoffer and Nir Ailon.  Deep metric learning using triplet network.  In International workshop on similarity-based pattern recognition, pages 84–92. Springer, 2015.
  * [30] Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang.  Dota: A large-scale dataset for object detection in aerial images.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3974–3983, 2018.
  * [31] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International journal of computer vision, 88(2):303–338, 2010.
  * [32] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.