# A Comparative Review of Recent Few-Shot Object Detection Algorithms

Jiaxu Leng, Taiyue Chen, Xinbo Gao∗, Yongtao Yu, Ye Wang, Feng Gao, and Yue
Wang Jiaxu Leng, Taiyu Chen, Xinbo Gao, Ye Wang and Feng Gao are with the Key
Laboratory of Image Cognition, Chongqing University of Posts and
Telecommunications, Chongqing 400065, ChinaYue Wang is with the School Of
Cyber Security and Information Law, Chongqing University of Posts and
Telecommunications, Chongqing 400065, ChinaYongtao Yu is with the Faculty of
Computer and Software Engineering, Huaiyin Institute of Technology, Huaian
223003, China∗Corresponding author: Xinbo Gao.

###### Abstract

Few-shot object detection, learning to adapt to the novel classes with a few
labeled data, is an imperative and long-lasting problem due to the inherent
long-tail distribution of real-world data and the urgent demands to cut costs
of data collection and annotation. Recently, some studies have explored how to
use implicit cues in extra datasets without target-domain supervision to help
few-shot detectors refine robust task notions. This survey provides a
comprehensive overview from current classic and latest achievements for few-
shot object detection to future research expectations from manifold
perspectives. In particular, we first propose a data-based taxonomy of the
training data and the form of corresponding supervision which are accessed
during the training stage. Following this taxonomy, we present a significant
review of the formal definition, main challenges, benchmark datasets,
evaluation metrics, and learning strategies. In addition, we present a
detailed investigation of how to interplay the object detection methods to
develop this issue systematically. Finally, we conclude with the current
status of few-shot object detection, along with potential research directions
for this field.

###### Index Terms:

Meta-Learning, Transfer-Learning, Deep Learning, Few-Shot Learning, Semi-
Supervised Learning, Weakly-Supervised Learning, Object Detection.

##  I Introduction

Given a set of classes, object detection aims to detect all instances of these
classes in an/a image or video. As a fundamental task of computer vision,
object detection has achieved great attention and been applied to numerous
downstream applications, e.g., intelligent monitoring [1], augmented reality
[2], automatic driving [3].

Earlier, traditional approaches attempted to exploit hand-crafted features to
exhaustively search objects [4, 5, 6, 7], requiring abundant prior knowledge
to manually design suitable features for special objects detection (e.g.,
face, pedestrian and traffic signs). Due to Alexnet’s remarkable performance
on ImageNet in 2012 [8], deep learning began to obtain increasing attention in
the computer vision community, since it could automatically mine implicit task
notions from training data and achieve huge performance gains when compared
with traditional approaches. Especially in recent years, deep-learning
approaches have made great breakthroughs in object detection [9, 10, 11, 12].
In order to extract robust concepts, deep learning models tend to acquire
abundant labeled data for training. Nevertheless, it is not always easy to
collect volumes of well-labeled data for a specific task: (1) data preparation
is considerably time-consuming and laborious where it would cost about 10
seconds to label an instance [13]; (2) some rare cases exist at very low
frequency, due to the inherent long-tail distribution of real-world data,
e.g., endangered animals. Specifically, daily applications are crying out for
few-shot learning to cut costs, while generic techniques and strategies could
be prone to either capture noise as common notions (i.e., overfitting) or
diverge (i.e., underfitting) in few-shot scenarios. However, even a child can
quickly extract task-specific notions when shown small data and associated
labels. Therefore, it encourages us to develop few-shot object detection
(FSOD) that not only needs as little supervision as possible but also should
be superior/close to many-shot detectors, as shown in Fig. 1. Especially, we
strictly restrict the total amount of supervision and do not limit the form of
supervision. Here, we mainly discuss three main types of few-shot settings, as
shown in Section I-B.

Figure 1: Many-shot vs few-shot object detection. (a) The pipeline of many-
shot object detection. It exploits a large-scale dataset with instance-level
labels to learn a robust detector. (b) The pipeline of few-shot object
detection. Only limited labeled data and extra datasets without target-domain
supervision can be accessed during the training stage. Note that the formation
of the target-domain supervision could be image-level tags.

In recent years, few-shot learning has achieved several crucial breakthroughs,
especially in few-shot classification (FSC) [14, 15, 16, 17, 18, 19, 20, 21,
22, 23, 24, 25, 26, 27]. Inspired by recent advances in FSC, earlier works
regarded FSOD as a FSC issue that first exploited a region proposal algorithm
(e.g., SS [28]) to generate preliminary regions of interest (RoIs) and
classify each RoI whether or not to contain an object. However, unlike FSC,
FSOD is more complex, which not only needs to classify each RoI, but also
should localize each RoI precisely. It is infeasible to look at two
complementary sub-tasks in isolation. Earlier works have relatively low
precision, since excessive low-quality and ambiguous proposals could confuse
the meta classifiers. Subsequent works began to adopt a new scheme to
simultaneously optimize few-shot detectors for both sub-tasks in order to
obtain high-quality proposals. Especially, several metric-based approaches
[29] provided category-specific notions (e.g., aspect ratios and sizes of
objects) to the bounding box regressor. Moreover, existing approaches still
rely on existing detectors, e.g., R-CNN, YOLO and SSD variants [30, 9, 10, 12,
31, 11], which were originally designed to tackle many-shot issues, and do not
take special considerations into few-shot issues. Classic architectures not
only should exhaustively search all locations whether to cover objects or not,
but also need to associate features with object shapes, which also requires
that backbones should effectively and efficiently encode both shape and class
notions into semantics for objects of novel classes. However, in low-shot
scenarios, too large and too low intra-class variations are very common where
large intra-class variations tend to bring with low inter-class distinction
and low intra-class variations usually lead to low data diversity (e.g.,
aspect ratios). It is hard to exploit limited data to learn a robust encoder
and thus few-shot detectors cannot extract high-quality proposals from non-
robust features. Therefore, many FSOD approaches utilized extra datasets [32,
33] to acquire generic notions (e.g., pre-trained backbones [34, 35, 36, 8])
for these heavy-weight frameworks, which were conducive to tackle few-shot
challenges, instead of training from scratch. To obtain high performance,
several works supposed that a novel category has close relation with base
categories, e.g., shared visual components (color/shape/texture), adding extra
constraints (i.e., KL divergence) to efficiently transfer shared notions to
novel classes. However, it led to some new issues, e.g., domain shifts [37,
38], where source-domain knowledge would not fit target domain well. In that
case, such pre-training phases could have little effect for a novel task and
FSOD approaches could very easily confuse highly similar classes and have
uncertainty in localizing objects of novel classes [38, 39, 40, 41], due to
little inter-domain and noisy intra-domain support (Section I-C). Moreover,
most FSOD methods focus on a classic $N$-way $K$-shot setting since it needs
not consider an imbalanced problem and has no requirement to obtain implicit
information from extra unlabeled data collected from the target domain when
compared with other classic settings, in Section I-B. In brief, FSOD still has
a long way to go.

Here, we limit the scope of this paper on how to learn a competent detector
under few-shot/limited-supervised settings. For content completeness, we also
present a compendious review of advances in object detection, few-shot
learning, semi-supervised learning and weakly-supervised learning. The main
contributions are summarized as follows:

  * •

We identify few-shot problems and propose a novel data-based taxonomy for
studying main challenges and existing solutions in FSOD.

  * •

We summarize existing solutions in a systematic manner. The outline of our
survey includes the definition of the few-shot problems, benchmark datasets,
evaluation metrics, a summary of the main approaches. Specially, for these
approaches, we provide a detailed analysis of how these methods interplay with
each other to promote development in this promising field.

  * •

We present and discuss the potential research directions in this issue.

The overall organization is presented in Fig. 2. We first provide a brief
review for recent advances in related tasks, such as few-shot learning, in
Section II. Following the taxonomy proposed in Section I, we respectively
present their definition, benchmark datasets, evaluation metrics and related
works of how to learn robust few-shot detectors under various limited-
supervised settings in Section III, Section IV and Section V. Specially, in
Section III, we also involve how they work together to promote technical
progress. Finally, we conclude this survey, along with the potential future
trends in this promising domain, in Section VI.

###  I-A Comparison with Previous Reviews

In recent years, miscellaneous generic object detection surveys have been
published [42, 43, 44, 45, 46, 47, 48]. Zhao et al. [42], Sharma et al. [43]
and Dhillon et al. [44] provided a detailed analysis, including classic
architectures, useful tricks, benchmarks, evaluation metrics, etc. Wang et al.
[46], Liu et al. [47] and Sultana et al. [48] highlighted recent developments
of deep-neural-network based detectors. Especially, there exist numerous
excellent surveys that focused on a kind of detectors designed for several
specific objects, such as pedestrian detection [49, 50, 51], moving object
detection [52, 53, 54], face detection [55, 56], traffic sign detection [57,
58, 59] and so on. Oksuz et al. [60] and Chen et al. [61] presented imbalance
problems existing in deep-neural-network based detectors, e.g., foreground-
background imbalance. Unlike previous surveys, we focus on the few-shot
challenge for object detection which does not systematically appear in
previous surveys.

As aforesaid, few-shot learning has received great attention in the computer
vision community and many well-written works have summarized it profoundly
[62, 63, 64, 65, 66, 67, 68]. Wang et al. [62] and Kadam et al. [63] indicated
core issues in few-shot learning and grouped few-shot approaches into three
categories (i.e., data-, model- and algorithm-based methods). Li et al. [64]
provided a comprehensive analysis of meta-learning methods. Jadon [65]
discussed deep-neural-network architectures designed for few-shot learning.
Apart from these surveys for generic few-shot learning, several surveys also
put emphasis on specific applications of few-shot learning: COVID-19 diagnosis
[66], natural language processing [67] and computer vision [68]. Here, except
classic $N$-way $K$-shot settings, we also discuss imbalance problems, semi-
supervised settings and weakly-supervised settings for FSOD, while previous
surveys failed to cover these concepts.

In addition, FSOD aims at learning to simultaneously localize and classify all
instances from a few labeled data, which is more challenging and under-
explored. We notice that existing approaches are still fragmentary and
unsystematic and there is no relative survey to present the current
development status and tendency in FSOD, which is detrimental to carry out
solid researches. Consequently, it is essential to present a comprehensive
survey of related works on FSOD and reveal inner relations and motivations
about how they promote development of this promising task. We hope that our
thorough survey can provide insights for further research.

###  I-B Taxonomy

Although abundant excellent FSOD works [39, 40, 41] have been published
recently, they are proposed under different settings or for separate
objectives in terms of data settings, training strategies and network
architectures, where it is improper to discuss them together. Due to limited
supervision, most few-shot detectors must rely on extra datasets to give an
appropriate initialization for these generic yet heavy-weight frameworks,
while there were large discrepancies among their settings of extra datasets.
Thus, according to data and associated supervision which could be accessed
during the training stage, as shown in Tab. I, we group these approaches into
three categories - limited-supervised based FSOD (LS-FSOD), semi-supervised
based FSOD (SS-FSOD) and weakly-supervised based FSOD (WS-FSOD), respectively.
In LS-FSOD, there is only a small dataset $D_{novel}$ with a few instance-
level labeled exemplars of each novel class to learn novel task notions and an
optional dataset without target supervision to learn generic notions. Unlike
LS-FSOD, there is an extra target-domain dataset $D_{novel}^{-}$ without
annotations in SS-FSOD to enforce few-shot detectors to automatically capture
implicit objects for reducing labour force. In WS-FSOD, there is a small
dataset $D_{novel}^{+}$ with a few image-level labeled exemplars of each novel
class to enforce few-shot detectors to mine implicit relations among image-
level tags and associated objects. In some cases, we also include a target-
domain dataset $D_{novel}^{-}$ and a base dataset $D_{base}$ to compensate
inaccurate supervisory signals from $D_{novel}^{+}$ to make the training
process more stable.

TABLE I: Comparison of three main types of FSOD. $\surd$/$\times$/$\bigcirc$ indicates inclusive/exclusive/optional, respectively. Type | $D_{novel}$ | $D_{novel}^{+}$ | $D_{novel}^{-}$ | $D_{base}$
---|---|---|---|---
LS-FSOD | $\surd$ | $\times$ | $\times$ | $\bigcirc$
SS-FSOD | $\surd$ | $\times$ | $\surd$ | $\bigcirc$
WS-FSOD | $\times$ | $\surd$ | $\bigcirc$ | $\bigcirc$

In all, it is infeasible to only rely on limited labeled data to learn a
robust model. As aforesaid, even a child could perform so well with very
little training data and it raises a question of why humans could quickly
adapt to a new task. Dubey et al. [69] proved that humans can exploit
history/prior knowledge to tackle their confronted issues. Referring to the
human learning process, the key is how to extract task-agnostic notions from
$D_{base}$ or capture task-specific guidance from $D_{novel}^{+}$ or
$D_{novel}^{-}$, which is helpful for a novel few-shot task. However, due to
supervision differences, there exist huge technical gaps among three types of
settings. For example, compared with LS-FSOD, SS-FSOD could exploit a large-
scale unlabeled dataset that contains objects of novel classes during
training, which means SS-FSOD must introduce a special framework to mine
underlying objects or generic notions from unlabeled data (Section IV). In the
following sections, we will elaborate the specific definition, challenges and
existing approaches respectively for all kinds of FSOD. In conclusion,
according to above requirements, we summarize a set of tools for FSOD, as
shown in Fig. 2.

Figure 2: Structure of this survey.

###  I-C Main Challenges

Currently, few-shot detectors still adopt classic deep-learning frameworks,
e.g., R-CNN, YOLO and SSD variants [30, 31, 11], which are inevitable to
confront with intrinsic challenges, e.g., imbalance problems [70, 71, 72],
large intra-category variations [73] and low inter-category distance (fine-
grain problems) [74, 75]. Besides, limited supervisory signals could further
make some issues more serious, where low-density sampling has a high
probability to lead to nasty data distributions, e.g., high intra-class
variations, low inter-class distance and data shift. Thus, few-shot detectors
still need to develop suitable learning strategies to overcome the degradation
phenomenon that deep-learning approaches are prone to acquire irrelevant
features (i.e., overfitting), which cannot be corrected automatically due to
inadequate support. Here, we will discuss main issues that make the training
process more challenging:

Figure 3: Performance of RPN with large/small domain shift. The backbone and
RPN are pretrained on the inat2017 dataset [76] (only animals & plants). Bird
is a class of the inat2017 dataset while the RPN generates good proposals.
Although aeroplane is not a class of the inat2017 dataset, aeroplane shares
similar visual components with bird and gets comparable results. TV/monitor
has large domain differences with the inat2017 dataset and gets worse results.
Figure 4:  (a) Several exemplars of data bias. There exist noisy backgrounds
and pose variations among these training images. (b) The negative effect of
data bias. The quality of training images sampled from the image pool could
seriously influence final performance.

  * •

Domain Shifts. Numerous FSOD approaches utilized a large-scale dataset to
learn generic notions that are subsequently transferred/fine-tuned to meet
requirements for a novel task. In some cases, the source domain shares little
cross-domain knowledge with the target domain, i.e., large domain shifts,
where “generic” notions learned from the source domain produce weak or even
negative effect for the target task. For instance, many works hold a principle
that the region proposal network (RPN) was an ideal proposal algorithm which
could generate high-quality regions of interest (RoIs) for all foreground
classes [29, 77, 78, 79]. However, foreground classes, which are defined as a
set of classes of interest, are task-specific, while all other classes are
defined as negatives. Therefore, such a class-agnostic RPN cannot provide RoIs
for novel classes as good as that for base classes, especially with large
domain shift [80, 81]. As illustrated in Fig. 3, the similarity between the
base and novel classes has a great effect on the quality of RoIs for novel
classes. In addition, the low quality of RoIs (e.g., TV/monitor) will degrade
the subsequent training of detection heads. Thus, generic notions should be
used carefully when existing domain shift.

  * •

Data Bias. A dataset is essentially a collection of observed exemplars from a
special data distribution. In reality, there are large intra-class variations
even for the same objects, such as appearance, posture and so on, while large
intra-category variations could relatively fuzzy decision boundaries, as
illustrated in Fig. 4. Unlike large-scale datasets, it is impossible to cover
all situations for a small-scale dataset that naturally has more data bias in
scale, context, intra-class diversity and so on [38]. Due to large capacities
for deep-learning methods, they could be susceptible to noise/bias to utilize
non-robust notions to make decisions (i.e., overfitting). Especially, metric-
learning based methods need to leverage the training set to learn a set of
robust category prototypes as task-specific parameters. It is hard to build
robust class prototypes when the training set has many outliers, such as
occlusion.

Due to low-density sampling, there may exist domain shifts among the training
dataset and the testing dataset and various dataset splitting may produce
unstable results. For a relatively reliable result, the average performance
from multiple runs is a feasible way.

  * •

Insufficient Instance Samples. Insufficient supervision could amplify implicit
noise and data bias in the dataset, which could easily lead traditional and
deep-learning approaches to overfitting or even underfitting. Especially,
insufficient supervision tends to form a loose cluster for each class and it
is unfeasible to learn a robust detector via increasing the intra-category
diversity, such as abundant simple and naive data augmentation methods.

  * •

Inaccurate Supervisory Signals. In WS-FSOD, this issue is mainly caused by the
ambiguous relation among image-level tags and associated objects. Due to
inaccurate supervision, it has some difficulty associating each image-level
tags with the whole objects appeared in this image and measuring the quality
of these proposals, where it tends to associate these image-level tags with
the most discriminative parts of these objects in most cases. In addition,
insufficient image-level tags could further have a negative effect on this
process, where it may have more inaccurate proposals.

  * •

Incomplete Annotation. As aforesaid, few-shot detectors usually need extra
datasets to learn robust notions to initialize these heavy-weight learning
frameworks. Due to annotation discrepancies, there may exist objects of novel
categories in the base dataset $D_{base}$ and it is time-consuming and
laborious to relabel these objects. When few-shot detectors are pre-trained on
this base dataset, they could treat these objects of novel categories as
negatives and learn to suppress these objects, which is harmful to detect
novel-class objects. Actually, this issue could be viewed as a semi-supervised
problem. Thus, Li et al. [82] exploited a semi-supervised solution to mine
these background proposals which probably contained objects of novel
categories and assign pseudo labels to these proposals to train a novel
detector. Due to lack of supervision, pseudo boxes could contain too many
background regions and propagate noise into the subsequent training process.

##  II Background

In this section, we propose a brief review of advances in object detection,
few-shot learning, semi-supervised learning and weakly-supervised learning. To
present a better understanding of this paper, it is essential for readers to
have some background knowledge of these topics. However, if you want to
acquire detailed background knowledge, we suggest several latest and
comprehensive surveys for readers [62, 45, 83, 84].

###  II-A Advances in Object Detection

Figure 5: (a) The pipeline of a classic two-stage object detection method
(Faster R-CNN [9]), (b) The framework of a classic single-stage object
detection method (YOLO [85]).

As aforementioned, object detection focuses on simultaneously localizing and
classifying all objects of classes of interest. Currently, mainstream
detection frameworks could be usually categorized into two types, i.e., one-
and two-stage detectors. The main difference between one- and two-stage
detectors is that two-stage detectors (e.g., R-CNN [30]) follow the coarse-to-
fine mechanism (i.e., the coarse and fine localization) while one-stage
detectors (e.g., YOLO [85]) directly make final results without the extra
refinement process (Fig. 5). Two-stage detectors first take a region proposal
algorithm (e.g., RPN [9] and selective search (SS) [28]) to screen preliminary
class-agnostic candidates (the coarse localization), then extract fixed-length
vectors for all candidates by ROI-Align/Pooling, and finally feed them into
two parallel branches and post-processing algorithms (e.g., non-maximum
suppress (NMS) [28]) to produce final proposals (the fine location) (Fig.
5(a)). One-stage detectors, like SSD/YOLO/CornerNet-style algorithms [11, 85,
86, 87], directly generate a set of bounding boxes and associated category
probability distributions for each spatial location, which are similar to the
class-specific RPN (Fig. 5(b)).

Likewise, these classic frameworks could be grouped into anchor-based and
anchor-free detectors as well, according to whether to exploit prior anchors
during the process of proposal generation. Prior anchors are a set of pre-
defined boxes with various aspect ratios and sizes and are initially used to
provide fairly good reference for RPN to avoid too large search space.
However, it is non-intuitive to specify hyper-parameters (i.e., the aspect-
ratios, sizes and number) of anchors. Although several works (e.g., dimension
clusters [10], GA-RPN [88]) have been proposed to tackle such problems, we
cannot completely drop all hyper-parameters of prior anchors. In 2018, Law, H.
and Deng, J. [86] re-introduced the anchor-free mechanism and viewed it as a
task of keypoint detection and matching. So far, keypoint has many kinds of
definitions, such as left-top and right-bottom corners of objects [86], four
extreme points (top-most, left-most, bottom-most, right-most) and one center
point of objects [89], etc.

Except for generic approaches above, there exist many excellent approaches
designed to tackle special problems, such as the imbalance problem [90, 91,
92, 93], the real-time problem [10, 85, 11], small target detection [94, 95,
96, 97], fine-grained object detection [98, 74, 75, 99], few-shot learning
[39, 29, 80] and so on. These works have further promoted applications of
object detection in real scenes.

###  II-B Advances in Few-Shot Learning

Few-shot learning aims to learn to enhance the generalization ability with
limited labeled data. In the deep-learning era, solutions of few-shot learning
can be classified into three main types: meta-learning, transfer-learning and
data augmentation methods. Meta-learning methods usually use abundant
episodes/tasks to acquire task-agnostic notions (e.g., meta-parameters), which
can be meaningful to quickly adapt to a new task. In addition, meta-learning
methods can be further divided into three types, i.e., metric-, optimization-
and model-based methods. Metric-based methods pay attention to learn a robust
embedding function and a scoring function that measures similarity between
embedding vectors of a query image and each class prototype[14, 15, 100, 16,
17]. Sung et al. [100] proposed an end-to-end automatic scoring module to
measure similarity. Koch et al. [14] proposed a siamese network to obtain
representatives for both query and support images, then utilized L1 distance
to fuse these representatives, and finally fed the fused features to a MLP for
evaluating similarity. In addition, metric-learning was also formed as a task
of an information retrieval [17]. Optimization-based methods attempted to
learn a meta-optimizer or meta-parameters for quick adaption to a new task
[20]. MAML variants [20] took a two-step strategy to learn meta-parameters for
a given task. Meta agents (e.g., LSTM) were designed to learn updating rules,
such as learning rate. Model-based methods design a specific network
architecture and corresponding learning strategies for quick adaption for a
novel task [25, 26, 101, 102, 22]. Several works [26, 22] grouped weights into
two types (i.e., task-specific and task-agnostic parameters) and only update
task-specific weights for quick and robust adaption to a new task. NTMs [101]
and MANN [22] introduced LSTMs and cache pools for quickly generating task-
specific parameters. Transfer-learning methods mainly rely on fine-tuning
general notions from source datasets to a novel task without training from
scratch. Chen et al. [27] proved that simple transfer-learning methods were
much effective, even with large domain shifts.

Furthermore, several works explored data augmentation for increasing data
diversity to mitigate overfitting. A GAN variant was proposed to transfer
intra-class variations from base classes to novel classes for more robust
prototypes [103, 104].

###  II-C Advances in semi-supervised learning

To get rid of over-dependence on abundant labeled data for deep-learning based
methods, a large number of semi-supervised methods explore a new paradigm to
enforce a learner to automatically acquire instance-level notions from
partially annotated data [105, 106, 107]. Clearly, there exists hidden target-
domain knowledge in unlabeled data and the key is to accurately extract this
knowledge to regularize detectors. In machine learning, a classic way was to
learn a teacher model from annotated data first, then apply it to generate
pseudo labels as ground-truth labels for unlabeled data, and finally sample a
list of reliable pseudo labels to train a student model [108, 109]. The
quality of pseudo labels played a key role in the training process of the
student learner. Therefore, multiple teacher models were ensembled to work
together to produce stable pseudo labels [110]. Besides, several works
proposed graph-based methods which used all labeled and unlabeled data and
their mutual relations to build a graph for label propagation to obtain pseudo
labels for unlabeled data [111, 112].

In addition, several works took it as a clustering problem and hold a core
hypothesis that all exemplars should have high inter-class distinction and low
intra-class variations [113, 114]. Thus, S3VM variants are designed to find a
proper decision boundary which ought to pass through low data-density regions
[114].

Recently, deep-learning approaches have received large attention in the field
of semi-supervised learning due to their large capacity [105, 106, 107, 115,
116]. Similarly, it is sub-optimal to simply use labeled data and drop
underlying notions of unlabeled data in training phase. CNNs were equipped to
mine pseudo labels in unlabeled data to train a student learner. Except for
pseudo labels, unlabeled data is also exploited to learn a pretext task (e.g.,
a reconstruction task) which enforces a learner to keep consistent among
multiple models or recover raw signals from features for a better encoder,
e.g., $\pi$-model, temporal ensembling and mean teacher [115, 116, 117, 118].

###  II-D Advances in weakly-supervised learning

To lower instance-level annotation burdens, weakly-supervised object detection
(WSOD) attempts to exploit a relative cheap alternative (e.g., image-level
tags or object locations) and automatically mine underlying cues among weak
supervision and objects for a novel task, which has received much attention
recently [119, 120, 121, 122].

However, unlike instance-level boxes, these cheap alternatives cannot be
directly applied to guide models to localize objects and instead bring
uncertain supervisory signals into the training process. In weakly-supervised
learning, it tends to propagate image-level tags for estimating instance-level
boxes as ground-truth pseudo labels for weakly annotated data. Thus, weakly-
supervised approaches have to tackle uncertainty problems raised by imprecise
supervision, where detectors do not have a suitable way to measure the quality
of pseudo boxes. Uncertainty problems tend to conclude with two aspects, i.e.,
low-quality pseudo boxes and inaccurate labels. Generally, detectors tend to
associate image-level tags with the most discriminative parts of objects which
leads to too small pseudo boxes [123, 124]. Likewise, if multiple instances
are clustered in unlabeled images, detectors could even attempt to utilize a
large box to cover all instances which brings too large pseudo boxes.
Moreover, when negatives have a certain IoU with foreground objects, it is
hard for detectors to suppress background proposals for high-quality pseudo
boxes and thus generate inaccurate labels. Thus, it still has huge performance
gaps between state-of-the-art weakly-supervised and fully-supervised
approaches so far [125, 126].

For high-quality pseudo labels, it could be grouped into two kinds, i.e.,
initialization [127, 128] and refinement [123, 126, 129, 124]. Initialization
aims to distinguish suitable pseudo boxes from abundant proposals, using prior
knowledge. There exist many kinds of prior knowledge, including
saliency/foreground heatmaps, inter-category variations, intra-category
similarity, object co-occurrence and so on. As for refinement, it mainly
focuses on learning strategies which could alleviate negative effects from
inaccurate pseudo boxes, e.g., multiple instance learning (MIL) [123, 126,
129] and class activation map (CAM) [124, 130]. MIL-style approaches aim to
mine underlying evidence among image-level tags and all RoIs in a specific
image, i.e., class probability distributions for each RoI and object
distributions for each class, where two kinds of evidence could be aggregated
into class probability distributions for such an image. It could guide models
to automatically distinguish various kinds of instances under a setting with
reliable supervision, instead of directly assigning a pseudo label for each
box. CAM-style approaches attempt to use image-level labels to learn a
classifier first and slide its weights upon image features to generate class-
sensitive heatmaps which could be segmented as bounding boxes. Except these
works, there exist a lot of works that restricted their scope into some
specific classes, e.g., pedestrians [131, 132, 133], vehicles [134, 135, 136],
face [137] and so on.

##  III Limited-Supervised Few-Shot Object Detection

LS-FSOD is a classic issue in the few-shot learning which only relies on very
limited supervision to learn task notions for novel classes. To alleviate
overfitting, it usually exploits large-scale open-access datasets [32, 76, 33,
138] to mine task-agnostic notions which are helpful to any other tasks. To
achieve stable support from the base dataset, it requires that the source task
should share some generic notions with the novel task (Section I-C). In this
section, LS-FSOD can be grouped into two types: balanced and imbalanced LS-
FSOD, according to whether there exists the foreground-foreground imbalance
problem in the novel dataset. Here, we have mainly discussed solutions of the
former and propose candidate solutions for the latter, although the latter
hasn’t been raised great attention.

###  III-A Problem Definition

As shown in Tab. I, let $C_{base}$ be a set of classes in a large-scale
dataset $D_{base}$. Similarly, $C_{novel}$ is a set of classes in a small-
scale dataset $D_{novel}$ with instance-level labels. Here, we assume
$D_{base}$ with instance-level labels for simplicity. Note that $C_{base}$ and
$C_{novel}$ are disjointed, i.e., $C_{base}\cap C_{novel}=\emptyset$. For each
sample $(I,Y)$ in $D_{base}\cup D_{novel}$, $I$ is an image
($I\in\mathbb{R}^{M\times N\times 3}$) and $Y=\\{(b_{n},y_{n})\\}^{N}$ is a
list of $N$ objects in $I$, where $b_{n}\in\mathbb{R}^{4}$ is the bounding box
of the n-th instance and $y_{n}\in\\{0,1\\}^{|C_{base}\cup C_{novel}|}$
denotes an associated one-hot label encoding. Especially, we mainly evaluate
the performance of the novel categories $C_{novel}$.

Let $D_{novel}^{n}$ be a set of all objects of the n-th class in $D_{novel}$.
For $D_{novel}$, we restrict the maximum of the number of instances per class
in $D_{novel}$: $\max{\\{|D_{novel}^{i}|,i\in C_{novel}\\}}\leq k$ ($k$ is
usually no more than 30). Generally speaking, LS-FSOD attempts to acquire
generic notions to mitigate too large parameter search space with very limited
supervision from $D_{novel}$.

###  III-B $N$-Way $K$-Shot Limited-Supervised Problem

####  III-B1 Definition

In the $N$-way $K$-shot setting, $N$ denotes the number of categories in
$D_{novel}$ (i.e., $|C_{novel}|$) and $K$ is the number of objects per
category (i.e., $\max{\\{|D_{novel}^{i}|,i\in
C_{novel}\\}}=\min{\\{|D_{novel}^{i}|,i\in C_{novel}\\}}=k$).

####  III-B2 Dataset

For convenience, benchmarks of the $N$-way $K$-shot problem consist of two
sub-benchmarks (i.e., the base and novel dataset) which are usually built upon
existing generic OD benchmarks, e.g., PASCAL VOC 07/12 [138], MSCOCO [33] and
ImageNet-LOC [32]. For a detailed review on these generic object detection
datasets, we refer readers to the latest and comprehensive surveys [139, 140].
Here, we list common settings of benchmarks in $N$-way $K$-shot FSOD in Tab.
II. Especially, in PASCAL VOC 07+12 [138], the splitting settings have been a
standard configuration in FSOD, i.e., {(bird, bus, cow, motorbike, sofa /
rest), (aero, bottle, cow, horse, sofa / rest), (boat, cat, motorbike, sheep,
sofa / rest)} [29, 141, 40, 38]. Similarly, the splitting setting of FSOD and
MSCOCO has been publicly available, as shown in Tab. II. However, the
novel/base class splittings of an existing dataset may cause that the base
dataset has many objects of novel classes, which is similar to the incomplete
annotation problem. To tackle this problem, a simple way is to remove all
images with objects of novel classes in the base dataset $D_{base}$ or view
all instances of novel categories as background [29]. To make full use of
latent exemplars in the base dataset $D_{base}$, Li et al. [82] allocated
pseudo labels for these negative proposals which not only have low IoU with
all ground-truth boxes but also have high similarity with novel category
prototypes to learn a more reliable detector in meta-training stage (a semi-
supervised style solution). Classic metrics in MSCOCO and $mAP_{50}$ are
generally exploited to evaluate the actual performance of FSOD methods.

TABLE II: Common settings of benchmarks in $N$-way $K$-shot FSOD. Note that $B$/$N$ represents base/novel categories. Meanwhile, we also give a set of frequently-used values of $K$. Benchmarks | Settings
---|---
PASCAL VOC 07+12 | 5B & 5N ($K\in\\{1,2,3,5,10\\}$) [29, 141, 40, 38]
MSCOCO | 60B & 20N (overlapped with PASCAL VOC 07+12) ($K\in\\{10,30\\}$) [29, 141, 40, 38]
FSOD | 300/500/800B & 200N ($K\in\\{5\\}$) [80]
MSCOCO $\rightarrow$ PASCAL VOC 07+12 | 60B (MSCOCO) & 20N (PASCAL VOC 07+12) ($K\in\\{10\\}$) [29, 141, 38]
MSCOCO $\rightarrow$ ImageNet-Loc | 80B (MSCOCO) & 50N (ImageNet-Loc) ($K\in\\{5\\}$) [80]
MSCOCO $\rightarrow$ FSOD | 80B (MSCOCO) & 200N (FSOD) ($K\in\\{5\\}$) [80]

####  III-B3 Solutions

There are three types of solutions for the $N$-way $K$-shot problem: (a) meta-
learning methods [29, 80, 142, 67], (b) transfer-learning methods [143, 39,
40], (c) data augmentation methods [144]. As aforesaid, meta-learning (learn-
to-learn) puts more emphasis on exploiting $D_{base}$ to elaborately design a
set of tasks to learn task-agnostic notions for quick adaption to a new task
when compared with transfer-learning. Data augmentation methods highlight how
to increase the intra-category diversity.

_1\. Meta-Learning Methods._ In the meta-learning, a task can be formulated as
$\mathcal{T}_{i}={(\mathcal{S}^{1},\mathcal{S}^{2},\ldots,\mathcal{S}^{N},Q)}$
where $\mathcal{S}^{j}$ denotes the j-th support images and Q is a query
image. First, we exploit $D_{base}$ to construct a meta-training task set
$\mathcal{T}^{train}$ for acquiring generic notions. Then, a fine-tuning task
set $\mathcal{T}^{finetuning}$ is adapted to fit in the new task space, which
is unnecessary for metric-based methods. Finally, a meta-testing task set
$\mathcal{T}^{test}$ is applied to evaluate the adapted model. Especially,
meta-learning methods can be further grouped into metric-, model- and
optimization-based methods, according to the way to acquire task-agnostic
notions in the meta-training stage.

Figure 6: Structure of Two-Stage Metric-Based Methods. Ⓕ represents a fusion
node and an feature aggregator. Note that we only show several fusion nodes
for simplicity and more details about fusion nodes are in Fig. 12 and Tab.
III.

Metric-based Methods. In this section, we will make a detailed review on
metric-based detectors from ten aspects, i.e., the data processing, the
embedding network, RPN vs meta-RPN, support-only vs support-query guidance,
the aggregator, the scoring function, the loss function, the fusion node, the
training/testing process and other settings for the support branch, as shown
in Fig. 6. These methods have two data flows, i.e., the support and query
branch. The support branch takes responsibility for providing task-specific
parameters and the query branch is in charge of combining task-specific
parameters and query features to generate proposals. These branches exchange
information via various fusion nodes and aggregators. To tackle the few-shot
issue, existing methods have proposed customed solutions in each aspect, while
they are fragmentary and unsystematic. Thus, we group their works into these
aspects and give a brief review for each approach in Tab. X as well, if you
want to know the complete strategy for a specific method. In addition, we also
provide a detailed analysis of how they interactive to have huge performance
gains, as shown in Fig. 6.

  * •

Data Preprocessing. In the support branch, all support instances for a
specific category are usually extracted from support images by ground-truth
boxes to generate a list of fixed-length category representatives for this
category (Fig. 7). However, this approach ignores contextual information
(e.g., the co-occurrence of objects), which can be employed to exploit inter-
category relations to get better class representatives. For more contextual
information, Fan et al. [80] and Han et al. [145] directly added 16-pixel
image context to each support instance. Wu et al. [81] extracted instance
representatives from feature maps to implicitly use contextual information
outside instances to enhance class representatives. Another way is to leverage
ground-truth boxes to generate a binary foreground mask
$M\in\mathbb{R}^{H\times W}$ that tends to be stacked with the corresponding
image along the channel axis (i.e., $[I,M]$) to guide the network not only to
focus on object area but also to acquire contextual information [29, 79, 146,
141]. To learn a more robust detector, Li et al. [79] proposed a feature
disturbance method to augment $M$, which truncated 15% pixels in $M$ to zero.
Here, 15% pixels were chosen as these with larger gradient, which tended to
correspond the most discriminative features of objects, to enforce it to
explore other equally good features (like dropout). In addition, the shape
information of objects is essential to build appropriate representatives for a
special category, e.g., the aspect ratios. To keep the structure of instances,
there exist two main solutions: (1) prior to resize cropped patches to a fixed
size, zero-padding is adopted to adjust aspect ratios of cropped patches
[147]; (2) a RoI-Pooling/Align layer is applied to get fixed-length features
[148, 80].

Figure 7: The process to generate a fixed-length category representative.

  * •

Embedding Network. An embedding network (backbone) is pretty important in
metric learning [15, 14]. It consists of parallel sub-networks $f$ and $g$
(usually $f=g$) respectively for the query and support set. For the support
set, all instances of a specific category are fed into $g$ to generate a list
of instance representatives, and a clustering algorithm (e.g., k-means/median)
is then applied to generate category prototypes from these instance
representatives of a given category [149, 78]. Likewise, a query image is fed
into $f$ to generate corresponding query features for matching.

In the metric learning, it relies on an important hypothesis that object
features in a query image should have high similarity with prototypes of the
corresponding category. However, due to low-shot scenarios, it is likely to
have noise in the support set, such as occlusion, which is harmful to generate
category prototypes. To solve such a problem, Li et al. [146] proposed a TIP
to impose a consistency constraint on the features of an image and a
corresponding corrupted image obtained by applying transformations on the
image to make $f$/$g$ invariant to transformations. A similarity-based
sampling strategy was designed to pair a query image with the most similar
support instances in the training stage to alleviate intra-class variations
[149] while it could use biased class prototypes during the inference stage.
Similarly, scale differences among objects of the query and support set could
violate the semantic consistency to confuse the scoring network. Singh et al.
[150] proved that there was a significant semantic difference between features
of a given image at various scales, even if the siamese network was used to
compute features. FPN was employed to match features of the support and query
set at each scale to reduce scale differences. Nevertheless, FPN introduced
more negative proposals while positive proposals were limited. Namely, it
enlarged the foreground-background imbalance. To get more positive samples, Wu
et al. [38] designed object pyramids to provide more positive supervision for
each level of FPN. Zhang et al. [151] presented a multi-scale fusion module,
which adopted up-sampling (i.e., bilinear interpolation) and down-sampling
(i.e., 1x1 strided convolution) methods to map all features to the same scale,
to explicitly mix scale information into feature maps. Meanwhile, unlike FPN,
it greatly reduced negative proposals.

  * •

RPN vs Meta-RPN. In RPN, it was regarded as a category-agnostic algorithm
which took a foreground-background classifier to screen RoIs regardless of
their actual category. Earlier, most works supposed that a pretrained RPN
could generate high-quality proposals for a novel task, and tended to freeze
all parameters of such an RPN to avoid overfitting [29, 77, 78, 152, 153]. In
the pretraining stage, only a base dataset with limited classes available was
utilized to learn such a class-agnostic RPN which could not cover all
situations to produce the same good proposals for unseen classes, especially
with large domain shift. As aforesaid, the notion of foreground classes is
task-specific. To tackle this problem, traditional RPN was revised as meta-RPN
which took query features conditioned by a set of prototypes for each category
as input and only output associated RoIs of that category (Fig. 8(b)) [145,
154, 149]. Meta-RPN relied on a hypothesis that a set of prototypes of a given
category could provide class-specific notions for RPN not only to suppress
heterogeneous/background features but also to enhance similar semantics for
more high-quality proposals. Zhang et al. [155] proposed CoRPNs to ensemble
multiple independent yet cooperative RPN to improve the quality of proposals
(Fig. 8(c)).

Figure 8: (a) The pipeline of original RPN; (b) The architecture of meta-RPN.
Let Ⓐ be a feature aggregator which takes query features and a set of
representatives of a given category (i.e., $v_{c}={v_{1},\ v_{2},\ldots,\
v_{k}}$) as input and outputs category-specific query features that are fed to
subsequent RPN; (c) The architecture of CoRPNs. Unlike original RPN, it
consists of multiple Fg/Bg classifiers which is independent yet cooperative.
Figure 9: The pipeline of support-only and support-query guidance. Especially,
only (a) and (b) are support-only guidance while the other are support-query
guidance. (a) Adaptive Fully-Dual Network (AFDN) [29]; (b) Hyper Attention RPN
(HA-RPN) [151]; (c) Dense Relation Distillation (DRD) [156]; (d) Non-Local Co-
Attention (NLCA) [142]; (e) Augmentation with Conditioned Prototypes (ACP)
[157]; (f) Multi-head Co-Attention (MHCA) [158]; (g) Cross-Image Spatial
Attention Block (CISA) [147]; (h) Spatial Alignment & Foreground Attention
Module (SAFA) [145].

  * •

Support-Only vs Support-Query Guidance. To better integrate category-specific
information into query features, there were two main ways to refine category
prototypes, i.e., support-only and support-query guidance. In support-only
guidance, it only takes category prototypes to refine features. Clearly,
simple global max/average pooling was a kind of support-only guidance to
capture global semantics which was helpful for the classification task [41,
141, 159]. However, it also lost spatial information (e.g., local structures)
which was critical for RPN to make high-quality proposals. An effective way
was to view a set of category prototypes as kernels to slide over query
features for feature fusion. Liu et al. [29] designed a self-attention based
fully-dual network that consisted of two parallel branches to simultaneously
capture global and local semantics to generate task-specific attention vectors
(Fig. 9(a)). Zhang et al. [151] designed a list of PN (e.g., SigmE) to filter
irrelevant factors in category prototypes (Fig. 9(b)).

In fact, objects in the support set usually have large differences with that
in the query set, e.g., various postures and viewpoints. Meanwhile, support-
only guidance does not take the misalignment problem between query and support
features into consideration. Thus, the scoring function could easily confuse
whether the RoI features are consistent with the prototypes of the associated
category. To tackle the misalignment problem, support-query guidance uses the
affinity matrix between query and support features to align features. As
illustrated in Fig. 9(c, e, g, h), these methods followed such a framework
that first employed two parallel branches to get key and value maps for both
query and support images, then took key maps to calculate an affinity matrix
$A$ between query and support images, and finally applied the affinity matrix
to align value maps of support to enhance value maps of query. Especially, it
could be viewed as a search process that we search all spatial location of key
maps of support to aggregate support evidence for feature fusion (like the
mutual information). To refine the affinity matrix, Chen et al. [147]
presented a new branch which applied a fully-connected layer with spatial
softmax to generate a pseudo foreground mask, which was used to reweight the
affinity matrix, to filter background information. Similarly, Han et al. [145]
leveraged the amount of category evidence (i.e., sum up the affinity matrix by
row) to determine each spatial location in query features whether or not a
foreground region. Meanwhile, in Fig. 9(d, f), several works explored a new
paradigm which the affinity matrix was applied not only to align value maps of
support to enhance value maps of query but also to align value maps of query
to enhance value maps of support. In fact, such a scheme utilized a kind of
the mutual relationship that both query and support objects would fetch enough
information from each other if they were the same category while they would
fetch very little from each other if They were of a different class. Thus, it
could amplify the gap between the consistent pair and the other inconsistent
pairs. Hsieh et al [142] shared the affinity matrix between two parallel
branches of query and support while Chen et al. [158] took separate affinity
matrixes for each branch. Zhang et al. [154] designed a QSW module to build a
category prototype from multiple support objects of a given category by their
relevance to a query RoI.

  * •

Aggregator. As for feature fusion, there existed many excellent feature
aggregators, e.g., channel-wise multiplication (MUL), element-wise subtraction
(SUB) and channel-wise concatenation (CAT). MUL was commonly known as feature
reweighting to learn feature co-occurrence. SUB was a kind of distance metric
(i.e., L1 distance) to measure L1 similarity. CAT was a special aggregator
that could stack features along the channel axis for subsequent networks to
automatically explore a good way for feature fusion. Most existing methods
only adopted MUL as their feature aggregator. Nevertheless, Han et al. [145]
and Liu et al. [29] combined MUL, SUB and CAT for better feature fusion (Fig.
10). In addition to traditional aggregators, another way was to transform
category prototypes into a set of parameters, which were employed to
initialize all weights of operators (e.g., a convolution operator), to enhance
query features [154].

Figure 10: The pipeline of feature fusion. (a) Dual Feature Aggregation; (b)
Feature Fusion Network. Figure 11: The pipeline of variants of RNs [100]. (a)
A Two-Layer MLP Relation Network; (b) Multi-Relation Network.

  * •

Scoring Function. A scoring function is another essential element for a
metric-based method, including cosine similarity [77], Pearson similarity
[152] and variants of relation networks (RNs) [29, 80, 158]. Especially,
cosine and Pearson similarity were fixed metrics which may not fit a given
task. To give a flexible metric for FSOD, variants of RNs learned a soft
metric to recognize various categories. Most existing methods followed the
one-vs-rest strategy to design a binary classifier (named the mete classifier)
which should effectively distinguish whether RoI features were consistent with
a set of prototypes of a given category or not (Fig. 11(a)). A multi-relation
network was designed to capture global-, local- and patch-relation between RoI
features and class prototypes for overall matching (Fig. 11(b)) [141, 151].
Zhang et al. [148] regarded the scoring function as the transformer decoder to
classify each query.

Unlike the task of image recognition, an object detection method should
distinguish negative proposals from all proposals, especially while negative
proposals could even have a certain overlap (i.e., similarity) with associated
class prototypes which may confuse the scoring function. To some extent, it
was hard to define a background category to represent various negative samples
with large intra-class variations. To tackle this problem, YOLO variants [41,
79] took the confidence score to indicate the associated box whether or not a
positive box. As aforementioned, R-CNN variants [29, 80] employed a binary
classifier to measure the similarity between RoI features with a set of
prototypes for each class, where an RoI was assigned with a class tag that had
the highest similarity score with this RoI. If the highest score was lower
than a pre-defined threshold, it would be defined as a negative proposal. Yang
et al. [78] used negative proposals which had low IoU with ground-truth boxes
to learn a list of negative prototypes for the background category. Bansal et
al. [77] explored two kinds of background definitions. First, a fixed vector
was utilized as the prototype for the background class (statically assigned
background). However, it could not really encode all background information,
which could thus produce more false-positive samples. Second, in each
iteration, we labeled background boxes with pseudo labels randomly sampled
from a list of unseen categories to distinguish negatives from other positive
proposals. Li et al. [152] randomly sampled boxes, which had no overlap with
ground-truth boxes, as exemplars of the background category to filter negative
proposals of RPN.

  * •

Loss Function. A well-defined loss function was essential to learn a good
model for a metric-based method. Many works inherited the loss functions from
generic object detection methods, which were originally designed to guide the
detectors to accurately detect all objects in an image. However, it did not
take class prototypes into account, which played a key role in a metric-based
method. Thus, several works introduced the customed loss function which could
guide the embedding network to simultaneously minimize intra-class variations
of prototypes and maximize inter-class distance of prototypes. Yan et al.
[141] appended an MLP with softmax to the embedding network and employed
cross-entropy (CE) loss to encourage the embedding network to explicitly
encode category information into category prototypes. Fan et al. [80]
presented a two-way contrastive training strategy. It first constructed a
training triplet $(q,s_{c},s_{n})$ where $q$ was a query image and $s_{i}$ was
a support instance of the i-th category ($c\neq n$). Then, it combined $s_{c}$
and $q$ to generate category-specific features fed to RPN to generate a set of
proposals $p$, where only proposals of the c-th category were regarded as
positive proposals. Next, it proportionally sampled training pairs to
construct a balanced training set, consisting of three types of pairs, i.e.,
$(p_{c},s_{c})$, $(p_{b},s_{c})$ and $(p_{\ast},s_{n})$ where $p_{i}$ was a
proposal of i-th category ($b\neq c$). Finally, it enforced a binary cross-
entropy (BCE) loss to learn a good representative. Margin-based ranking loss
[158] was a kind of multi-task loss, consisting of two parts. One was a hinge
loss variant for the foreground-background classification. The other was a
max-margin contrastive loss to enforce all RoIs to satisfy max-margin category
separation and semantic space clustering as possible. Bansal et al. [77]
defined a margin loss to enforce a constraint that the matching score of a
proposal with its true category should be higher than that with other
categories. Li et al. [79] assumed that novel classes had implicit relation
with base classes and prototypes of novel classes could be embedded into
margins between prototypes of base classes. However, large inter-class
distance provided the safe decision boundary for the classification while
large margin between prototypes of base class makes it hard for novel classes
to find appropriate class prototypes. Thus, Li et al. [79] proposed a max-
margin loss which was formulated as a sum of intra-class distance of all
classes over a sum of inter-category distance of all classes to adaptively
adjust class-margin. Zhang et al. [154] improved traditional contrastive loss
from two aspects. First, learnable margins were designed to adaptively adjust
inter-class distance. Especially, these margins were initialized by inter-
class semantic similarity. Second, instead of hard sampling in the basic
contrastive loss, focal loss was imposed to adaptively adjust contributions of
various kinds of samples to the gradient. Li et al. [146] also proposed a
transformation invariant principle to learn a robust embedding network which
should produce consistent category prototypes between an image and an
associated transformed image, except for CE loss.

  * •

Fusion Node. As described above, class prototypes encoded class-specific
information, e.g., the shape of objects, which were helpful for the detectors
to accurately localize and classify objects of the associated category. Thus,
it was important to determine where to place fusion nodes to aggregate query
features and class prototypes in one-/two-stage detectors. Tab. III presented
common fusion nodes and affected modules of one- and two-stage detectors.

For a two-stage detector, many works have explored six fusion nodes, in Fig.
12(a). Several works revised the multi-class classifier as the meta-classifier
which combined RoI features and category prototypes of a given category as
inputs and output a matching score, i.e., only the 5th fusion node [77, 152,
160, 153]. However, it ignored the implicit shape and position information in
category prototypes, which had a positive effect on the bounding box
regressor. Thus, the 4th or 6th fusion node was widely added to provide
category-sensitive information for the R-CNN head [29, 79, 146, 141]. Due to
differences between the classification and localization, Liu et al. [29]
explicitly decomposed the 4th fusion node into the 5th and 6th feature node
which provided task-specific information respectively for two sub-tasks. In
addition, to get more high-quality proposals from RPN, many works inserted the
1st or 2nd fusion node to provide class-sensitive information for RPN (meta-
RPN) [80, 147, 149]. The 1st fusion node could affect the whole Faster R-CNN
head while the 2nd fusion node only served RPN [148, 156]. For one-stage
detectors, almost all methods chose the 1st fusion node to make better results
under the help of category-sensitive information (Fig. 12(b)) [159, 79, 41].
More details about combinations of fusion nodes were shown in Tab. X.

TABLE III: Modules Affected by Fusion Node. Note that a-i is the i-th fusion node of two-stage detectors while b-i is the i-th fusion node of one-stage detectors, as in Fig. 12. Fusion Nodes | Affected Modules
---|---
a-1 | RPN & Cls & Loc
a-2 | RPN
a-3 | Cls & Loc
a-4
a-5 | Cls
a-6 | Loc
b-1 | Cls & Loc
b-2 | Cls
b-3 | Loc
Figure 12: Fusion Nodes of two parallel branches of support and query in (a)
two- stage detectors/(b) one-stage detectors. Note that ⓘ represents the i-th
fusion node which is actually a feature aggregator to aggregate query features
and category prototypes.

  * •

Training/Testing Process. The overall training/testing process is illustrated
in Algorithm 1, i.e., meta-training/finetuning/testing. Almost all metric-
based methods are usually pretrained in existing benchmarks [33, 32] to learn
basic notions (e.g., low-level visual features) which could be applied into
our $N$-way $K$-shot problem, instead of training from scratch. For a fair
comparison, it is essential to remove all categories in ImageNet overlapped
with unseen categories in target benchmarks. In meta-training stage, we first
construct a set of tasks/episodes built upon $D_{base}$ to let $M^{base}$
learn how to leverage limited reliable data to match all instances in a query
set. In other words, a base learner is actually a task-agnostic learner which
could be instantiated with task-specific notions from a given support set to
solve a novel task. Thus, high-quality support data is crucial to learn a
robust $M^{base}$. Tab. IV presents several solutions to pair a relatively
appropriate support for a query image when plenty of small/hard objects are in
$D_{base}$. Especially, meta-finetuning is an optional step in a metric-based
method. In most cases, meta-finetuning could further improve the learner’s
performance in meta-testing stage. However, it is unrealistic to directly
leverage $D_{base}\cup D_{novel}$ to finetune $M^{finetune}$ owing to the
extreme class imbalance between $D_{base}$ and $D_{novel}$. In general, we
need to construct a balanced subset which is randomly sampled from $D_{base}$
($K_{b}$ exemplars per category). In most cases, $K_{b}$ is equal to $K$
($N$-way $K$-shot). However, considering a large number of training exemplars
for base classes, $K_{b}$ can be increased properly to get more robust
prototypes for base classes to speed up the training process, e.g., $K_{b}=3K$
[141]. In meta-testing, we could preprocess $D_{novel}$ to get class-sensitive
vectors for each novel class as task-specific parameters for $M^{finetune}$
instead of temporarily extracting for all tasks which is time-consuming. Due
to low-shot scenarios, base/novel-category splitting settings have a great
effect on final results. Thus, two main ways are proposed to evaluate
performance of $M^{finetune}$ more accurately, in Tab. V.

TABLE IV: Solutions to Construct a More Suitable Episode from A Difficult Benchmark. Solution | Explanation
---|---
Hsieh et al. [142] |  Take a pretrained Mask R-CNN to evaluate the difficulty of a ground-truth box and only ground-truth boxes with high-quality proposals would be cropped out as support patches.
Fan et al. [80] |  Remove all images with a box whose size is less than $32\times 32$ in the training stage.
Li et al.[149] |  Randomly sample $m$ support patches and only choose a support patch which has the most similarity with the query images.
Liu et al. [29] |  Sample more images to build a larger support set for each task in $\mathcal{T}^{train}$ ($K=200$).
TABLE V: Details of Two Main Evaluation Settings. Method | Detail
---|---
Karlinsky et al. [160] |  Randomly sample 500 episodes ($n$-way $k$-shot tasks) where each episode consists of a support set with $k$ instance per category ($|S|=nk$) and a query set $Q$ with 10 images per category ($|Q|=10n$). Thus, we have at least 10 instances of each category in meta-testing stage. ($n$ is usually assigned by 5.)
Fan et al. [80] |  (1) Randomly sample several base and novel class splitting settings;
(2) Construct a list of tasks $\mathcal{T}^{test}$ built upon
$(D_{novel},D_{novel}^{\prime})$ for each splitting setting to evaluate
performance;
(3) Report average performance for each splitting setting.
TABLE VI: Summarization of details of representative methods for two other settings of support branches. Group | Method | Explanation
---|---|---
Learnable Visual Prototype (LVP) | Yang et al. [78] |  Meta-Training: (1) Assign $K$ learnable positive & negative prototypes per base category; (2) Randomly initialize all prototypes; (3) Leverage $D_{base}$ to create $n$-way $k$-shot episodes (sample $n$ support categories instead of using all categories for computational efficiency) to learn all trainable parameters.
Meta-Testing: (1) Initialize task-specific parameters: (a) Novel-Category
Prototypes: Use weight imprinting to initialize novel-category prototypes; (b)
Background-Category Prototypes: Apply spectral clustering on embedding vectors
of $M$ hard negative prototypes to generate $K$ negative prototypes
respectively for each novel category.; (2) Evaluate NP-RepMet on
$D_{novel}^{\prime}$.
Semantic Prototype (SP) | Bansal et al. [77] |  Meta-Training: (1) Prepare Training Data: (a) Exploit EB [161] to generate raw proposals; (b) Sample both positive and negative proposals as training data; (c) Assign base class labels to positive proposals; (d) Randomly sample a list of negative classes that have no overlap with these base classes and assign these negative class to negative proposals; (2) Take a word vector as prototypes for each base and negative class; (3) Exploit LAB to learn a meta detector.
Meta-Testing: (1) Use a word vector as the category prototype for each novel
category; (2) Evaluate ZSD on $D_{novel}^{\prime}$.
Algorithm 1 Meta-Training/Finetuning/Testing of A Metric-based Method

1:

2:A pre-defined model, $M=\\{M^{train},M^{finetune}\\}$;

3:A large-scale benchmark for base classes, $D_{base}$;

4:A small-scale training benchmark for novel classes (N samples per class),
$D_{novel}$;

5:A testing benchmark for novel classes, $D^{\prime}_{novel}$; //
$D^{\prime}_{novel}\cap D_{novel}=\emptyset$

Meta-Training

1:Initialize all parameters of $M^{train}$;

2:for $i\leftarrow 0,1,...,e_{0}-1$ do // $e_{0}$: the number of epochs in
meta-training;

3: Construct a set of tasks/episodes $\mathcal{T}^{train}$ built upon
$D_{base}$;

4: for $T$ in $\mathcal{T}^{train}$ do

5: $l^{train}\leftarrow Loss^{train}(M^{train},T)$

6: Update all trainable parameters $\theta^{train}$ of $M^{train}$ by
backprop;

7: end for

8:end for

Meta-Finetuning

1:Initialize all parameters of $M^{finetune}$; // shared parameters copied
from $M^{train}$;

2:for $i\leftarrow 0,1,...,e_{1}-1$ do // $e_{1}$: the number of epochs in
meta-finetuning;

3: Construct a set of tasks $\mathcal{T}^{finetune}$ built upon $D_{base}\cup
D_{novel}$;

4: for $T$ in $\mathcal{T}^{finetune}$ do

5: $l^{finetune}\leftarrow Loss^{finetune}(M^{finetune},T)$

6: Update all trainable parameters $\theta^{finetune}$ of $M^{finetune}$ by
backprop;

7: end for

8:end for

Meta-Testing

1:Construct a set of tasks $\mathcal{T}^{test}$ built upon $D_{novel}\cup
D^{\prime}_{novel}$; // $(S_{i},Q_{i})=T_{i}\in\mathcal{T}\&S_{i}\subset
D_{novel}\&Q_{i}\subset D^{\prime}_{novel}\&Q_{1}\cup Q_{2}\cup...=D^{test}$

2:$ans\leftarrow\\{\\}$

3:for $T$ in $\mathcal{T}^{test}$ do

4: $bboxes=M^{finetune}(T)$

5: $ans\leftarrow ans\cup\\{(Q,bboxes)\\}$

6:end for

7:Take $ans$ to evaluate performance of $M^{finetune}$ by a metric, e.g.,
mAP50.

  * •

Other Settings of Support Branch. Earlier, we mainly discussed a kind of
method that employed a list of support images fed to a customed backbone for
category prototypes named Generative Visual Prototype (GVP). Currently, two
other settings of support branches have been proposed to get rid of sampling a
set of support images per category for all episodes, i.e., LVP and SP, as
illustrated in Tab. VI. To mitigate noise in a small support set, LVP aims to
exploit a list of learnable kernels to automatically acquire class prototypes
for each class, without dependency on a specific object. Several works [77]
explored to use of semantic prototypes (SPs) learned from a large-scale corpus
to provide reliable task-specific parameters for these detectors.

_2\. Optimization-based Methods._ Optimization-based methods assume that
generic notions (i.e., meta parameters and a meta optimizer) could be learned
from the base datasets to provide a suitable gradient guidance or a uniformly
optimal initial weight for quick adaption to a new task. However, unlike FSC,
it is difficult for a meta-optimizer or meta-parameters to adapt to a wider
parameter space or to balance two sub-tasks of object detection. Therefore,
optimization-based learning has very few applications in LS-FSOD.

  * •

Meta RetinaNet, proposed by Li et al. [162], was an optimization-based
detector which took a MAML variant [20] to learn meta-parameters for a new
task. To reduce too large parameter space, kernels $k$ of convolution layers
in RetinaNet was reformulated as: $k^{\prime}=k\odot w$, where $w$ were
learnable coefficient vectors initialized by ones and $k$ were constant
kernels initialized by associated kernels pretrained on $D_{base}$. In the
meta-training phase, balanced loss (BL) was designed to replace simple
summation in original MAML to adaptively down-weight easy tasks and focus on
hard tasks to update meta parameters (i.e., coefficient vectors and parameters
of the last classification layer).

_3\. Model-based Methods._ For model-based methods, the key is to design a
model and associated learning strategies which quickly adapts for a new
episode. Most model-based methods mainly relied on RNNs [163] with memory to
utilize training samples of an episode to predict task-specific parameters for
such an episode [164, 102]. Parameters could be further divided into fast/slow
(task-specific/agnostic) parameters which were combined to make more stable
predictions. However, like optimization-based learning, model-based learning
was barely noticed in FSOD.

  * •

MetaDet. Inspired by a category-agnostic transformation, Wang et al. [165]
first explored a paradigm for a meta generator $G$: $w^{\ast}=G(w)$, where $w$
and $w^{\ast}$ were task-specific parameters learnt respectively from an
episode and a large-scale dataset. In MetaDet, only R-CNN head was defined as
task-specific modules. Especially, its training consisted of three stages.

1\. Pre-Training Phase: Take $D_{base}$ to learn a large-sample detector
$D(I;\theta^{\ast})$ in a standard way, where $\theta^{\ast}$ is a set of
parameters of the detector.

2\. Meta-Training Phase: First, learn a $n$-way $k$-shot detector
$D(I;\theta^{\ast}\cup w_{det}^{c_{base}}\backslash w_{det}^{c_{base},\ast})$
where $w_{det}^{c_{base},\ast}$ and $w_{det}^{c_{base}}$ represent category-
specific parameters learned in $D_{base}$ or a $n$-way $k$-shot episode in the
top layer of Faster R-CNN. Then, add consistency loss on the pair
$(w_{det}^{c_{base},\ast},w_{det}^{c_{base}})$ of each episode to learn a meta
generator G.

3\. Meta-Testing Phase: Fist, learn a $n$-way $k$-shot detector
$D(I;\theta^{\ast}\cup w_{det}^{c_{novel}}\backslash w_{det}^{c_{base},\ast})$
where $w_{det}^{c_{novel}}$ denote category-specific parameters learned in
$D_{novel}$. Then, feed $w_{det}^{c_{novel}}$ to G for a more robust version
$w_{det}^{c_{novel},\ast}$. Finally, fine-tune $D(I;\theta^{\ast}\cup
w_{det}^{c_{novel},\ast}\backslash w_{det}^{c_{base},\ast})$.

However, MetaDet mainly relied on a principle that base categories shared
similar distribution with novel categories, which greatly limited its scope of
applications.

Transfer-Learning Methods. Compared with meta-learning algorithms, transfer-
learning methods usually have two phases of training, i.e., the pre-training
and fine-tuning stage. In the pre-training stage, a large-scale dataset
$D_{base}$ is employed to train a base detector for general notions under the
official setting. There are two steps during the fine-tuning stage: (1)
Initialization. The novel learner inherits general notations/parameters from
the base learner while the rest parameters are initialized randomly or by a
weight imprinting technique. (2) Fine-tuning. A small-scale dataset
$D_{base}^{\prime}\cup D_{novel}$ is constructed to fine-tune the novel
learner where $D_{base}^{\prime}$ is a balanced subset of $D_{base}$ ($K$
shots per base class). To sample a more suitable subset $D_{base}^{\prime}$,
Li et al. [166] proposed a clustering-based exemplar selection algorithm which
first calculated intra-category mean features/prototypes for each image in
$D_{base}$, then applied the k-means algorithm to generate $K$ clusters for
each base category, and finally obtained $K$ centroids (exemplars) per base
category. Like aforesaid metric-based learning, we will make a detailed review
on the fine-tuned module, regularization, classifier and loss function for
transfer-learning methods. If you want to know specific contributions, we also
present them in Tab. XI.

  * •

Fine-tuned Module. In the early stage of transfer-learning approaches [40],
the backbone and RPN were usually taken as task-agnostic modules and it was
ineffective to utilize a small-scale dataset $D_{base}^{\prime}\cup D_{novel}$
to fine-tune all parameters of a base detector. Thus, wang et al. [40]
explored a FIX_ALL mode which only adapted the final layer of both
classification and localization in the fine-tuning stage and achieved
promising results. However, as aforesaid in Section I-C, RPN showed relatively
poor performance for novel classes. Sun et al. [167] demonstrated the number
of positive proposals for novel categories was about a quarter of that for
base categories. Hence, there were not enough positive proposals to fine-tune
RPN in the fine-tuning stage. Meanwhile, it implicitly introduced the
foreground-background imbalance problem, due to low-quality proposals for
novel categories. Sun et al. [167] proposed a two-step procedure to re-balance
sampling ratio of the positive and negative proposals: (1) reuse positive
proposals with low confidence suppressed by NMS; (2) discard negative
proposals by half. Moreover, backbones were proved to have weaker response for
novel categories than that for base categories. Thus, many works explored an
extreme mode (named FIT_ALL) to obtain a de-biased backbone/FPN for better
features [157, 38, 37]. However, low density sampling easily leaded to
overfitting under the mode FIT_ALL.

Figure 13: The pipeline of two classic regularization methods. (a) Transfer-Knowledge (TK) Regularization; (b) Background-Depression (BD) Regularization. TABLE VII: The sub-modules of FAFRCNN. Sub-Module | Detail
---|---
Split Pooling |  (1) Apply shared random shifts on a set of pre-defined anchors to crop multi-scale features patches for both source and target domain.
(2) Take feature patches to generate two kinds of pairs at each scale (i.e.,
source-source and source-target pairs).
(3) Adopt generative adversarial learning to alternatively optimize the
generator (backbone) and the discriminator that classifies feature patches
whether or not belong to target domain.
Instance-level Adaptation |  (1) Sample foreground RoIs with high IoU threshold (i.e., 0.7) in source and target domain.
(2) Take RoI features to generate two kinds of pairs (i.e., source-source and
source-target pairs).
(3) Adopt generative adversarial learning to alternatively optimize the
generator (RoI feature encoding network) and the discriminator that classifies
RoI features.
  * •

Regularization. To alleviate overfitting, several studies added extra
constraints on the parameter space for a robust model. Earlier, the base
classifier was usually dropped in the fine-tuning stage [167, 40, 38].
Especially, it could be viewed as a special encoder that took RoI features $R$
as input and output a similarity score for each pair in a set
$\\{(R,C_{base}^{i})|i=1,2,\ldots,|C_{base}|\\}$. In other words, it could be
utilized to indicate shared features among novel and base classes to
regularize the models. Thus, as shown in Fig. 13(a), KL loss was applied to
enforce the model to reuse shared features for a novel task [166, 157, 39].
Moreover, Wu et al. [157] leveraged a consistent constraint (KL loss) among
probability distributions generated by a shared classifier and reserved a set
of class-agnostic prototypes learnt from a large-scale dataset $D_{base}$ for
a novel task as well for full use of generic notions from base classes.

Complex background may degenerate performance in a few-shot scenario. As shown
in Fig. 13(b), Chen et al. [39] utilized ground-truth boxes in a given image
for making a rough estimation of the background mask to explicitly depress
background features. In addition, several works leveraged a pre-trained
saliency model to generate saliency maps to reweight features for background
depression [168].

FPN [169] was widely utilized to alleviate large scale variations. FPN
employed several feature maps at different scales for region proposals.
However, due to low-density sampling, it was hard to provide enough positive
exemplars at each scale to train FPN [169] while more negative proposals were
generated by FPN. Thus, Wu et al. [38] proposed object pyramids for more
exemplars to refine FPN to distinguish positive-negative exemplars at each
scale. Instead of directly extracting objects, a small random shift was
applied on square boxes for some disturbance to learn a more robust FPN [169].
Then, we resized objects to six scales (i.e., $\\{{32}^{2},\
{64}^{2},{128}^{2},{256}^{2},{512}^{2},{800}^{2}\\}$). Next, objects were fed
into shared Faster R-CNN [9]. Finally, for objects at a special scale, only
feature maps at an associated scale were leveraged to calculate RPN and RoI
classification loss.

As described above, many works supposed that the novel classes $C_{novel}$
shared generic notions with the base classes $C_{base}$ (i.e., low domain
shift). Nevertheless, it was hard to collect a proper dataset $D_{base}$ in
some special cases. Hence, Wang et al. [37] proposed an adversarial learning
to mitigate domain shifts (Tab. VII).

  * •

Classifier. For most transfer-learning methods, error rates in classification
are much higher than that in localization. Thus, several works replaced the
softmax classifier with the cosine classifier to get rid of irrelevant
factors, e.g., various feature norms. Moreover, Yang et al. [143] designed
Context-Transformer to provide more contextual information for the novel
classifier. It first leveraged spatial pooling to implicitly extend
receptive/contextual fields of original features $F_{k}$ at each scale
$k\in\\{1,2,\ldots,K\\}$ for context features $Q_{k}$. Then, an affinity
matrix $A_{k}$ was calculated among features $F_{k}$ & a set of context
features $Q=\\{Q_{1},Q_{2},\ldots,Q_{K}\\}$ to adaptively match effective
supporting context at various scales for feature fusion.

  * •

Loss Function. Multi-class cross entropy (MCE) loss was generally used for
classification. However, MCE aimed at increasing inter-class distinction which
pay little attention to lower intra-class variations. Thus, Chen et al.[168]
utilize a variant of cosine loss to form compact category cluster. Sun et al.
[167] introduced a contrastive learning strategy which leveraged a variant of
cross entropy to introduce competition between homogeneous and heterogeneous
pairs for high inter-category distinction and low inter-category distance.

_3\. Data Augmentation Methods._ Data augmentation aims at increasing intra-
class variations to enforce the model to utilize more robust features. Li et
al. [146] proved, in few-shot scenarios, most naive data augmentation methods
could introduce large intra-class variations which had negative effect on its
performance. Therefore, Li et al. [146] proposed a TIP module to construct
contrasting pairs to make the encoder invariant to various intra-class
variations by KL loss. Likewise, Zhang et al. [144] designed a hallucination
network to transfer the intra-category variations from base categories to
novel categories. Simple horizontal flipping was proved effective for the
performance improvement due to moderate variations [149].

###  III-C Imbalanced Limited-Supervised Problem

####  III-C1 Definition

In a low-shot scenario, it is likely to collect a imbalanced dataset due to
the long-tail distributions of real world data. Objects have great
discrepancies in the occurrence frequencies (i.e., $\min{\\{|D_{novel}^{i}|,\
i\in C_{novel}\\}}\ll\max{\\{|D_{novel}^{i}|,i\in C_{novel}\\}}=k$). Moreover,
it is also sub-optimal to adopt an under- or over-sampling strategy to build a
$N$-way $K$-shot benchmark. However, such a problem has not attracted enough
attention.

####  III-C2 Solutions

So far, there were a few works [170] to present the detailed description on
this imbalanced problem within our capacity. In addition, we also propose
several candidate solutions for this problem, i.e., group-based methods and
generative methods.

Inspired by focal loss, a solution could be implemented by a balanced loss
which could adaptively re-balance gradient from all categories. Zhang et al.
[170] proposed CI loss which employed the imbalance degree in a dataset to
automatically select appropriate parameters for gradient re-balance. Zhang et
al. exploited NUDT-AOSR15 to construct both the training and testing set and
chose mAP as the evaluation metric.

Inspire by the long-tail distribution in the large-scale dataset (e.g., PASCAL
VOC07/12), we propose two kinds of solutions that could be applied to tackle
this imbalanced limited-supervised problem. First, GAN variants were
introduced to produce high-quality exemplars (e.g., images) for re-balancing
the foreground-foreground class [171, 172, 173]. Second, a classification tree
was built upon lexical or semantic relations for a coarse-to-fine strategy to
alleviate class imbalance [174], instead of directly classifying all classes.
In addition, there are some tricks to tackle this problem, such as an NMS
resampling algorithm [174] which dynamically adjusted NMS thresholds for each
category to provide enough RoIs for the class with a few training exemplars
according to the occurrence frequency of each category.

##  IV Semi-Supervised Few-Shot Object Detection

So far, most works on semi-supervised object detection (SSOD) collected about
half (or 10 percent) of data with instance-level labels, which is
fundamentally different from few-shot settings and still costs a lot.
Especially, if there exist over 1000 images in a dataset for SSOD, SSOD will
have far more labeled data than that annotated for SS-FSOD. Thus, compared
with traditional SSOD, SS-FSOD can further reduce the annotation burden and
exploit labeled data more effectively. There are two main ways for SS-FSOD,
i.e., self-training and self-supervised based solutions.

###  IV-A Problem Definition

Let $D_{novel}$ represent a small dataset with a few labeled exemplars for a
set of classes $C_{novel}$ and $D_{novel}^{-}$ represent a large-scale dataset
sampled in the target domain without target supervision. For each sample
$(I,Y)$ in $D_{novel}$, $I$ is an image ($I\in\mathbb{R}^{M\times N\times 3}$)
and $Y=\\{(b_{n},y_{n})\\}^{N}$ is a set of $N$ instances of $C_{novel}$ in
$I$, where $b_{n}\in\mathbb{R}^{4}$ is a bounding box of the n-th instance and
$y_{n}\in\\{0,1\\}^{|C_{novel}|}$ denotes an one-hot class encoding. In some
cases, we also have an extra dataset $D_{base}$ without target-domain
supervision to learn generic notions instead of training from scratch.

Let $D_{novel}^{n}$ be a set of instances of the n-th category in $D_{novel}$.
Like in limited-supervised FSOD, we restrict the maximum of the number of
instances per category in $D_{novel}$: $\max{\\{|D_{novel}^{i}|,\ i\in
C_{novel}\\}}\leq k$ ($k$ is usually no more than 30). In all, SS-FSOD tries
to extract latent notions in $D_{novel}^{-}$ (e.g., pseudo labels) to
refine/pre-train detectors for avoiding overfitting in $D_{novel}$.

###  IV-B Dataset

In SS-FSOD, it mainly uses existing benchmarks to form a semi-supervised few-
shot dataset, including PASCAL VOC07/12, MSCOCO and ImageNet-LOC. We usually
take mAP for PASCAL VOC07/12 and mAP/mAP75 for MSCOCO to evaluate the mean
detection performance. Likewise, we also use CorLoc for PASCAL VOC07/12 and
MSCOCO to evaluate the localization performance. More details are shown in
Tab. VIII.

TABLE VIII: Dataset Settings for SS-FSOD. Note that digits represent how many exemplars are labeled for each class.  Methods | Detail
---|---
MSPLD [175] | PASCAL VOC07/12 (CorLoc/mAP, 3), MSCOCO (CorLoc/mAP, 3), ImageNet-LOC (CorLoc/mAP, 3)
DETReg [176] | MSCOCO (mAP/mAP75, 10/30)
CGDP [82] | PASCAL VOC07/12 (mAP, 1/2/3/5/10), MSCOCO (mAP/mAP75, 10)
TIP [146] | PASCAL VOC07/12 (mAP, 1/2/3/5/10)

###  IV-C Solutions

####  IV-C1 Self-training Based Methods

Self-training is a classic way in semi-supervised learning, aiming to extract
proper pseudo labels as ground-truth labels for unannotated data. As
aforesaid, a classic self-training based approach tends to use all labeled
data to pre-train a teacher detector, then applies it to generate pseudo
labels for all unlabeled images, and finally samples a set of robust pseudo
labels to learn a student detector. The key for self-training based methods is
the quality of pseudo labels and it is sub-optimal to exploit a single teacher
detector for pre-generated pseudo labels or only update pseudo labels once to
train a student detector, which can limit its performance.

Figure 14: The pipeline of four classic pseudo-label generations. (a) MSPLD
[175]; (b) DETReg [176]; (c) FCOS Ensemble++ [177]; (d) CGDP [82].

  * •

MSPLD.  Dong et al. [175] proposed MSPLD that was based on self-learning and
multi-modal learning to obtain high-quality pseudo labels for robust training,
as illustrated in Fig. 14(a). It could be concluded as three strategies: (1)
_Hard Example Removing._ Dong et al. first assumed that images with over 4
pseudo boxes for a specific class or over 4 classes were likely to have no
reliable labels, which should be removed during each iteration step; (2)
_Model Ensemble._ Multiple detectors were utilized together to produce raw
labels, and NMS with confidence-based box filtering was applied on raw labels
for pseudo ground-truth labels; (3) _Training Pool._ It was defined as an
indicator whether or not an image should be used in training phase. Specially,
it would affect an image’s access threshold to training pools for a specific
model, when this model had a number of images used at the last iteration, low
discrepancies among proposals and pseudo labels and high consistency
exploiting such an image in other detectors’ training pools. In training
stage, Dong et al [175] adopted an iterative policy to update pseudo labels,
training pool and parameters of a specific model one by one, i.e., solid lines
in Fig. 14(a).

  * •

DETReg. Bar et al. [176] adopted a two-step training pipeline which
disentangled pseudo labels and ground-truth labels for training, avoiding to
bring too much noise into the fine-tuning stage, as depicted in Fig. 14(b).
Thus, pseudo labels were exploited in a pretext task to learn how to localize
objects and how to encode robust features.

In such a pretext task, it mainly relied on a hypothesis that, compared with
non-object boxes, object boxes should have less variations where DERT should
learn not only to distinguish such variations but also to cluster
automatically. To tackle it, several prediction heads were added upon DERT,
i.e., a bounding box regressor for localization, a feature encoder for SwAV
descriptor matching and a foreground-background classifier. As for pseudo
labels, Bar et al. exploited SS to produce thousands of RoIs for high recall.
In view of low precision for SS, a list of sampling strategies (i.e., Top-K,
Random-K and Importance Sampling) were presented to fetch proper RoIs as
ground-truth boxes (labeled by 1). Specially, Top-K showed uniformly optimal
results in the testing phase, due to its relatively robust pseudo labels. In
fine-tuning phase, we only saved the bounding box regressor while this binary
classifier was replaced with a multi-category classifier to fit a specific
task.

  * •

CGDP. In limited-supervised FSOD, it usually split existing benchmarks (e.g.,
Pascal VOC07/12) into a large-scale base dataset and a small-scale novel
dataset, as illustrated in Fig. 14(d). It was inevitable to contain novel/base
objects in the base/novel dataset which was bad for the few-shot learner
(equivalent to incomplete labeling). Actually, it could be treated as a semi-
supervised problem where unlabeled regions in an image should be handled
carefully. Li et al. [82] proposed a similarity-based strategy to inspect
negative proposals, which had low IoU with all ground-truth annotations, for
novel instance mining. First, weight imprinting was applied to extract a
normalized intra-class mean prototype per novel class. Then, only high-
possibility negative proposals were sampled to match various class prototypes
by cosine similarity for pseudo-label generation.

####  IV-C2 Self-Supervised Based Methods

Although pseudo labels could provide guidance information, it also brings
noise into training student detectors. Thus, it is necessary to exploit other
inherent relations among unannotated data. Apart from localizing and
classifying as accurately as possible, a qualified detector should meet
several requirements, e.g., robust feature encoding. Naturally, we could
simply exploit unlabeled images to enforce backbones to encode discriminative
features which could be applied to recover raw image signals as possible. In
the semi-supervised learning, several works have developed associated pretext
tasks, such as VAE/GAN based reconstruction, [178, 179, 117, 180]. In some
way, consistency regularization can be regarded as a supervised solution to
extract underlying information from unlabeled data.

  * •

TIP. In low-shot scenarios, data distributions appear discontinuous very
commonly, and naive data augmentation could make it worse which could lower
inter-category distance. Especially, a metric-based approach tends to use an
encoder to generate category prototypes which is crucial for category
matching. In this case, an encoder should be invariant to any kinds of
transformations. Thus, Li et al. [146] utilized unlabeled images to add a
consistent constraint for transformation invariance where features of both an
image and its corrupted image should be consistent/similar.

##  V Weakly-Supervised Few-Shot Object Detection

WSOD tends to require 10/50 percent of images with manual tags in datasets,
e.g., Pascal VOC07/12 [138] and MSCOCO [33]. Compared with instance-level
boxes, tags save unit labor cost while it fails to reduce the size of a
dataset needing to be labeled. It still brings large annotation burdens into
data preparation. Thus, an intuitive solution is to exploit a few images with
tags for each category in training phase (usually no more than 200). We
properly relax restraints on the size of dataset with image-level labels due
to its imprecise signals. Here, we will first give a more formal definition
for WS-FSOD.

###  V-A Problem Definition

As aforesaid, $D_{novel}^{+}$ denotes a small dataset with image-level labels
or object locations and $D_{novel}^{-}$ denotes an optional large unlabeled
dataset sampled in the target domain. For each sample $(I,Y)$ in
$D_{novel}^{-}$, $I$ is an image ($I\in\mathbb{R}^{M\times N\times 3}$) and
$Y=\\{y_{n}\\}^{N}$ is a subset of $C_{novel}$ which appears in $I$. In some
cases, we also have a dataset without target supervision to learn generic
notions instead of training from scratch. Let $D_{novel}^{n}$ be a set of
images which contains instances of the n-th category in $D_{novel}$. Like in
LS-FSOD, we restrict the maximum of the number of images per category in
$D_{novel}$: $\max{\\{|D_{novel}^{i}|,\ i\in C_{novel}\\}}\leq k$ (k is
usually no more than 200). In all, WS-FSOD aims at actively associating image-
level tags with all object regions to learn a robust learner which could be
competent to the fully-supervised counterpart.

###  V-B Dataset

Here, we list dataset settings of all existing WS-FSOD approaches, as shown in
Tab. IX. Existing approaches still exploit several benchmarks to build a
weakly-supervised few-shot benchmark, including KITTI, Cityscapes, VisDA-18,
MSCOCO, PASCAL VOC 07/12, ImageNet-LOC and CUB. In these excellent benchmarks,
they could be grouped into two types, i.e., intra-domain and cross-domain
benchmarks. Su et al. [181] combined two distinct benchmarks (i.e., KITTI and
Cityscapes) to construct a cross-domain benchmark. Another common practice is
to split a benchmark into two disjoint sets to form a intra-domain benchmark.

TABLE IX: Dataset Settings for WS-FSOD. Note that digits represent how many exemplars are labeled for each class.  Methods | Detail
---|---
AADA [181] | KITTI $\rightarrow$ Cityscapes (mAP, 10/20/30/50/100/200), VisDA-18 (mAP, N/S)
vMF-MIL [182] | MSCOCO (mAP, 5/10), PASCAL VOC 07 (mAP, 5/10)
StarNet [183] | ImageNet-LOC (mAP30/mAP, 1/5), CUB (mAP30/mAP, 1/5), PASCAL VOC (mAP30/mAP, 1/5)
NSOD [184] | PASCAL VOC07/12 (mAP, 20)

###  V-C Solutions

As WS-FSOD mainly attempts to exploit a few image-level labels, its approaches
need not only to tackle classic issues in object detection, such as occlusion,
deformation, domain shifts, various imbalance problems and so on, but also to
confront with imprecise and less annotations. Owing to less supervisory
signals, generic WSOD approaches cannot figure out underlying relations among
image-level tags and RoIs, where it further increases data uncertainty
instead. More specifically, low sampling density easily leads to large intra-
category variations or low inter-category distance and such loose data streams
can not only have almost no statistic laws but also hinder normal label
propagation in the learning processes of WSOD. It is crucial to exploit
weakly-labeled data more efficiently and eliminate uncertainty as much as
possible.

So far, WS-FSOD has not received enough attention in the computer vision
community. Here, it mainly includes three types, i.e., active adversarial
learning, multiple instance learning and metric-based learning.

  * •

AADA. Su et al. [181] employed adversarial learning to force feature
extractors to transform a WS-FSOD problem into a fully-supervised problem with
a large-scale dataset for full use of source-domain knowledge. If a specific
image cannot fit source domain well, detectors would apply non-matched notions
to get poor results. Thus, Su et al. [181] also applied a well-optimized
discriminator to indicate whether unlabeled data was similar to labeled data
or not. A subset of exemplars with high distinction was sampled to obtain
image-level tags to mitigate outliers which cannot fit source domain well.
What’s more, Su et al. [181] explored several sampling strategies to select
images for maximizing performance gain, i.e., importance weight, K-means
clustering, K-center, diversity, Best-versus-Second Best and random selection.

  * •

vMF-MIL. Shaban et al. [182] proposed vMF-MIL that was a probabilistic
multiple instance learning approach for WS-FSOD. It employed a two-step
strategy. It first obtained a base learner from a large-scale dataset with
instance-label boxes. To utilize source-domain notions as much as possible,
Shaban et al. [182] reformed Faster R-CNN [9] and make it fully category-
agnostic by removing its traditional multi-category classifier. During the
pseudo-label generation, it held a statistic assumption that RoI features for
each category should form a cluster and have high distinction with that for
other categories. For a specific category, it applied EM clustering on those
features for finding exact a pseudo label with highest confidence for each
image whose tags should include such a category. Then, an off-the-shelf FSOD
approach [40] could be applied to learn a novel task with pseudo boxes.

  * •

StarNet. Karlinsky [183] proposed StarNet that was also a metric-based
solution. It mainly exploited feature co-occurrences for reliable evidence to
mine all instances in query images without pseudo-label generation. It also
consisted of two important fusion nodes: (1) Voting Heatmaps. It first
measures a point-to-point similarity map by L1 norm, and calculates all
permutations as voting heatmaps for matching support and query features. (2)
Back-Projection Maps. It takes a permutation with highest matching score to
suppress low-quality evidence. Except for an image-level category constraint
on query features, it performed a consistency constraint among support and
query features, which could form a close cluster for each category as well.

  * •

NSOD. Yang et al. [184] proposed NSOD that was a metric-based MIL-style
framework which attempted to mine implicit relations among intra-category mean
prototypes and all RoIs. Especially, it could directly start from a small set
of annotated images to propagate image-level tags to instance-level boxes for
unlabeled data. It first utilized a backbone pretrained in ImageNet to extract
features for both support images and all RoIs of query images (without any
supervision). Then, for a specific category, all global features were averaged
as an intra-category mean prototype which was combined with all RoI features
to obtain pseudo labels for each query image. Next, pseudo labels were
exploited to learn a teacher which was exploited to refine pseudo labels.
Finally, two kinds of pseudo labels are averaged to form ground-truth pseudo
labels for student learning. It was inevitable to include noisy background
information into these intra-category mean prototypes, which was detrimental
to final results.

##  VI Conclusion

Few-shot detectors have obtained some key achievements to mitigate urgent need
of abundant labeled training data for classic deep-learning architectures,
playing an important role in various applications, such as wise medical [185,
186]. Therefore, we provide a comprehensive survey on few-shot learning for
object detection. To provide a detailed analysis of this few-shot issue, this
survey introduces a data-based taxonomy according to training data and
associated supervisory signals of a novel task in terms of definitions,
datasets, criteria, strengths and weaknesses for each kind of approaches.
Besides, we discuss main challenges that needed to be tackled and how these
approaches interplay and boost performance.

As aforesaid, existing few-shot detectors still need to tackle some imperative
problems (e.g., domain shifts) and have a huge performance gap with many-shot
detectors and human. Here, we will discuss future trends in this promising
domain.

Domain Transfer. In a real-world task, it is not always easy to find a
suitable dataset that could be exploited for cross-domain knowledge and have
low domain shifts with this task. Although many works attempted to employ GAN
variants to align the semantic space between the source and target domain,
task-specific notions could be non-matched with the source domain and easily
lost in this alignment process due to lack of associated supervision,
especially with large domain shifts [187, 188, 189]. As active-learning
strategies in [181], we notice that it is key to combine effectively and
efficiently incorporate cross-domain and task-specific knowledge for better
domain transfer.

Efficient Architectures. Existing solutions for few-shot object detection
still inherit the architectures designed for generic object detection [30, 9,
10, 11, 12, 31] which contain abundant learnable parameters. In these common
architectures, more data and layers (parameters) usually mean higher
performance. Thus, a novel and efficient architecture should be designed to
tackle this issue while there is no work in this direction.

Robust Feature Extractors. In general, feature extractors are viewed as a
task-agnostic component, which is typically pre-trained in a large dataset and
kept fixed in another task. As in [150], feature extractor is susceptible to
scale variations. Meanwhile, existing architectures extremely rely on the
features produced by backbones for dense proposals generation, which is
crucial to the final performance. Thus, there has been growing interest in
learning a robust feature extractor [146, 38, 39].

Mixed-Supervised Learning. In WS-FSOD, it is hard to use a few images with
image-level tags to associate underlying objects with these image-level tags.
If we provide a small training set with instance-level labels except for the
aforementioned weakly-labeled data, it only slightly increases annotation
costs while it could give a definite signal for low-shot detectors to have
more high-quality proposals when compared with weakly-supervised few-shot
detectors. However, there are limited works in the mixed-supervised few-shot
object detection [190]. According to the above analysis, we think that mixed-
supervised few-shot object detection may be a hot research direction.

Unsupervised Learning. Currently, state-of-the-art architectures have an
urgent need for abundant labeled training data to learn an excellent detector.
However, in some cases, training data is hard to be collected. Even if we cost
significantly to collect a large annotated dataset, it could still contain
nasty issues, such as foreground-foreground imbalance and data bias, which go
against the normal training process. Therefore, it is essential to study a
strategy [191, 35] to train CNNs without the need to collect and annotate a
large-scale dataset for a novel task.

Data Augmentation. Data augmentation is a considerably intuitive way to
alleviate overfitting by increasing intra-category variations. However, large
intra-category variations could relatively lower inter-category distinction,
which is interferential to explore suitable category prototypes. Zhang et al.
[144] proposed a meta strategy that moderately increased the intra-category
diversity which could be transferred from other large-scale datasets while it
cannot work with large domain shifts, since the source-domain intra-category
diversity could not fit the target domain. A better choice is to actively
measure intra-category variations and inter-category distance to provide
suitable strategies for data augmentation.

## References

  * [1] Y.-X. Liu, Y. Yang, A. Shi, P. Jigang, and L. Haowei, “Intelligent monitoring of indoor surveillance video based on deep learning,” in _Proceedings of the IEEE International Conference on Advanced Communication Technology (ICACT)_ , 2019, pp. 648–653.
  * [2] L. Liu, H. Li, and M. Gruteser, “Edge assisted real-time object detection for mobile augmented reality,” in _Proceedings of the ACM International Conference on Mobile Computing and Networking (MOBICOM)_ , 2019, pp. 1–16.
  * [3] Z. Zhong, M. Lei, D. Cao, J. Fan, and S. Li, “Class-specific object proposals re-ranking for object detection in automatic driving,” _Neurocomputing_ , vol. 242, pp. 187–194, 2017.
  * [4] P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , vol. 1, 2001, pp. I–I.
  * [5] N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , vol. 1, 2005, pp. 886–893.
  * [6] P. F. Felzenszwalb, R. B. Girshick, and D. McAllester, “Cascade object detection with deformable part models,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2010, pp. 2241–2248.
  * [7] P. Felzenszwalb, D. McAllester, and D. Ramanan, “A discriminatively trained, multiscale, deformable part model,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2008, pp. 1–8.
  * [8] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Proceedings of the Neural Information Processing Systems (NIPS)_ , vol. 25, 2012, pp. 1097–1105.
  * [9] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in _Proceedings of the Neural Information Processing Systems (NIPS)_ , vol. 28, 2015, pp. 91–99.
  * [10] J. Redmon and A. Farhadi, “Yolo9000: better, faster, stronger,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 7263–7271.
  * [11] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2016, pp. 21–37.
  * [12] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * [13] D. Zhang, J. Han, G. Cheng, and M.-H. Yang, “Weakly supervised object localization and detection: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , 2021.
  * [14] G. Koch, R. Zemel, R. Salakhutdinov _et al._ , “Siamese neural networks for one-shot image recognition,” in _Proceddings of the International Conference on Machine Learning Workshop (ICMLW)_ , vol. 2, 2015.
  * [15] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra _et al._ , “Matching networks for one shot learning,” in _Proceedings of the Neural Information Processing Systems (NIPS)_ , vol. 29, 2016, pp. 3630–3638.
  * [16] J. Snell, K. Swersky, and R. Zemel, “Prototypical networks for few-shot learning,” in _Proceedings of the International Conference on Neural Information Processing Systems (ICNIPS)_ , 2017, pp. 4080–4090.
  * [17] B. N. Oreshkin, P. Rodriguez, and A. Lacoste, “Tadam: Task dependent adaptive metric for improved few-shot learning,” _arXiv preprint arXiv:1805.10123_ , 2018.
  * [18] E. Triantafillou, R. Zemel, and R. Urtasun, “Few-shot learning through an information retrieval lens,” in _Proceedings of the International Conference on Neural Information Processing Systems (ICNIPS)_ , 2017, pp. 2252–2262.
  * [19] Y. Chen, X. Wang, Z. Liu, H. Xu, and T. Darrell, “A new meta-baseline for few-shot learning,” _arXiv preprint arXiv:2003.04390_ , 2020.
  * [20] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _Proceddings of the International Conference on Machine Learning (ICML)_ , 2017, pp. 1126–1135.
  * [21] A. Nichol, J. Achiam, and J. Schulman, “On first-order meta-learning algorithms,” _arXiv preprint arXiv:1803.02999_ , 2018.
  * [22] T. Munkhdalai and H. Yu, “Meta networks,” in _Proceddings of the International Conference on Machine Learning (ICML)_ , 2017, pp. 2554–2563.
  * [23] T. Munkhdalai, X. Yuan, S. Mehri, and A. Trischler, “Rapid adaptation with conditionally shifted neurons,” in _Proceddings of the International Conference on Machine Learning (ICML)_ , 2018, pp. 3664–3673.
  * [24] L. Bertinetto, J. F. Henriques, J. Valmadre, P. Torr, and A. Vedaldi, “Learning feed-forward one-shot learners,” in _Proceedings of the Neural Information Processing Systems (NIPS)_ , 2016, pp. 523–531.
  * [25] S. Gidaris and N. Komodakis, “Dynamic few-shot visual learning without forgetting,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 4367–4375.
  * [26] X. Wang, F. Yu, R. Wang, T. Darrell, and J. E. Gonzalez, “Tafe-net: Task-aware feature embeddings for low shot learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 1831–1840.
  * [27] W.-Y. Chen, Y.-C. Liu, Z. Kira, Y.-C. F. Wang, and J.-B. Huang, “A closer look at few-shot classification,” in _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2018.
  * [28] J. R. Uijlings, K. E. Van De Sande, T. Gevers, and A. W. Smeulders, “Selective search for object recognition,” _International Journal of Computer Vision (IJCV)_ , vol. 104, no. 2, pp. 154–171, 2013.
  * [29] L. Liu, B. Ma, Y. Zhang, X. Yi, and H. Li, “Afd-net: Adaptive fully-dual network for few-shot object detection,” _arXiv preprint arXiv:2011.14667_ , 2020.
  * [30] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2014, pp. 580–587.
  * [31] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “Yolov4: Optimal speed and accuracy of object detection,” _arXiv preprint arXiv:2004.10934_ , 2020.
  * [32] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2009, pp. 248–255.
  * [33] X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick, “Microsoft coco captions: Data collection and evaluation server,” _arXiv preprint arXiv:1504.00325_ , 2015.
  * [34] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 770–778.
  * [35] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 1492–1500.
  * [36] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [37] T. Wang, X. Zhang, L. Yuan, and J. Feng, “Few-shot adaptive faster r-cnn,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 7173–7182.
  * [38] J. Wu, S. Liu, D. Huang, and Y. Wang, “Multi-scale positive sample refinement for few-shot object detection,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2020, pp. 456–472.
  * [39] H. Chen, Y. Wang, G. Wang, and Y. Qiao, “Lstd: A low-shot transfer detector for object detection,” in _Proceedings of the AAAI conference on Artificial Intelligence (AAAI)_ , vol. 32, no. 1, 2018.
  * [40] X. Wang, T. E. Huang, T. Darrell, J. E. Gonzalez, and F. Yu, “Frustratingly simple few-shot object detection,” _arXiv preprint arXiv:2003.06957_ , 2020\.
  * [41] B. Kang, Z. Liu, X. Wang, F. Yu, J. Feng, and T. Darrell, “Few-shot object detection via feature reweighting,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019, pp. 8420–8429.
  * [42] Z.-Q. Zhao, P. Zheng, S.-t. Xu, and X. Wu, “Object detection with deep learning: A review,” _IEEE Transactions on Neural Networks and Learning Systems (TNNLS)_ , vol. 30, no. 11, pp. 3212–3232, 2019.
  * [43] K. U. Sharma and N. V. Thakur, “A review and an approach for object detection in images,” _International Journal of Computational Vision and Robotics_ , vol. 7, no. 1-2, pp. 196–237, 2017.
  * [44] A. Dhillon and G. K. Verma, “Convolutional neural network: a review of models, methodologies and applications to object detection,” _Progress in Artificial Intelligence_ , vol. 9, no. 2, pp. 85–112, 2020.
  * [45] Z. Zou, Z. Shi, Y. Guo, and J. Ye, “Object detection in 20 years: A survey,” _arXiv preprint arXiv:1905.05055_ , 2019.
  * [46] W. Zhiqiang and L. Jun, “A review of object detection based on convolutional neural network,” in _Proceedings of the Chinese Control Conference (CCC)_ , 2017, pp. 11 104–11 109.
  * [47] L. Liu, W. Ouyang, X. Wang, P. Fieguth, J. Chen, X. Liu, and M. Pietikäinen, “Deep learning for generic object detection: A survey,” _International Journal of Computer Vision (IJCV)_ , vol. 128, no. 2, pp. 261–318, 2020.
  * [48] F. Sultana, A. Sufian, and P. Dutta, “A review of object detection models based on convolutional neural network,” _Intelligent Computing: Image Processing Based Applications_ , pp. 1–16, 2020.
  * [49] B. Li, Q. Yao, and K. Wang, “A review on vision-based pedestrian detection in intelligent transportation systems,” in _Proceedings of the IEEE International Conference on Networking, Sensing and Control (ICNSC)_ , 2012, pp. 393–398.
  * [50] P. Hurney, P. Waldron, F. Morgan, E. Jones, and M. Glavin, “Review of pedestrian detection techniques in automotive far-infrared video,” _IET Intelligent Transport Systems_ , vol. 9, no. 8, pp. 824–832, 2015.
  * [51] Z. Li, K. Wang, L. Li, and F.-Y. Wang, “A review on vision-based pedestrian detection for intelligent vehicles,” in _Proceedings of the IEEE International Conference on Vehicular Electronics and Safety (ICVES)_ , 2006, pp. 57–62.
  * [52] J. S. Kulchandani and K. J. Dangarwala, “Moving object detection: Review of recent research trends,” in _Proceedings of the IEEE International Conference on Pervasive Computing (ICPC)_ , 2015, pp. 1–5.
  * [53] R. B. Hatwar, S. D. Kamble, N. V. Thakur, and S. Kakde, “A review on moving object detection and tracking methods in video,” _International Journal of Pure and Applied Mathematics_ , vol. 118, no. 16, pp. 511–526, 2018.
  * [54] B. Karasulu, “Review and evaluation of well-known methods for moving object detection and tracking in videos,” _Journal of Aeronautics and Space Technologies_ , vol. 4, no. 4, pp. 11–22, 2010.
  * [55] A. Kumar, A. Kaur, and M. Kumar, “Face detection techniques: a review,” _Artificial Intelligence Review_ , vol. 52, no. 2, pp. 927–948, 2019.
  * [56] N. Nivedha, G. Pai, and M. S. Kumari, “Recent trends in face detection algorithm,” in _Proceedings of the Recent Trends in Electrical Science & Information Technology_, 2019.
  * [57] C. Liu, S. Li, F. Chang, and Y. Wang, “Machine vision based traffic sign detection methods: Review, analyses and perspectives,” _IEEE Access_ , vol. 7, pp. 86 578–86 596, 2019.
  * [58] A. V. Deshpande, “A brief overview of traffic sign detection methods,” _International Journal of Engineering Research_ , vol. 5, no. 2, pp. 141–144, 2016.
  * [59] R. Mukhometzianov and Y. Wang, “Machine learning techniques for traffic sign detection,” _arXiv preprint arXiv:1712.04391_ , 2017.
  * [60] K. Oksuz, B. C. Cam, S. Kalkan, and E. Akbas, “Imbalance problems in object detection: A review,” _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , 2020.
  * [61] J. Chen, Q. Wu, D. Liu, and T. Xu, “Foreground-background imbalance problem in deep object detectors: A review,” in _Proceedings of the IEEE Conference on Multimedia Information Processing and Retrieval (MIPR)_ , 2020, pp. 285–290.
  * [62] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni, “Generalizing from a few examples: A survey on few-shot learning,” _ACM Computing Surveys (CSUR)_ , vol. 53, no. 3, pp. 1–34, 2020.
  * [63] S. Kadam and V. Vaidya, “Review and analysis of zero, one and few shot learning approaches,” in _Proceedings of the Intelligent Systems Design and Applications (ISDEA)_ , 2018, pp. 100–112.
  * [64] X. Li, Z. Sun, J.-H. Xue, and Z. Ma, “A concise review of recent few-shot meta-learning methods,” _arXiv preprint arXiv:2005.10953_ , 2020.
  * [65] S. Jadon, “An overview of deep learning architectures in few-shot learning domain,” _arXiv preprint arXiv:2008.06365_ , 2020.
  * [66] M. Rezaei and M. Shahidi, “Zero-shot learning and its applications from autonomous vehicles to covid-19 diagnosis: A review,” _Intelligence-Based Medicine_ , p. 100005, 2020.
  * [67] W. Yin, “Meta-learning for few-shot natural language processing: A survey,” _arXiv preprint arXiv:2007.09604_ , 2020.
  * [68] X. Li, X. Yang, Z. Ma, and J.-H. Xue, “Deep metric learning for few-shot image classification: A selective review,” _arXiv preprint arXiv:2105.08149_ , 2021\.
  * [69] R. Dubey, P. Agrawal, D. Pathak, T. L. Griffiths, and A. A. Efros, “Investigating human priors for playing video games,” _arXiv preprint arXiv:1802.10217_ , 2018.
  * [70] Y. Li, T. Wang, B. Kang, S. Tang, C. Wang, J. Li, and J. Feng, “Overcoming classifier imbalance for long-tail object detection with balanced group softmax,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 10 991–11 000.
  * [71] Z. Ge, Z. Jie, X. Huang, C. Li, and O. Yoshie, “Delving deep into the imbalance of positive proposals in two-stage object detection,” _Neurocomputing_ , vol. 425, pp. 107–116, 2021.
  * [72] T. H. Phan and K. Yamamoto, “Resolving class imbalance in object detection with weighted cross entropy losses,” _arXiv preprint arXiv:2006.01413_ , 2020\.
  * [73] S. Zhang, X. Cheng, N. Li, and Z. Wu, “Implicit jointboost for multiclass object detection under high intra-class variation,” in _Proceedings of the IEEE International Conference on Information Science and Technology (ICIST)_ , 2012, pp. 615–618.
  * [74] A. Angelova and S. Zhu, “Efficient object detection and segmentation for fine-grained recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2013, pp. 811–818.
  * [75] Z. Lv, W. Wang, Z. Xu, K. Zhang, Y. Fan, and Y. Song, “Fine-grained object detection method using attention mechanism and its application in coal-gangue detection,” _Applied Soft Computing_ , p. 107891, 2021.
  * [76] G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and S. Belongie, “The inaturalist species classification and detection dataset,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 8769–8778.
  * [77] A. Bansal, K. Sikka, G. Sharma, R. Chellappa, and A. Divakaran, “Zero-shot object detection,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 384–400.
  * [78] Y. Yang, F. Wei, M. Shi, and G. Li, “Restoring negative information in few-shot object detection,” _arXiv preprint arXiv:2010.11714_ , 2020.
  * [79] B. Li, B. Yang, C. Liu, F. Liu, R. Ji, and Q. Ye, “Beyond max-margin: Class margin equilibrium for few-shot object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 7363–7372.
  * [80] Q. Fan, W. Zhuo, C.-K. Tang, and Y.-W. Tai, “Few-shot object detection with attention-rpn and multi-relation detector,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 4013–4022.
  * [81] X. Wu, D. Sahoo, and S. Hoi, “Meta-rcnn: Meta learning for few-shot object detection,” in _Proceedings of the ACM International Conference on Multimedia (MM)_ , 2020, pp. 1679–1687.
  * [82] Y. Li, H. Zhu, Y. Cheng, W. Wang, C. S. Teo, C. Xiang, P. Vadakkepat, and T. H. Lee, “Few-shot object detection via classification refinement and distractor retreatment,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 15 395–15 403.
  * [83] Y. Ouali, C. Hudelot, and M. Tami, “An overview of deep semi-supervised learning,” _arXiv preprint arXiv:2006.05278_ , 2020.
  * [84] Z.-H. Zhou, “A brief introduction to weakly supervised learning,” _National science Review_ , vol. 5, no. 1, pp. 44–53, 2018.
  * [85] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 779–788.
  * [86] H. Law and J. Deng, “Cornernet: Detecting objects as paired keypoints,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 734–750.
  * [87] K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, and Q. Tian, “Centernet: Keypoint triplets for object detection,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019, pp. 6569–6578.
  * [88] J. Wang, K. Chen, S. Yang, C. C. Loy, and D. Lin, “Region proposal by guided anchoring,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 2965–2974.
  * [89] X. Zhou, J. Zhuo, and P. Krahenbuhl, “Bottom-up object detection by grouping extreme and center points,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 850–859.
  * [90] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 2980–2988.
  * [91] A. Shrivastava, A. Gupta, and R. Girshick, “Training region-based object detectors with online hard example mining,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 761–769.
  * [92] Z. Cai and N. Vasconcelos, “Cascade r-cnn: Delving into high quality object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 6154–6162.
  * [93] J. Pang, K. Chen, J. Shi, H. Feng, W. Ouyang, and D. Lin, “Libra r-cnn: Towards balanced learning for object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 821–830.
  * [94] J. Li, X. Liang, Y. Wei, T. Xu, J. Feng, and S. Yan, “Perceptual generative adversarial networks for small object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 1222–1230.
  * [95] M. Kisantal, Z. Wojna, J. Murawski, J. Naruniec, and K. Cho, “Augmentation for small object detection,” _arXiv preprint arXiv:1902.07296_ , 2019.
  * [96] C. Eggert, S. Brehm, A. Winschel, D. Zecha, and R. Lienhart, “A closer look: Small object detection in faster r-cnn,” in _Proceedings of the IEEE International Conference on Multimedia and Expo (ICME)_ , 2017, pp. 421–426.
  * [97] F. Ozge Unel, B. O. Ozkalayci, and C. Cigla, “The power of tiling for small object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2019, pp. 0–0.
  * [98] N. Zhang, J. Donahue, R. Girshick, and T. Darrell, “Part-based r-cnns for fine-grained category detection,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2014, pp. 834–849.
  * [99] L. Song, Y. Li, Z. Jiang, Z. Li, H. Sun, J. Sun, and N. Zheng, “Fine-grained dynamic head for object detection,” _arXiv preprint arXiv:2012.03519_ , 2020\.
  * [100] F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. Torr, and T. M. Hospedales, “Learning to compare: Relation network for few-shot learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 1199–1208.
  * [101] A. Graves, G. Wayne, and I. Danihelka, “Neural turing machines,” _arXiv preprint arXiv:1410.5401_ , 2014.
  * [102] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap, “Meta-learning with memory-augmented neural networks,” in _Proceedings of the International Conference on Machine Learning (ICML)_ , 2016, pp. 1842–1850.
  * [103] Y.-X. Wang, R. Girshick, M. Hebert, and B. Hariharan, “Low-shot learning from imaginary data,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 7278–7286.
  * [104] B. Hariharan and R. Girshick, “Low-shot visual recognition by shrinking and hallucinating features,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 3018–3027.
  * [105] D.-H. Lee _et al._ , “Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks,” in _Proceddings of the International Conference on Machine Learning Workshop (ICMLW)_ , vol. 3, no. 2, 2013, p. 896.
  * [106] K. Sohn, Z. Zhang, C.-L. Li, H. Zhang, C.-Y. Lee, and T. Pfister, “A simple semi-supervised learning framework for object detection,” _arXiv preprint arXiv:2005.04757_ , 2020.
  * [107] Q. Zhou, C. Yu, Z. Wang, Q. Qian, and H. Li, “Instant-teaching: An end-to-end semi-supervised object detection framework,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 4081–4090.
  * [108] Z.-H. Zhou, M. Li _et al._ , “Semi-supervised regression with co-training.” in _Proceedings of the International Joint Conferences on Artificial Intelligence (IJCAI)_ , vol. 5, 2005, pp. 908–913.
  * [109] H. Valizadegan, R. Jin, and A. K. Jain, “Semi-supervised boosting for multi-class classification,” in _Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD)_ , 2008, pp. 522–537.
  * [110] Z. Jiang and S. Zhang, “A semi-supervised ensemble learning algorithm,” in _Proceedings of the IEEE International Conference on Cloud Computing and Intelligence Systems (CCIS)_ , vol. 2, 2012, pp. 913–918.
  * [111] X. Zhu, _Semi-supervised learning with graphs_. Carnegie Mellon University (CMU), 2005.
  * [112] A. Subramanya and P. P. Talukdar, “Graph-based semi-supervised learning,” _Synthesis Lectures on Artificial Intelligence and Machine Learning_ , vol. 8, no. 4, pp. 1–125, 2014.
  * [113] Y. Grandvalet, Y. Bengio _et al._ , “Semi-supervised learning by entropy minimization.” _CAP_ , vol. 367, pp. 281–296, 2005.
  * [114] K. Bennett, A. Demiriz _et al._ , “Semi-supervised support vector machines,” in _Proceedings of the Neural Information Processing Systems (NIPS)_ , 1999, pp. 368–374.
  * [115] J. Jeong, S. Lee, J. Kim, and N. Kwak, “Consistency-based semi-supervised learning for object detection,” in _Proceedings of the Neural Information Processing Systems (NIPS)_ , vol. 32, 2019, pp. 10 759–10 768.
  * [116] A. Rasmus, H. Valpola, M. Honkala, M. Berglund, and T. Raiko, “Semi-supervised learning with ladder networks,” _arXiv preprint arXiv:1507.02672_ , 2015\.
  * [117] A. Odena, “Semi-supervised learning with generative adversarial networks,” _arXiv preprint arXiv:1606.01583_ , 2016.
  * [118] S. Laine and T. Aila, “Temporal ensembling for semi-supervised learning,” _arXiv preprint arXiv:1610.02242_ , 2016.
  * [119] K. Yang, D. Li, and Y. Dou, “Towards precise end-to-end weakly supervised object detection network,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019, pp. 8372–8381.
  * [120] P. Tang, X. Wang, S. Bai, W. Shen, X. Bai, W. Liu, and A. Yuille, “Pcl: Proposal cluster learning for weakly supervised object detection,” _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , vol. 42, no. 1, pp. 176–191, 2018.
  * [121] H. Bilen, M. Pedersoli, and T. Tuytelaars, “Weakly supervised object detection with posterior regularization,” _Proceedings of the British Machine Vision Conference (BMVC)_ , pp. 1–12, 2014.
  * [122] G. Cheng, J. Yang, D. Gao, L. Guo, and J. Han, “High-quality proposals for weakly supervised object detection,” _IEEE Transactions on Image Processing (TIP)_ , vol. 29, pp. 5794–5804, 2020.
  * [123] H. Bilen and A. Vedaldi, “Weakly supervised deep detection networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 2846–2854.
  * [124] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Learning deep features for discriminative localization,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 2921–2929.
  * [125] P. Shamsolmoali, J. Chanussot, M. Zareapoor, H. Zhou, and J. Yang, “Multipatch feature pyramid network for weakly supervised object detection in optical remote sensing images,” _IEEE Transactions on Geoscience and Remote Sensing (TGRS)_ , 2021.