proofatend+=

######  .

∎

# Few-shot Weakly-Supervised Object Detection via Directional Statistics

Amirreza Shaban
University of Washington
Equal Contribution. Contact at<EMAIL_ADDRESS>or<EMAIL_ADDRESS>Amir
Rahimi∗
ANU & ACRV
Thalaiyasingam Ajanthan
ANU & ACRV
Byron Boots
University of Washington
Richard Hartley
ANU & ACRV

###### Abstract

Detecting novel objects from few examples has become an emerging topic in
computer vision recently. However, these methods need fully annotated training
images to learn new object categories which limits their applicability in real
world scenarios such as field robotics. In this work, we propose a
probabilistic multiple instance learning approach for few-shot Common Object
Localization (COL) and few-shot Weakly Supervised Object Detection (WSOD). In
these tasks, only image-level labels, which are much cheaper to acquire, are
available. We find that operating on features extracted from the last layer of
a pre-trained Faster-RCNN is more effective compared to previous episodic
learning based few-shot COL methods. Our model simultaneously learns the
distribution of the novel objects and localizes them via expectation-
maximization steps. As a probabilistic model, we employ von Mises-Fisher (vMF)
distribution which captures the semantic information better than Gaussian
distribution when applied to the pre-trained embedding space. When the novel
objects are localized, we utilize them to learn a linear appearance model to
detect novel classes in new images. Our extensive experiments show that the
proposed method, despite being simple, outperforms strong baselines in few-
shot COL and WSOD, as well as large-scale WSOD tasks.

##  1 Introduction

In this paper we address the problem of $N$-way, $K$-shot Weakly Supervised
Object Detection (WSOD), and develop a method with the following capabilities.
Suppose that we are given a set of $N\times K$ previously unseen images
consisting of $K$ images of objects from each of $N$ previously unknown
(novel) classes. These will be called the “support images”. Each training
image has image-level labels, indicating which classes are present in the
image. Typically, the number of novel classes $N$ may be up to $20$ and the
number of training images $K$ from each class may be $5$ or $10$, but there is
no requirement that the number of images in each novel class are equal.

Given this small number of support images, the algorithm learns to find
instances of (possibly multiple) objects from any of the novel classes in a
query image, and will put a bounding box around all such positive instances.

Figure 1:  Few-shot WSOD problem. Similar to the few-shot classification
problem, the input training set (support set) only contains image labels (car,
cow and person are novel classes in this example). Model learns to detect the
target objects in the test (query) image. Few-shot WSOD bridges few-shot
classification and object detection by learning to detect the novel objects in
the query images while only needs image-level labels for the support images.
Figure 2:  The feature maps are shown as the shape of their tensors. $Q$ and
$M$ denote the total number of query and support images, respectively. A
Faster-RCNN (shown in Fig. 3) trained on the base dataset is used to extract
$P$ proposals from each input image. The embeddings are grouped based on their
corresponding image-level labels and each group is fed into a separate Common
Object Localization (COL) module. The COL module (shown in detail on the
right) receives proposal embeddings of images of a class ($M_{c}$ is the
number of images within class $c$) and simultaneously estimates the common
class mean ${\bm{\theta}}_{c}$ and bounding-box level labels $\mathbf{w}_{c}$
via EM steps. Object Detection module uses the top predictions in
$\mathbf{w}_{c}$ to learn an appearance model for each novel class in the
support set. This appearance model is then tested on the query proposals to
detect novel objects in the query set.

As summarized in Fig. 1, our system provides a flexible object detection
algorithm that requires a very small training set of images of novel objects,
where each image is annotated only with image-level labels. As such, it is
suitable for classifying and detecting objects given only images provided, for
instance, by an internet image search for images of novel classes. In
comparison to supervised few-shot object detection approaches, e.g., [43, 42,
41, 26], where manually labeled bounding box annotations are required, this is
a more realistic setting to learn an object detector on novel examples with
applications like robotics [16] or video object segmentation [22].

We first use a Faster-RCNN network to produce bounding box proposals of
possible regions containing an object with their associated feature vectors.
This network is pre-trained on a fully annotated “base dataset”, with bounding
boxes of objects of various classes, not containing any of the novel training
classes. Then, the novel objects are learned in a two steps process: 1) A
common object localization (COL) module is used first to localize the novel
objects in the support images. 2) An object detection module to learn the
novel object classes found in the COL step. We explain each of these modules
in more details below.

COL module. To localize the novel objects in the support images, COL module
finds the common object in the $K$ images provided for each of the $N$ novel
classes. The COL module is run separately on the images from each of the $N$
novel classes. For a novel class $c$, the input to the COL module is the set
of normalized feature vectors (corresponding to bounding boxes provided by the
proposal network) from the images of class $c$. An EM algorithm on these
feature vectors determines the direction parameter of von Mises-Fisher (vMF)
distribution on the unit sphere most likely to favour a common object
representative from each image. The closest feature from each image identifies
the bounding box containing the common object. A distribution for a background
class is also trained, using the base dataset to steer the COL away from
selecting background objects.

Detection module.  Once the novel objects are found by the COL module, their
bounding boxes are used to train a box classifier for each novel class $c$.
The classification is done by a 2-class (contains / does not contain the
object) classification algorithm, once again working on normalized feature
vectors. These bounding boxes (and their associated features) are labeled
either as positive or negative for containing the object of the novel class,
or not. The positive bounding boxes are those that are determined by the COL
module to contain the common object from class $c$, and the negative samples
are chosen from proposals selected from the images of the other classes. Thus,
the classifier for class $c$ is trained to distinguish features corresponding
to bounding boxes containing an object of class $c$ from those that do not.

Finally, at test time, a query image is passed through the proposal network,
to provide bounding boxes (and their features). These bounding boxes are then
evaluated by each of the classifiers, to determine whether they contain a
novel class object, or belong to the background.

The proposed method is summarized in Fig. 2. We make several contributions and
important observations: 1) We propose a simple yet powerful COL that uses
directional statistics for modeling. Our COL module can be built on top of
off-the-shelf pre-trained Faster-RCNN models without extra parameters. We
observe that using feature vector directions in our probabilistic model, we
can better capture the semantic information compared to Gaussian distribution.
To our knowledge, employing directional statistics for multiple instance
learning is new. 2) We employ a detection module to extend COL to few-shot
WSOD. To the best of our knowledge few-shot WSOD not been studied in the
literature before. 3) Despite its simplicity, our method outperforms
sophisticated few-shot COL algorithms [31, 13] on PASCAL VOC [7], MS COCO
[20], and ILSVRC detection [4] benchmarks. In WSOD, our method outperforms
recent knowledge transfer based approaches [37, 12] in both few-shot and
large-scale settings.

##  2 Details of Methodology

###  2.1 Few-Shot WSOD and COL Tasks Definition

The goal in few-shot WSOD is to learn a model that, given support images
$\mathcal{D}_{\rm train}$ containing a set of novel classes $\mathcal{L}$,
detects instances of the novel classes in query images $\mathcal{D}_{\rm
test}$. The support set consists of image-label pairs
$(\mathbf{I},\mathbf{y})\in\mathcal{D}_{\rm train}$ where image-level label
$\mathbf{y}\subseteq\mathcal{L}$ is a subset of classes present in the image
$\mathbf{I}$111In contrast to few-shot image classification, few-shot WSOD
images can have multiple labels.. The support set is typically a small
$K$-shot, $N$-way set sampled from a large dataset $\mathcal{D}_{\rm novel}$
with variety of novel classes $\mathcal{C}_{\rm novel}$. The sampling process
for few-shot WSOD follows rules that are similar to few-shot classification
problems [17, 35]. A set of $N$ classes $\mathcal{L}\subset\mathcal{C}_{\rm
novel}$, called target classes, are first sampled. Then, for each target class
$c\in\mathcal{L}$, $K$ images containing at least an instance of class $c$ are
sampled _without replacement_ to create the support set $\mathcal{D}_{\rm
train}$. The query set $\mathcal{D}_{\rm test}$ is sampled similarly, but
unlike the support set, query labels also contain bounding box annotations in
addition to the image-level labels as the goal is to detect target objects in
the query data. These bounding box annotations are only used for evaluation.
Few-shot COL [31, 13] is a special case of few-shot WSOD where there is only
one target class in the support set, i.e., $N=1$.

For pre-training, the algorithm has access to a large dataset
$\mathcal{D}_{\rm base}$ with a set of base classes $\mathcal{C}_{\rm base}$.
Typically, there is no image in common between the base and novel datasets.
Moreover, the set of base classes is disjoint from the set of novel classes
used in evaluation, i.e., $\mathcal{C}_{\rm base}\cap\mathcal{C}_{\rm
novel}=\varnothing$.

Figure 3:  Feature Extraction. We use a pre-trained Faster-RCNN on the base
dataset to extract $P$ proposals from each input image. A $\ell_{2}$
normalization layer is employed to project all the features onto the unit
hypersphere.

###  2.2 Pre-training and Feature Extraction

We pre-train a Faster-RCNN [29] on the base dataset for bounding box and
feature extraction. The overall architecture is shown in Fig. 3. To train the
network, we use the original bounding box labels within the base dataset to
define the Region Proposal Network (RPN) and second-stage losses of the
Faster-RCNN. We adapt class-agnostic bounding box regression model in the
second-stage to get one bounding box per feature proposal regardless of the
number of base classes. Once trained, we use the trained Faster-RCNN to
extract $P$ bounding box proposals $B\in\mathbb{R}^{P\times 4}$ and their
corresponding d-dimensional features $F\in\mathbb{R}^{P\times d}$ from each
input image $\mathbf{I}$. We also apply an $\ell_{2}$ normalization layer to
project all the features to the unit hypersphere. As discussed later, the
normalization step is important as our model uses cosine similarity measure
for better generalization.

We need the feature extracted from the full image bounding box to initialize
our COL method. This is accomplished by manually appending the full image
bounding box to the box proposals of the RPN, thus its feature is extracted by
the Faster-RCNN second-stage feature extractor. We denote the first proposal
in $B$ and $F$, the complete image bounding box and its feature, respectively.

###  2.3 Statistical Model Assumptions

Since the support set $\mathcal{D}_{\rm train}$ provided to the learner is
limited, it is crucial to employ proper learning biases in the model to combat
overfitting. Inspired by the success of prototypical networks [35], we design
our model based on the assumption that features of each object class form a
single cluster in the embedding space. We propose to use directional data
based on the von Mises-Fisher (vMF) distribution, which arises naturally when
each cluster is distributed on the unit hypersphere. Formally, we assume
features of each foreground class follow vMF distribution with mean direction
${\bm{\theta}}$ and positive concentration parameter $\kappa$

$p^{+}_{{\bm{\theta}}}(\mathbf{x})=\frac{1}{Z}\exp\left(\kappa{\bm{\theta}}^{\top}\mathbf{x}\right)\text{
s.t. }\|{\bm{\theta}}\|=1~{},$ (1)

where $Z$ is the normalizing constant and input $\mathbf{x}\in\mathbb{R}^{d}$
is a unit vector, i.e., $\|\mathbf{x}\|=1$ or equivalently
$\mathbf{x}\in\mathbb{S}^{d-1}$. We assume the concentration hyperparameter is
constant and the same for all novel classes. In Section 2.4, we propose an
expectation maximization algorithm to estimate the mean direction of a novel
class from the support set.

We could also use Gaussian distribution for our model whose effect was
analogous to using Euclidean distance. We empirically show that vMF provides
superior results to Gaussian distribution when using pre-trained features. Our
results support related works in supervised few-shot learning [27, 9] where
using cosine similarity outperforms euclidean distance measure. The underlying
reason for this is well-studied by Wang _et al_. [40]; Softmax loss used in
the pre-training tends to create a ‘radial’ feature distribution where
direction specifies the semantic classes while magnitude decides the
classification confidence.

Additionally, a background class distribution is learned to steer the learner
toward objects and away from background proposals. Let

$p_{{\bm{\omega}}}^{-}(\mathbf{x})=\frac{1}{U}u^{-}_{\bm{\omega}}(\mathbf{x})~{},$
(2)

represent the background class distribution where $U$ is a constant
normalizer. As the base dataset provides a reach set of examples for learning
the background model, the background distribution is learned from the base
dataset and remains fixed when evaluating on WSOD examples sampled from the
novel data.

To learn the background distribution, we collect a set of background proposals
with low intersection-over-union (IoU) score ($<0.3$) to the objects within
the base dataset and use maximum likelihood estimation in [3] to find the
parameters of vMF distribution for the background data.

###  2.4 COL

We first explain the method for few-shot COL with a single novel common object
within the support set and employ it for few-shot WSOD later. As shown in Fig.
4, COL module’s goal is to find the common object representation across a set
of images with one novel object in common. Let
$\mathcal{F}=\\{F_{i}\\}_{i=1}^{M}$ denote the Faster-RCNN feature proposals
extracted from the input images where $M$ is number of images. Each proposal
has a (latent) binary label that indicates whether the proposal tightly
encloses the common object. Namely, $\mathbf{z}_{ij}\in\\{0,1\\}$ is the label
of the $j$-th proposal in the $i$-th image.

Figure 4:  Example of COL across three images. Data points on the unit sphere
represents feature proposals extracted from all input images. Features
extracted from each image are colored the same (shown in white, gray, black
colors). Background score function $u^{-}_{{\bm{\omega}}}(\mathbf{x})$ is also
shown on unit sphere where blue and red indicated highest and lowest
background scores, respectively. COL unit goal is to find a common object
representation ${\bm{\theta}}$ (shown by green arrow) which is close to at
least a white, gray, and black data point. Note that the area marked with
dashed circle is also close to proposals from all three images but direction
${\bm{\theta}}$ is favored as it has a lower background score.

Starting from an initial guess for the direction parameter ${\bm{\theta}}$ of
the novel common class, the algorithm alternately refines the mean direction
and label estimations in an expectation-maximization optimization framework.
We present the update rules here and defer the derivations that bring
interesting insights into the proposed method to Appendix A. In the E-step,
the algorithm uses the current direction to estimate soft labels $\mathbf{w}$,
where $\mathbf{w}_{ik}\in[0,1]$ is the soft label for the $k$-th proposal
within the $i$-th image, via attention over the proposals within each image

$\mathbf{w}_{ik}=\frac{p^{+}_{{\bm{\theta}}}(F_{ik})/p_{{\bm{\omega}}}^{-}(F_{ik})}{\sum_{j=1}^{P}p^{+}_{{\bm{\theta}}}(F_{ij})/p_{{\bm{\omega}}}^{-}(F_{ij})}$
(3)

where $F_{ij}\in\mathbb{S}^{d-1}$ is the feature of the $j$-th proposal in
$F_{i}$. Recall that $p^{+}_{{\bm{\theta}}}$ and $p_{{\bm{\omega}}}^{-}$ are
our foreground and learned background distributions introduced in Section 2.3.
Note that it is unnecessary to know the normalization factors $Z$ and $U$,
since they cancel. In this step, proposal with a high foreground to background
score ratio gets the highest label value within each image.

In the M-step, the mean direction ${\bm{\theta}}$ is updated given the new
labels

$\begin{split}{\bm{\theta}}&\leftarrow\rm{norm}(\sum_{i=1}^{M}\tilde{\mathbf{x}}_{i})~{},\\\
\text{where}\quad\tilde{\mathbf{x}}_{i}&=\mathbf{w}_{i}^{\top}F_{i}=\sum_{k=1}^{P}\mathbf{w}_{ik}F_{ik}~{},\\\
\end{split}$ (4)

where $\rm{norm}(.)$ is $\ell_{2}$ normalization operator. Intuitively, one
can see $\tilde{\mathbf{x}}_{i}$ as the common object representation within
the $i$-th image which is estimated by computing the weighted average over all
the proposals where the contribution of proposals are controlled by their soft
labels. Given $\tilde{\mathbf{x}}_{i}$, the novel class direction
${\bm{\theta}}$ is estimated as the mean of the common object representations
similar to the prototypical networks [35]. Finally, the estimated mean is
projected back onto the unit hypersphere.

Algorithm 1 summarizes our COL method. The problem is solved in an iterative
fashion by alternating between E-step in Eq. 3 and M-step in Eq. 4 until
convergence. Following the common practice in WSOD [28, 37, 25], we use the
bounding box feature extracted from the complete support images to initialize
${\bm{\theta}}$. Recall that we use the first proposal in $F_{i}$ to represent
the complete image feature. Thus, the initialization step can be written as

${\bm{\theta}}_{\rm init}\leftarrow\rm{norm}(\sum_{i=1}^{M}F_{i1})~{}.$ (5)

Input: $\mathcal{F}=\\{F_{1},\dots,F_{M}\\}$, $u^{-}_{\bm{\omega}}$

Output: Common class mean direction ${\bm{\theta}}$

${\bm{\theta}}\leftarrow\rm{norm}(\sum_{i=1}^{M}F_{i1})$

// Initialization

1 for  _$t\leftarrow 1$ to T_ do // Iterations

2 for  _$i\leftarrow 1$ to M_ do // E-step

3 $\mathbf{o}_{ij}\leftarrow\kappa{\bm{\theta}}^{\top}F_{ij}-\log
u_{{\bm{\omega}}}^{-}(F_{ij})~{}~{}~{}\forall j\in[1,P]$

$\mathbf{w}_{i}\leftarrow\text{softmax}(\mathbf{o}_{i})$

// Update Soft labels

4

${\bm{\theta}}\leftarrow\rm{norm}(\sum_{i=1}^{M}\mathbf{w}_{i}^{\top}F_{i})$

// M-step

5

Algorithm 1 Common Object Localization

We remark that our initial direction is similar to what is used as class mean
in prototypical networks [35]. What makes us different is EM steps that refine
the estimated mean by focusing on the common objects and discarding background
parts of the image.

##### Finding the Common Object in the Query Set

For a single feature proposal $\mathbf{x}\in\mathbb{S}^{d-1}$ extracted from
query image $\mathbf{I}$, our goal is to estimate the class label
$c\in\\{0,1\\}$ which indicates if query proposal tightly encloses the target
object. Given the estimated common object mean ${\bm{\theta}}$ and the
background class distribution $p_{\bm{\omega}}^{-}$, we compute conditional
class distribution function $P(c|\mathbf{x})\propto P(c)p(\mathbf{x}|c)$ where
$P(c)$ and $p(\mathbf{x}|c)$ are the class prior and likelihood, respectively.
Let $P(c=1)=\alpha$, using the background and vMF foreground class
likelihoods, the conditional class distribution is written as

$P(c|\mathbf{x})\propto\begin{cases}\exp\left(\kappa{\bm{\theta}}^{\top}\mathbf{x}-\log
u_{\bm{\omega}}^{-}(\mathbf{x})\right)&~{}~{}c=1\\\
\lambda&~{}~{}c=0~{},\end{cases}$ (6)

where $\lambda=(1-\alpha)/\alpha\times Z/U$ ($Z$ is vMF normalizer)
encapsulates all the constants. Equivalently, proposal $\mathbf{x}$ can be
classified via a softmax over the logits

${\rm
logit}(c|\mathbf{x})=\begin{cases}\kappa{\bm{\theta}}^{\top}\mathbf{x}-\log
u_{\bm{\omega}}^{-}(\mathbf{x})&~{}~{}c=1\\\
\log\lambda&~{}~{}c=0~{}.\end{cases}$ (7)

We set $\lambda=1$ for all the COL experiments. Changing $\lambda$ adjusts the
confidence values but keep the order of the final scores the same, therefore,
its value does not affect the mean Average Precision (mAP) or Correct
Localization (CorLoc) metrics.

###  2.5 WSOD

For the task of WSOD where we have more than one target class, our COL
algorithm is first used to label instances of each class. Once the support set
is labeled, an off-the-shelf few-shot object detection model can be used for
learning novel classes. Inspired by the success of the recent few-shot object
detection method in [41], we employ a single layer cosine similarity
classifier for learning.

Learning is performed on one target class $c\in\mathcal{L}$ at a time. Let
$\mathbf{v}_{c}\in\mathbb{R}^{d}$ denote the classifier weight for class $c$.
The classification score for this class is computed as

$s_{c}(\mathbf{x})=\frac{\tau\mathbf{v}_{c}^{\top}\mathbf{x}}{\|\mathbf{v}_{c}\|}~{},$
(8)

where $\mathbf{x}\in\mathbb{S}^{d-1}$ is the $\ell_{2}$ normalized feature
proposal extracted by our Faster-RCNN model and $\tau$ is temperature
hyperparameter. For class $c$, the input training set $\mathcal{D}_{\rm
train}$ is split into positive images $\mathcal{D}^{c}_{\rm train}$ of images
that have the target class and negative set $\mathcal{D}_{\rm
train}\setminus\mathcal{D}^{c}_{\rm train}$, images without the target class.
Then, we label $\mathcal{D}^{c}_{\rm train}$ by running the COL algorithm on
the positive images and select the proposal with the highest soft label from
each image as common object representative. All the proposals in the negative
set are used as negative examples. Finally, $\mathbf{v}_{c}$ is learned by
minimizing the sigmoid cross entropy loss over the positive and negative
proposals. We use L-BFGS optimizer with strong Wolfe line search for faster
convergence.

At the test time, test proposal $\mathbf{x}$ from the query set
$\mathcal{D}_{\rm test}$ is scored using the classifiers learned for each
novel class.

##  3 Related Work

Multiple instance learning methods such as MI-SVM [2] have been extensively
used for large-scale weakly supervised object detection. In a standard
multiple instance learning framework, latent bounding box labels and the
appearance model are estimated jointly in an alternating optimization process
with the constraint that at least one bounding box should be positive in each
image. Alternating optimization combined with modern deep neural network
architecture is a dominant technique in the literature showing the state-of-
the-art performance in WSOD [28, 37, 10, 34]. Ilse _et al_. [14] propose an
attention-based deep multiple instance learning architecture where bag label
probability distribution is learned by neural networks. More related to our
work are a class of WSOD algorithms that use knowledge transfer from a fully
annotated base dataset to aid WSOD for a set of novel classes [28, 37, 12, 5].
In [37], a class agnostic objectness score is learned from the base dataset
and is utilized to guide the multiple instance learning optimization steering
toward objects and away from the background. These methods rely on a
relatively large dataset to learn novel categories.

Co-localization [19], co-segmentation [18, 38], and co-saliency [46] methods
have the same kind of output as weakly-supervised object localization but they
typically do not utilize negative examples. More recently, several methods
were developed for localizing the common novel object under few-shot setting
[31, 13, 32]. Shaban _et al_. [31] learn a pairwise potential function between
proposals of the base classes and use this pairwise metric to solve a minimum-
energy labeling problem over a bidirectional graphical model to co-localize
novel classes. SILCO [13] finds the common object by computing a dense
similarity map between each support image and the query while only exploring
the similarity among support images using their coarse image-level features
via a global average pooling. Although using global average pooling reduces
the computation, ignoring the dense similarities among support images
negatively affects the common object localization.

Few-shot learning has gained a lot of attention in image classification [6,
30, 21, 8, 39]. Prototypical networks [35] use the mean of embedded support
examples to represent novel class prototypes and classifies query examples by
comparing their distances to the class prototypes. More recently, Qi _et al_.
[27] propose a weight imprinting process to learn the prototypes on the unit
hypersphere. Learning on the unit hypersphere has been employed by other few-
shot learning algorithms for better generalization [9] and stabilizing the
training [41]. Most recently, Yang _et al_. [45] propose to make the
distributions more Gaussian like by transforming the features of the support
set and query set using Tukey’s Ladder of Powers transformation [36]. It is
shown that Tukey’s normalization significantly improves the performance of
few-shot prototypical learning. As the scope of these methods is limited to
supervised learning, we compare different normalization and transformations
used in the literature for the weakly-supervised task of COL in Section 4.4.

##  4 Experiments

We evaluate the proposed method in few-shot COL and WSOD problems. We compare
our work (vMF-MIL) with Greedy Tree [31] and SILCO [13], two state-of-the-art
methods for the task of few-shot common object localization.

To the best of our knowledge there is no WSOD algorithm for few-shot setting
in the literature. However, WSOD with knowledge transfer methods [28, 37, 12,
5] are closely related to our work. We describe a slightly modified version of
[37], called MI-SVM in our experiments, in Appendix C and discuss its
differences to the proposed method. The MI-SVM baseline is not applicable to
COL problem as it always requires negative examples for training. To compare
MI-SVM against other COL methods, we provide MI-SVM with an extra set of $K$
negative images that do not have the target class when sampling the support
set.

The original version of Greedy Tree selects only one proposal from each image
in the support set and does not perform detection on a new query image. To
make it compatible with other methods, we add a simple inference step to the
Greedy Tree. Let $\mathcal{O}=\\{\mathbf{x}_{1},\dots,\mathbf{x}_{M}\\}$
denote the set of selected proposals, one from each image in the support set.
We score feature proposal $\mathbf{x}$ from the query image as a sum of its
pairwise similarities to all the selected proposals, i.e.,
$\text{score}(\mathbf{x})=\sum_{j=1}^{M}r(\mathbf{x},\mathbf{x}_{j})$, where
$r$ is the learned pairwise similarity function by Greedy Tree. The computed
score measures the negative change in the energy value if $\mathbf{x}$ were
added as a new node to the graph labeling problem used in [31].

In all the methods, we first hold out $20$ base classes for validation and
hyperparameter tuning and then re-train on all the base classes with the best
found parameters. For evaluation, we compute the correct localization (CorLoc)
rate [5] and mean Average Precision (mAP) with IoU overlap threshold of $0.5$
on the query image.

###  4.1 Common Object Localization

Table 1: CorLoc (top) and mAP (bottom) performance of different few-shot
common object localization methods on VOC07 test set. All of the models are
trained on COCO60 and evaluated on a test query with $K=5$ images in the
support set. The best and second best performing methods are shown in bold and
gray background respectively. ∗MI-SVM receives $K$ extra negative images.

method | aero | bike | bird | boat | bottle | bus | car | cat | chair | cow | table | dog | horse | mbike | person | plant | sheep | sofa | train | tv | CorLoc
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
MI-SVM∗ [37] | $29.4$ | $13.2$ | $53.7$ | $32.7$ | $12.9$ | $70.4$ | $66.4$ | $67.4$ | $15.7$ | $81.6$ | $10.0$ | ${\bf 67.6}$ | $67.1$ | $27.1$ | $10.1$ | ${\bf 16.7}$ | $84.2$ | $38.9$ | $43.5$ | $41.1$ | $42.5$
SILCO [13] | $51.0$ | $30.3$ | $50.7$ | $34.5$ | $11.3$ | $72.2$ | $63.6$ | $58.9$ | $11.2$ | $86.8$ | $6.7$ | $56.9$ | $51.9$ | ${\bf 49.2}$ | $13.0$ | ${\bf 16.7}$ | $52.6$ | $41.1$ | $46.8$ | $34.2$ | $42.0$
Greedy Tree [31] | $35.3$ | $21.1$ | ${\bf 59.7}$ | $34.5$ | ${\bf 24.2}$ | ${\bf 77.8}$ | $73.4$ | $61.1$ | ${\bf 23.1}$ | $89.5$ | $15.0$ | $64.7$ | $73.4$ | $25.4$ | $12.8$ | $13.3$ | ${\bf 100.0}$ | ${\bf 64.2}$ | $61.3$ | ${\bf 46.6}$ | $48.8$
vMF-MIL (ours) | ${\bf 62.7}$ | ${\bf 42.1}$ | $53.7$ | ${\bf 49.1}$ | $6.5$ | $68.5$ | ${\bf 73.8}$ | ${\bf 69.5}$ | $19.4$ | ${\bf 97.4}$ | ${\bf 36.7}$ | $65.7$ | ${\bf 82.3}$ | $40.7$ | ${\bf 21.7}$ | $15.0$ | $94.7$ | ${\bf 64.2}$ | ${\bf 69.4}$ | $31.5$ | ${\bf 53.2}$
method | aero | bike | bird | boat | bottle | bus | car | cat | chair | cow | table | dog | horse | mbike | person | plant | sheep | sofa | train | tv | mAP
MI-SVM∗ [37] | $17.7$ | $7.7$ | $31.6$ | $10.6$ | $4.3$ | $46.5$ | $40.1$ | $53.3$ | $3.6$ | $56.8$ | $3.3$ | ${\bf 56.3}$ | $42.3$ | $17.1$ | $1.7$ | ${\bf 8.1}$ | $37.9$ | $25.9$ | $27.3$ | $19.2$ | $25.6$
SILCO [13] | $33.0$ | $13.4$ | $34.5$ | ${\bf 14.8}$ | $3.9$ | $48.8$ | $38.4$ | $55.5$ | $4.0$ | $52.8$ | $4.5$ | $54.2$ | $36.3$ | ${\bf 27.3}$ | $3.3$ | $7.7$ | $27.0$ | $31.3$ | $36.7$ | $23.5$ | $27.5$
Greedy Tree [31] | $26.0$ | $8.7$ | $37.4$ | $11.5$ | ${\bf 7.5}$ | $52.4$ | $47.7$ | $45.8$ | ${\bf 7.4}$ | $61.1$ | $4.5$ | $47.3$ | $50.7$ | $15.6$ | $2.3$ | $5.0$ | ${\bf 40.0}$ | ${\bf 46.3}$ | $47.3$ | ${\bf 25.7}$ | $29.5$
vMF-MIL (ours) | ${\bf 36.7}$ | ${\bf 20.6}$ | ${\bf 38.1}$ | $14.2$ | $1.9$ | ${\bf 55.4}$ | ${\bf 50.2}$ | ${\bf 56.5}$ | ${\bf 7.4}$ | ${\bf 71.4}$ | ${\bf 9.5}$ | $56.2$ | ${\bf 63.4}$ | $16.8$ | ${\bf 4.9}$ | $3.4$ | $39.3$ | $43.0$ | ${\bf 51.2}$ | $23.0$ | ${\bf 33.1}$

Table 2: CorLoc(%) and mAP(%) results of different methods for the task of
common object localization on novel object classes on the COCO60 dataset with
support set size $K=5$ and $K=10$. ∗MI-SVM receives $K$ extra negative images.

Model | K = 5 | K=10
---|---|---
<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
MI-SVM∗ [37] | $30.5$ | $15.9$ | $33.2$ | $16.2$
SILCO [13] | $29.7$ | $14.8$ | $31.3$ | $15.8$
Greedy Tree [31] | $32.7$ | $16.0$ | $33.8$ | $16.4$
vMF-MIL (ours) | ${\bf 34.8}$ | ${\bf 18.6}$ | ${\bf 36.9}$ | ${\bf 20.0}$

We use the official implementations of SILCO and Greedy Tree for this
experiment. To have a fair comparison with SILCO, we employ Faster-RCNN with a
VGG16 [33] backbone architecture for feature extraction in both Greedy Tree
and our method.

We evaluate on a popular MS COCO 2014 [20] split used in few-shot object
detection methods [41, 26, 43, 44, 15], named COCO60. In COCO60 split, $60$
categories disjoint with the PASCAL VOC dataset are used as base classes and
the remaining $20$ classes are used as novel classes. This allows us to also
perform a cross-dataset evaluation on PASCAL VOC07 [7] test set. We evaluate
the performance of each method over $2000$ randomly sampled tasks.

Table 1 and Table 2 summarize the results on PASCAL VOC and MS COCO datasets,
respectively. Despite its simplicity, our method outperforms all the methods
by a large margin, followed by Greedy Tree and SILCO. Specifically, we gain
between $10\%$ to $20\%$ relative improvement in mAP metric against the second
best performing method. The proposed method and Greedy Tree both estimate
latent proposal-level labels of the support images to find the common object.
However, SILCO explores the dense similarity between each support image and
the query image while using coarse image-level features via a global average
pooling to estimate the relation of support images. This experiment confirms
estimating proposal-level labels within the support images is quite important
for common object localization.

####  4.1.1 Direct Comparison to Greedy Tree

To ensure a fair comparison, we also compare our common object localization
unit to the Greedy Tree by exactly following the original experimental
protocol in [31]. Their algorithm utilizes a split of the COCO 2017 dataset
with $63$ base classes for training and $17$ held-out novel classes for
testing the algorithm. The trained model is also tested on a subset of the
ILSVRC 2013 detection dataset with $148$ novel classes that have no overlap
with the base classes. In Greedy Tree, a Faster-RCNN with ResNet50 [11]
backbone is first trained on the base classes and used to extract features
from all the images. To allow a fair comparison, we use the same feature set
provided by the authors. To mimic common object localization during training,
we sample tasks with $N=1$ and $K=8$ for training.

Similar to [31], we evaluate our model over $1000$ randomly sampled tasks each
containing $K=8$ images with an object class in common. For each image, the
proposal with the highest soft label in Eq. 3 is returned as the common
object. We report the class-agnostic CorLoc ratio on COCO and ILSVRC datasets
in Table 3 and compare it with the results in [31]. vMF-MIL outperforms Greedy
Tree by $2.20\%$ and $1.75\%$ in MS COCO and ILSVRC datasets, respectively.

Table 3: Class-agnostic CorLoc(%) with $95\%$ confidence interval of the
methods in [31] compared to our method. All methods use $K=8$ positive images
for finding the common object.

method | COCO | ILSVRC13
---|---|---
TRWS [31] | $64.53\pm 1.05$ | $52.95\pm 1.09$
ASTAR [31] | $64.54\pm 1.05$ | $52.89\pm 1.09$
Greedy Tree [31] | $64.65\pm 1.05$ | $53.00\pm 1.10$
vMF-MIL (ours) | ${\bf 66.85\pm 1.03}$ | ${\bf 54.75\pm 1.09}$

###  4.2 Few-shot WSOD

We train our model on COCO60 for the task of few-shot WSOD with different
$N$-way, $K$-shot problems and compare it with the knowledge transfer MI-SVM
model described in Appendix C on PASCAL VOC 2007 and MS COCO novel classes in
Table 4. To highlight the importance of EM refinement, we also train our model
with full image prototypical initialization without EM refinement. In both
datasets, vMF-MIL outperforms MI-SVM in all the scenarios, demonstrating the
strong generalization ability of our learning approach.

Table 4: mAP(%) of different few-shot WSOD methods on COCO60 and PASCAL VOC
datasets.

Method | Dataset | $N=5$ | $N=10$ | $N=20$
---|---|---|---|---
$K=5$ | K=10 | K=5 | K=10 | K=5 | K=10
Prototypical Init | VOC07 | $16.01$ | $17.93$ | $10.56$ | $11.02$ | $5.41$ | $6.72$
MI-SVM [37] | $17.99$ | $20.27$ | $12.09$ | $13.07$ | $7.04$ | $8.32$
vMF-MIL (ours) | ${\bf 21.22}$ | ${\bf 22.01}$ | ${\bf 14.54}$ | ${\bf 15.83}$ | ${\bf 8.83}$ | ${\bf 10.19}$
Prototypical Init | COCO60 | $8.90$ | $9.28$ | $4.65$ | $6.07$ | $2.99$ | $3.26$
MI-SVM [37] | $11.40$ | $11.60$ | $7.30$ | $7.80$ | $2.97$ | $3.70$
vMF-MIL (ours) | ${\bf 12.35}$ | ${\bf 13.19}$ | ${\bf 8.53}$ | ${\bf 10.07}$ | ${\bf 4.23}$ | ${\bf 4.85}$

Figure 5:  mAP(%) vs. number of EM iterations in common object localization
task with $K=5$ on COCO60 and VOC07 datasets. The performance reaches a
plateau at step $4$.

###  4.3 Large-Scale WSOD

Although our method is designed for low-shot settings, it is interesting to
evaluate its performance in the standard WSOD setting as well. Related to our
work is transfer learning approaches for large-scale WSOD [37, 12] with
ImageNet detection as the standard benchmark. Typically, the first $100$
classes are used as the base dataset and the rest $100$ with $65$k images are
used as novel objects. We follow the setup in [37] and use the pre-trained
Inception-Resnet Faster-RCNN model and weights provided by the authors to
extract proposals, features, and the objectness scores. We apply our EM
algorithm to the extracted features and use
$u^{-}_{\bm{\omega}}(\mathbf{x})=\alpha(1-\rm{obj}(\mathbf{x}))$ for each
proposal $\mathbf{x}$ where $\rm{obj}(\mathbf{x})$ is the Faster-RCNN
objectness score and $\alpha$ is a hyper-parameter we tuned for the task. To
our surprise, vFM-MIL outperforms [37] in Table 5 while being about
$100\times$ faster. We believe this results can be further improved by
relaxing some of the assumptions in our statistical model as overfitting may
not be as significant in large-scale. For instance, we can learn a separate
concentration parameter for each novel class in the EM steps. Furthermore, we
can utilize the novel dataset to update the background scoring function
$u^{-}_{\omega}$. We defer these improvements and further analysis to the
future work.

Table 5: Large-Scale WSOD on ImageNet Detection.

Model |<EMAIL_ADDRESS>| Time (min.)
---|---|---
LSDA (JMLR 2016) [12] | $28.8$ | -
Uijlings et al. (CVPR 18) [37] | $74.2$ |  $900$ (estimated)
vMF-MIL (ours) | $\mathbf{76.5}$ | $10$

###  4.4 Ablation Study

To understand which parts of the proposed method are critical for common
object localization, we analyzed results in Table 2 with $K=5$ for each of the
important components of the proposed method in Table 6. These components are:
initializing ${\bm{\theta}}$ with the mean of complete image features
(Prototypical Init), EM updates, and learning background distribution
$p_{\bm{\omega}}^{-}$ to steer the algorithm toward objects. The first entry
(#1) in Table 6 shows that there is a huge performance gap when the background
model is not used. This is expected, since without using the background model
it may localize non-object patterns such as grass, water, building, etc. with
similar appearances as the common object. Comparing the third entry (#3) with
the complete model (#5) reveals that using EM refinements alone increases
CorLoc by $3.9\%$ and mAP by $5.3\%$. Finally, the fourth entry shows that the
EM steps are only effective if ${\bm{\theta}}$ is initialized with the
complete image proposal.

Second part of Table 6 shows the advantage of using vMF to Gaussian
distribution in the EM algorithm (see Appendix B for the details). Tukey’s
transformation [36] furthur improves the performance of Gaussian model but vMF
distribution still exhibits the best performance. We believe this is because
feature vectors’ direction better capture the semantic information.

Table 6: Ablation study on COCO60 dataset. #1-5 show the importance of
initialization, iterative EM updates, and learning the background model. #6-8
compare different statistical models in the EM algorithm.

# | Random Init | Prototypical Init | EM Updates | $p_{\bm{\omega}}^{-}$ | CorLoc | mAP
---|---|---|---|---|---|---
1 |  | ✓ | ✓ |  | $1.9$ | $0.6$
2 |  |  |  | ✓ | $22.8$ | $9.3$
3 |  | ✓ |  | ✓ | $30.9$ | $13.3$
4 | ✓ |  | ✓ | ✓ | $30.1$ | $14.2$
5 |  | ✓ | ✓ | ✓ | ${\bf 34.8}$ | ${\bf 18.6}$
| Gaussian | Tukey+Gaussian | vMF |  |  |
6 | ✓ |  |  |  | $29.8$ | $13.9$
7 |  | ✓ |  |  | $34.0$ | $17.3$
8 |  |  | ✓ |  | ${\bf 34.8}$ | ${\bf 18.6}$

Finally, we illustrate the performance improvement vs. the number of EM steps
in Fig. 5. In both VOC07 and COCO60 datasets, mAP reaches a plateau showing
that the algorithm converges quickly. Qualitative results in Fig. 6 depict
successful cases where EM refinements improve the top prediction.

Figure 6:  Bounding box adjustments at each iteration for the common object
localization experiment on COCO60 with $K=5$. Only the top prediction in the
query image is shown(in pink color) for each iteration. Ground-truth bounding
boxes of the target classes are shown in green. EM refinements improve the
target object localization in query image.

##  5 Conclusion

We have presented vMF-MIL, a multiple instance learning framework to address
the problem of few-shot common object localization and WSOD. vMF-MIL uses a
simple inductive bias in learning to combat the overfitting issue in few-shot
learning. Specifically, instances of each class are assumed to form a cluster
on a unit hypersphere, where the mean corresponds to the class prototype. Our
experiments on few-shot common object localization illustrate the advantage of
our simple approach over several state-of-the-art methods, improving the few-
shot WSOD performance compared with the strong MI-SVM baseline.

## References

  * [1] Brandon Amos and J. Zico Kolter.  OptNet: Differentiable optimization as a layer in neural networks.  In Int. Conf. Mach. Learn., 2017.
  * [2] Stuart Andrews, Ioannis Tsochantaridis, and Thomas Hofmann.  Support vector machines for multiple-instance learning.  In Adv. Neural Inform. Process. Syst., 2003.
  * [3] Arindam Banerjee, Inderjit S Dhillon, Joydeep Ghosh, Suvrit Sra, and Greg Ridgeway.  Clustering on the unit hypersphere using von mises-fisher distributions.  Journal of Machine Learning Research, 2005.
  * [4] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A Large-Scale Hierarchical Image Database.  In IEEE Conf. Comput. Vis. Pattern Recog., 2009.
  * [5] Thomas Deselaers, Bogdan Alexe, and Vittorio Ferrari.  Localizing objects while learning their appearance.  In Eur. Conf. Comput. Vis., 2010.
  * [6] Carl Doersch, Ankush Gupta, and Andrew Zisserman.  Crosstransformers: spatially-aware few-shot transfer.  Adv. Neural Inform. Process. Syst., 2020.
  * [7] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results.  http://www.pascal-network.org/challenges/VOC/voc2007/workshop/index.html.
  * [8] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  Int. Conf. Mach. Learn., 2017.
  * [9] Spyros Gidaris and Nikos Komodakis.  Dynamic few-shot visual learning without forgetting.  In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
  * [10] Ramazan Gokberk Cinbis, Jakob Verbeek, and Cordelia Schmid.  Multi-fold mil training for weakly supervised object localization.  In IEEE Conf. Comput. Vis. Pattern Recog., 2014.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In IEEE Conf. Comput. Vis. Pattern Recog., 2016.
  * [12] Judy Hoffman, Deepak Pathak, Eric Tzeng, Jonathan Long, Sergio Guadarrama, Trevor Darrell, and Kate Saenko.  Large scale visual recognition through adaptation using joint representation and multiple instance learning.  The Journal of Machine Learning Research, 2016.
  * [13] Tao Hu, Pascal Mettes, Jia-Hong Huang, and Cees GM Snoek.  Silco: Show a few images, localize the common object.  In Int. Conf. Comput. Vis., 2019.
  * [14] Maximilian Ilse, Jakub M Tomczak, and Max Welling.  Attention-based deep multiple instance learning.  Int. Conf. Mach. Learn., 2018.
  * [15] Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and T revor Darrell.  Few-shot object detection via feature reweighting.  In Int. Conf. Comput. Vis., 2019.
  * [16] Daesik Kim, Gyujeong Lee, Jisoo Jeong, and Nojun Kwak.  Tell me what they’re holding: Weakly-supervised object detection with transferable knowledge from human-object interaction.  In AAAI, 2020.
  * [17] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto.  Meta-learning with differentiable convex optimization.  In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
  * [18] Weihao Li, Omid Hosseini Jafari, and Carsten Rother.  Deep object co-segmentation.  In ACCV, 2018.
  * [19] Yao Li, Lingqiao Liu, Chunhua Shen, and Anton van den Hengel.  Image co-localization by mimicking a good detector’s confidence score distribution.  In Eur. Conf. Comput. Vis., 2016.
  * [20] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Eur. Conf. Comput. Vis., 2014.
  * [21] Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang.  Learning to propagate labels: Transductive propagation network for few-shot learning.  Int. Conf. Learn. Represent., 2019.
  * [22] Xiankai Lu, Wenguan Wang, Chao Ma, Jianbing Shen, Ling Shao, and Fatih Porikli.  See more, know more: Unsupervised video object segmentation with co-attention siamese networks.  In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
  * [23] Kanti V Mardia and Peter E Jupp.  Directional statistics.  John Wiley & Sons, 2009.
  * [24] Oded Maron and Tomás Lozano-Pérez.  A framework for multiple-instance learning.  In Adv. Neural Inform. Process. Syst., 1998.
  * [25] Minh Hoai Nguyen, Lorenzo Torresani, Fernando De La Torre, and Carsten Rother.  Weakly supervised discriminative localization and classification: a joint learning process.  In Int. Conf. Comput. Vis., 2009.
  * [26] Juan-Manuel Perez-Rua, Xiatian Zhu, Timothy M Hospedales, and Tao Xiang.  Incremental few-shot object detection.  In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
  * [27] Hang Qi, Matthew Brown, and David G Lowe.  Low-shot learning with imprinted weights.  In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
  * [28] Amir Rahimi, Amirreza Shaban, Thalaiyasingam Ajanthan, Richard Hartley, and Byron Boots.  Pairwise similarity knowledge transfer for weakly supervised object localization.  In Eur. Conf. Comput. Vis., 2020.
  * [29] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  IEEE Trans. Pattern Anal. Mach. Intell., 2016.
  * [30] Pau Rodríguez, Issam Laradji, Alexandre Drouin, and Alexandre Lacoste.  Embedding propagation: Smoother manifold for few-shot classification.  Eur. Conf. Comput. Vis., 2020.
  * [31] Amirreza Shaban, Amir Rahimi, Shray Bansal, Stephen Gould, Byron Boots, and Richard Hartley.  Learning to find common objects across few image collections.  In Int. Conf. Comput. Vis., 2019.
  * [32] Mennatullah Siam, Naren Doraiswamy, Boris N Oreshkin, Hengshuai Yao, and Martin Jagersand.  Weakly supervised few-shot object segmentation using co-attention with visual and semantic embeddings.  IJCAI, 2020.
  * [33] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  Int. Conf. Learn. Represent., 2015.
  * [34] Parthipan Siva and Tao Xiang.  Weakly supervised object detector learning with model drift detection.  In Int. Conf. Comput. Vis., 2011.
  * [35] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  In Adv. Neural Inform. Process. Syst., 2017.
  * [36] John W Tukey.  Exploratory data analysis.  Addison-Wesley Series in Behavioral Science. Addison-Wesley, Reading, MA, 1977.
  * [37] Jasper Uijlings, Stefan Popov, and Vittorio Ferrari.  Revisiting knowledge transfer for training object class detectors.  In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
  * [38] Sara Vicente, Carsten Rother, and Vladimir Kolmogorov.  Object cosegmentation.  In IEEE Conf. Comput. Vis. Pattern Recog., 2011.
  * [39] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  In Adv. Neural Inform. Process. Syst., 2016.
  * [40] Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille.  Normface: L2 hypersphere embedding for face verification.  In Proceedings of the 25th ACM international conference on Multimedia, pages 1041–1049, 2017.
  * [41] Xin Wang, Thomas E Huang, Trevor Darrell, Joseph E Gonzalez, and Fisher Yu.  Frustratingly simple few-shot object detection.  Int. Conf. Mach. Learn., 2020.
  * [42] Yu-Xiong Wang, Deva Ramanan, and Martial Hebert.  Meta-learning to detect rare objects.  In Int. Conf. Comput. Vis., 2019.
  * [43] Yang Xiao and Renaud Marlet.  Few-shot object detection and viewpoint estimation for objects in the wild.  Eur. Conf. Comput. Vis., 2020.
  * [44] Xiaopeng Yan, Ziliang Chen, Anni Xu, Xiaoxi Wang, Xiaodan Liang, and Liang Lin.  Meta r-cnn: Towards general solver for instance-level low-shot learning.  In Int. Conf. Comput. Vis., 2019.
  * [45] Shuo Yang, Lu Liu, and Min Xu.  Free lunch for few-shot learning: Distribution calibration.  Int. Conf. Learn. Represent., 2021.
  * [46] Dingwen Zhang, Junwei Han, Chao Li, and Jingdong Wang.  Co-saliency detection via looking deep and wide.  In IEEE Conf. Comput. Vis. Pattern Recog., 2015.

Appendix

Few-shot Weakly-Supervised Object Detection via Directional Statistics

##  Appendix A Expectation-Maximization Derivation

Recall that each proposal has a (latent) binary label
$\mathbf{z}_{ij}\in\\{0,1\\}$ that indicates whether the proposal tightly
encloses the common object. Following the best practice of the previous works
in WSOD [28, 37, 10], we assume there is exactly one proposal with label $1$
(positive proposal) in each image and the rest are negative proposals, i.e.,
$\mathbf{z}_{i}\in\\{\mathbf{e}_{1},\dots\mathbf{e}_{P}\\}$ where
$\mathbf{e}_{j}\in\\{0,1\\}^{P}$ is the $j$-th canonical basis.

Assuming that the images are sampled independently given the common class $c$,
the full likelihood function is given by

$\begin{split}l({\bm{\theta}};\mathcal{F})&=p(\mathcal{F}|{\bm{\theta}})=\prod_{i=1}^{M}p(F_{i}|{\bm{\theta}})\\\
&=\prod_{i=1}^{M}\sum_{j=1}^{P}p(F_{i},\mathbf{z}_{i}=\mathbf{e}_{j}|{\bm{\theta}})~{},\end{split}$
(9)

where ${\bm{\theta}}\in\mathbb{S}^{d-1}$ is the mean direction of the common
class distribution. Note that the last equation integrates over all possible
values of $\mathbf{z}_{i}$. Assuming that proposals are i.i.d samples from
their corresponding distributions given their labels $\mathbf{z}_{i}$222i.i.d
assumption is a standard approach in MIL and works well in practice. See [24]
for more details., we can write

$p(F_{i}|\mathbf{z}_{i}=\mathbf{e}_{j},{\bm{\theta}})=p_{{\bm{\theta}}}^{+}(F_{ij})\prod_{\begin{subarray}{c}k=1\\\
k\neq j\end{subarray}}^{P}p_{{\bm{\omega}}}^{-}(F_{ik})~{},$ (10)

where $F_{ij}$ is the feature of the $j$-th proposal in $F_{i}$,
$p_{{\bm{\theta}}}^{+}$ is the generic distribution that generates the common
class proposals, and $p_{{\bm{\omega}}}^{-}$ represents background proposals’
distribution. For brevity, let us re-write Eq. 10 in a more compact form

$p(F_{i}|\mathbf{z}_{i}=\mathbf{e}_{j},{\bm{\theta}})=q_{{\bm{\theta}}}(F_{ij})p^{-}_{{\bm{\omega}}}(F_{i})~{},$
(11)

where $q_{{\bm{\theta}}}$ is quotient of object and background distributions
$q_{{\bm{\theta}}}(\mathbf{x})=p_{{\bm{\theta}}}^{+}(\mathbf{x})/p_{{\bm{\omega}}}^{-}(\mathbf{x})$,
and
$p^{-}_{{\bm{\omega}}}(F_{i})=\prod_{j=1}^{P}p_{{\bm{\omega}}}^{-}(F_{ij})$.

We adopt the EM algorithm to maximize the likelihood in Eq. 9 by iteratively
optimizing the surrogate expected log-likelihood which is easier to compute.
In the E-step, the posterior distribution of the latent variables
$\mathbf{w}_{ik}=p(\mathbf{z}_{i}=\mathbf{e}_{k}|F_{i},{\bm{\theta}})$ are
computed for the current ${\bm{\theta}}$. By Eq. 11, using Bayes’ theorem, and
assuming a uniform distribution over image labels $\mathbf{z}_{i}$, the
posterior can be expressed in terms of quotient of distributions defined above

$\mathbf{w}_{ik}=\frac{q_{{\bm{\theta}}}(F_{ik})}{\sum_{j=1}^{P}q_{{\bm{\theta}}}(F_{ij})}~{},$
(12)

yielding soft label vector $\mathbf{w}_{i}\in\mathbb{R}^{P}$ for the $i$-th
image proposals.

By plugging in the vMF probability density function of the common class and
background probability density function, the quotient $q_{{\bm{\theta}}}$ can
be written as

$q_{{\bm{\theta}}}(\mathbf{x})\propto\exp\Big{(}\kappa{\bm{\theta}}^{\top}\mathbf{x}-\log
u_{{\bm{\omega}}}^{-}(\mathbf{x})\Big{)}~{}.$ (13)

As shown in Algorithm 1, one can compute the soft labels via the softmax
operation, resembling the attention mechanism recently used for MIL [14].

In the M-step, parameters ${\bm{\theta}}$ are updated by maximizing the
surrogate expected log-likelihood using the posteriors computed in the E-step

$l({\bm{\theta}}^{\prime};{\bm{\theta}})=\sum_{i=1}^{M}\mathbb{E}_{p(\mathbf{z}_{i}|F_{i},{\bm{\theta}})}\Big{[}\log
p(F_{i}|\mathbf{z}_{i},{\bm{\theta}}^{\prime})\Big{]}=\sum_{i=1}^{M}\sum_{k=1}^{P}\mathbf{w}_{ik}\log
p(F_{i}|\mathbf{z}_{i}=\mathbf{e}_{k},{\bm{\theta}}^{\prime})~{},$ (14)

where the weights $\mathbf{w}_{ik}$ are computed in Eq. 3. Lagrangian function
is written as

$\mathcal{L}({\bm{\theta}}^{\prime},\lambda)=l({\bm{\theta}}^{\prime};{\bm{\theta}})-\lambda(\|{\bm{\theta}}^{\prime}\|^{2}-1)$
(15)

By plugging in the log-likelihood term in Eq. 15 and computing the derivative
w.r.t. ${\bm{\theta}}^{\prime}$ and $\lambda$ we have

$\begin{split}\nabla_{{\bm{\theta}}^{\prime}}\mathcal{L}({\bm{\theta}}^{\prime},\lambda)&=\sum_{i=1}^{M}\sum_{k=1}^{P}\mathbf{w}_{ik}\nabla_{{\bm{\theta}}^{\prime}}\log\Big{(}p_{{\bm{\theta}}^{\prime}}^{+}(F_{ij})\prod_{\begin{subarray}{c}k=1\\\
k\neq
j\end{subarray}}^{P}p_{{\bm{\omega}}}^{-}(F_{ik})\Big{)}-2\lambda{\bm{\theta}}^{\prime}=\kappa\sum_{i=1}^{M}\sum_{k=1}^{P}\mathbf{w}_{ik}F_{ik}-2\lambda{\bm{\theta}}^{\prime}~{}.\\\
\nabla_{\lambda}\mathcal{L}({\bm{\theta}}^{\prime},\lambda)&=-\|{\bm{\theta}}^{\prime}\|^{2}+1.\end{split}$
(16)

Finally, closed-form update rule

$\begin{split}{\bm{\theta}}&\leftarrow\rm{norm}(\sum_{i=1}^{M}\tilde{\mathbf{x}}_{i})~{},\\\
\text{where}\quad\tilde{\mathbf{x}}_{i}&=\mathbf{w}_{i}^{\top}F_{i}=\sum_{k=1}^{P}\mathbf{w}_{ik}F_{ik}~{},\end{split}$
(17)

is derived by setting the derivatives to zero and solving for
${\bm{\theta}}^{\prime}$ and $\lambda$.

###  A.1 Updating $\kappa$ in M-Step

In the paper, $\kappa$ is a constant hyperparameter for all the novel classes
and only ${\bm{\theta}}$ gets updated in the M-step. In this section, we
propose a simple update rule for parameter $\kappa$ that can be used along Eq.
17 in the M-step. As shown in Table 7, updating $\kappa$ with a our order-$0$
rule further improves our vMF-MIL COL results of the paper.

To find the optimal $\kappa$, we compute the derivative of the Lagrangian
function in Eq. 15 w.r.t. $\kappa$

$\begin{split}\partial_{\kappa}\mathcal{L}({\bm{\theta}}^{\prime},\lambda)&=\sum_{i=1}^{M}\sum_{k=1}^{P}\mathbf{w}_{ik}\partial_{\kappa}\log\Big{(}p_{{\bm{\theta}}^{\prime}}^{+}(F_{ij})\prod_{\begin{subarray}{c}k=1\\\
k\neq
j\end{subarray}}^{P}p_{{\bm{\omega}}}^{-}(F_{ik})\Big{)}=\sum_{i=1}^{M}\sum_{k=1}^{P}\mathbf{w}_{ik}\partial_{\kappa}\log
p_{{\bm{\theta}}^{\prime}}^{+}(F_{ik})\\\
&=\sum_{i=1}^{M}\sum_{k=1}^{P}\mathbf{w}_{ik}\partial_{\kappa}(-\log
Z(\kappa)+\kappa
F_{ik}^{\top}{\bm{\theta}}^{\prime})=-M\frac{\partial_{\kappa}Z(\kappa)}{Z(\kappa)}+\mathbf{r}^{\top}{\bm{\theta}}^{\prime}~{},\end{split}$
(18)

where $\mathbf{r}=\sum_{i=1}^{M}\sum_{k=1}^{P}\mathbf{w}_{ik}\mathbf{x}$ and
$Z(\kappa)$ is vMF distribution normalization factor. A precise formula is
known for $Z(\kappa)$, namely

$Z(\kappa)=\frac{(2\pi)^{d/2}\,I_{d/2-1}(\kappa)}{\kappa^{d/2-1}}~{}.$ (19)

where $d$ is the feature dimension and $I$ is the modified Bessel function.
This formula is quoted in [3](2.2), but it is upside-down compared to Eq. 19,
since we are defining $c_{d}(\kappa)=1/Z(\kappa)$.

By plugging in ${\bm{\theta}}^{\prime}=\mathbf{r}/\|\mathbf{r}\|$ from Eq. 17
and setting the derivative to zero we get

$\frac{\partial_{\kappa}Z(\kappa)}{Z(\kappa)}=\frac{\|\mathbf{r}\|}{M}=\bar{r}~{}.$
(20)

Equation 20 is similar to what we see in vMF maximum-likelihood estimation,
therefore, we can use the maximum-likelihood derivations from now on (See
Appendix of [3] equations A.7 to A.8) which leads to maximum-likelihood
estimation

$\kappa=A^{-1}_{d}(\bar{r})~{},$ (21)

where $A_{d}$ is the ratio of Bessel functions,

$A_{d}(\kappa)=\frac{I_{d/2}(\kappa)}{I_{d/2-1}(\kappa)}~{}.$ (22)

It is still numerically difficult to compute the Bessel function in cases
where $d$ and $\kappa$ are large. We are able to compute it in python using
the `scipy.special.iv` function, only for values of $d$ up to about $120$ and
$\kappa$ up to about $700$.

Figure 7: Plot of different estimates of $\hat{\kappa}$ as a function of
$\bar{r}$, for dimension $d=100$. At this resolution, the exact estimate is
indistinguishable from the estimate Eq. 24. The graph also shows
approximations of different orders, such as Eq. 26 and Eq. 27, which are
accurate for small-to-medium values of $\bar{r}$, but not for larger values.
However, the exact value of $\bar{\kappa}$ is extremely sensitive to small
variations in the value of $\bar{r}$, and it diverges to infinity as $\bar{r}$
approaches $1$. For this reason it may not be good practice (as is verified by
our experiments) to use the exact estimate of $\bar{\kappa}$ in clustering.

To address this difficulty, different formulae are given for estimating the
optimal value of $\hat{\kappa}$.

  1. 1.

A formula due to Mardia and Jupp [23] is given in [3](4.2) as

$\hat{\kappa}\approx
d\bar{r}\left(1+\frac{d}{d+2}\bar{r}^{2}+\frac{d^{2}(d+8)}{(d+2)^{2}(d+4)}\bar{r}^{4}\right)~{}.$

For large values of $d$, this is almost the same as

$\hat{\kappa}\approx d\bar{r}\left(1+\bar{r}^{2}+\bar{r}^{4}\right)~{}.$ (23)

  2. 2.

Banerjee _et al_. [3] derive an estimate (unnumbered, above [3] (4.4)),

$\hat{\kappa}\approx\frac{d\bar{r}}{1-\bar{r}^{2}}$ (24)

which has series expansion

$\hat{\kappa}\approx d\bar{r}(1+\bar{r}^{2}+\bar{r}^{4}+\ldots)~{}.$ (25)

Since $0\leq\bar{r}<1$, this series will converge, albeit slowly for $\bar{r}$
close to $1$. Thus, it is seen that Eq. 23 is simply a truncated approximation
to the infinite series Eq. 25. We shall refer to Eq. 23 (perhaps somewhat
inexactly) as the “order-$2$” approximation to Eq. 25, since the approximation
to $1/(1-\bar{r}^{2})$ contains terms up to second order in $\bar{r}^{2}$.

  3. 3.

It is also possible to consider approximations of other orders for Eq. 24,
including in particular the $0$-order approximation

$\hat{\kappa}\approx d\bar{r}~{},$ (26)

first order approximation

$\hat{\kappa}\approx d\bar{r}\left(1+\bar{r}^{2}\right)~{},$ (27)

and the third-order approximation.

  4. 4.

Another empirically derived formula is also given in [3](4.4). However, we
observe (see Fig. 7) that the approximation Eq. 24 is already a very close
approximation, and the use of [3](4.4) is not warranted.

We show graphs of the approximations of $\hat{\kappa}$ for various
approximation, and the exact solution in Fig. 7.

##### Results

As pointed out in the caption to Fig. 7, the exact estimate of $\hat{\kappa}$
may not be a good choice for clustering, in the case where $\bar{r}$
approaches $1$ (meaning that the data has small spread).

In Table 7, we try all the different formulas described above to estimate a
value of $\hat{\kappa}$. We also report the results in the main paper where
$\kappa$ is kept constant. The experiments show the following outcomes.

  1. 1.

Order-$\infty$ in Eq. 24 performs notably worse, presumably because of the
sensitivity to the value of $\bar{r}$, which is computed from a relatively
small number of samples in our few-shot learning scenario.

  2. 2.

The order-$1$ to order-$3$ estimates perform approximately the same as fixing
$\hat{\kappa}$.

  3. 3.

The order-$0$ approximation, $\hat{\kappa}=d\bar{r}$, gives the best results.
In our COL experiments, $d=512$, so setting $\hat{\kappa}=d\bar{r}$ places an
upper bound of $512$ on the value of $\hat{\kappa}$.

Table 7: CorLoc(%) and mAP(%) results with $\kappa$ estimations for the task
of COL on novel object classes on the COCO60 dataset with support set size
$K=5$ and $K=10$.

$\hat{\kappa}$ | K = 5 | K=10
---|---|---
<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
Constant | $34.8$ | $18.6$ | $36.9$ | $20.0$
$d\bar{r}(1+\bar{r}^{2}+\bar{r}^{4}+\ldots)$ | $20.1$ | $11.3$ | $24.6$ | $13.7$
$d\bar{r}(1+\bar{r}^{2}+\bar{r}^{4}+\bar{r}^{6})$ | $31.8$ | $17.7$ | $34.7$ | $19.0$
$d\bar{r}(1+\bar{r}^{2}+\bar{r}^{4})$ | $33.1$ | $18.6$ | $36.3$ | $19.5$
$d\bar{r}(1+\bar{r}^{2})$ | $34.7$ | $19.0$ | $37.5$ | $19.9$
$d\bar{r}$ | ${\bf 35.7}$ | ${\bf 19.6}$ | ${\bf 38.2}$ | ${\bf 20.2}$

##  Appendix B Modeling with Gaussian Distribution

In Section 4.4, we conduct experiment with Gaussian distribution used to model
the common object distribution. We assume the common object distribution is
Gaussian with mean ${\bm{\theta}}$ and diagonal covariance matrix
$\sigma^{2}I$, i.e., $\mathbf{x}\sim\mathcal{N}({\bm{\theta}},\sigma I)$

$p^{+}_{{\bm{\theta}}_{c}}(\mathbf{x})=\frac{1}{Z}\exp\left(-\frac{\|\mathbf{x}-{\bm{\theta}}\|^{2}}{2\sigma^{2}}\right)~{},$
(28)

and plug the distribution into Eq. 12 to get the soft label update rule as

$\mathbf{w}_{ik}=\frac{\exp\Big{(}-\frac{\|F_{ik}-{\bm{\theta}}\|^{2}}{2\sigma^{2}}-\log
u_{{\bm{\omega}}}^{-}(F_{ik})\Big{)}}{\sum_{j=1}^{P}\exp\Big{(}-\frac{\|F_{ij}-{\bm{\theta}}\|^{2}}{2\sigma^{2}}-\log
u_{{\bm{\omega}}}^{-}(F_{ij})\Big{)}}~{}.$ (29)

E-step is computed by setting the derivative of Eq. 14 w.r.t. ${\bm{\theta}}$
to zero

${\bm{\theta}}\leftarrow\frac{1}{M}\sum_{i=1}^{M}\sum_{k=1}^{P}\mathbf{w}_{ik}F_{ik}=\frac{1}{M}\sum_{i=1}^{M}\mathbf{w}_{i}^{\top}F_{i}~{},$
(30)

##  Appendix C MI-SVM WSOD Baseline

To the best of our knowledge there is no WSOD algorithm for few-shot setting
in the literature. However, WSOD with knowledge transfer methods [28, 37, 12,
5] are closely related to our work. In this section, we describe a slightly
modified version of [37] and discuss its differences to the proposed method.
In Section 4, we empirically compare our work against [37].

Similar to our approach, each image is represented as a bag of bounding box
proposals $B$ and their features $F$. Learning is performed on one target
class $c\in\mathcal{L}$ at a time. Similar to our WSOD approach, the support
set is split into positive images which has the target class and a negative
set of images without the target class. Then, a linear SVM appearance model is
employed to iteratively learn class $c$ by alternating between two steps:

  * •

Re-training: Train a binary SVM given the currently selected proposals from
the positive images and the proposals in negative images.

  * •

Re-localization: Given the current SVM select the proposal with the highest
score from each positive image. In [37], the re-localization is guided by a
class-agnostic objectness measure to guide the selection toward objects.
Therefore, the selection for a positive image $\mathbf{I}$ with bounding box
proposals $B$ is updated as

$b^{*}=\operatorname*{arg\,max}_{b\in B}\quad\text{SVM}(\mathbf{I},b)+\gamma
O(\mathbf{I},b)~{},$ (31)

where $O$ is the objectness model and $\gamma$ adjusts its importance.

The algorithm is initialized with complete image bounding box proposal and
alternates between above steps until convergence. We use highly efficient GPU
solver in [1] for SVM optimization. Finally, test proposal $\mathbf{x}$ from
the test set $\mathcal{D}_{\rm test}$ is scored using the SVM trained for each
class. To have a fair comparison, we use the same Faster-RCNN model trained on
the base classes as we used in our model to extract bounding box and feature
proposals from all images. For the objectness model $O$, we learn a class-
agnostic logistic regression model on the extracted feature. We employ the
hard negative mining in [37] to improve the performance of the classifier. In
this approach the negative set is initialized with full negative image
features and the hardest negative proposal within each image is added to the
negative set after each re-training step.

The expectation and maximization steps in our method are analogous to re-
localization and re-training steps in [37]. In MI-SVM, only the proposal with
the highest score is labeled positive in the re-localization step while our
COL method infers soft labels in the expectation step via an attention
mechanism. Using soft labels could be beneficial as they reflect the
uncertainty in choosing the common object.

##  Appendix D Qualitative Results

We show some of the success cases of the experiments in the paper. Our first
example in Fig. 8 shows vMF-MIL performance on a single few-shot WSOD problem.
Given the support set with only image-level annotations the algorithm learns
to detect the target objects in the query set. We sample $4$ query images to
evaluate the algorithm performance in detecting different target objects.
Except person in the first query image, vMF-MIL successfully detects other
target objects. More few-shot WSOD tasks are shown in Fig. 9, 10, 11, 12 and
13.

Figure 8:  Few-shot WSOD on PASCAL VOC with $N=K=5$. Given the support set
shown on the left side the algorithm detects the object on the $4$ different
query images on the right side. The algorithm fails to detect person in the
first query image but successfully detects other target objects.

Finally, Fig. 14 shows some of the success cases in localizing the target
object in the query image for the task of common object localization in
Section 4.1. All the target objects (dog, car, cow, train, boat, bus, sofa,
horse, person) shown in this figure are novel. Also, ground-truth annotations
are only shown for better visualization and are not used in learning.

Figure 9:  Few-shot WSOD on PASCAL VOC with $N=K=5$. Given the support set
shown on the left side the algorithm detects the object in query images on the
right side. Figure 10:  Few-shot WSOD on PASCAL VOC with $N=K=5$. Given the
support set shown on the left side the algorithm detects the object in query
images on the right side. Figure 11:  Few-shot WSOD on PASCAL VOC with
$N=K=5$. Given the support set shown on the left side the algorithm detects
the object in query images on the right side. Figure 12:  Few-shot WSOD on
PASCAL VOC with $N=K=5$. Given the support set shown on the left side the
algorithm detects the object in query images on the right side. Figure 13:
Few-shot WSOD on PASCAL VOC with $N=K=5$. Given the support set shown on the
left side the algorithm detects the object in query images on the right side.
Figure 14:  5-shot common object localization ($N=1$) on MS COCO. Each row
shows one common object localization problem. Ground-truth annotations (shown
in green) are just for visualization and are not used in the algorithm. Top
query bounding box prediction for each problem is shown in pink.


SF2A 2023

# Challenges and limitations of future exoplanet space imagers

L. Leboulleux  Univ. Grenoble Alpes, CNRS, IPAG, 38000 Grenoble, France

###### Abstract

The James Webb Space Telescope was not even launched yet when the Astro2020
Decadal Survey American report recommended the development of what is now
called the Habitable World Observatory, also mentioned by the Voyage 2050
European report. This future space telescope, at $11$ billions dollars and at
least $6$ m diameter, should allow, around $2040$, the characterization of at
least 25 exoplanets similar to Earth and orbiting around main sequence stars,
with the hope of discovering one where life could have developed. This
objective represents a technological challenge since it requires the design of
spectro-imagers able to access very high contrasts ($10^{-8}-10^{-10}$) at low
angular separations (smaller than $100$ mas). This proceeding and the talk it
is associated to address various obstacles that remain to be overcome in order
to one day allow HabWorld to reach its ultimate performance.

###### keywords:

exoplanets, Habitable World Observatory, HabWorld, instrumentation, high-
contrast imaging

##  1 Introduction

Every ten years, the National Academies of Sciences, Engineering, and Medicine
(NASEM) publish recommendations for large science projects in various fields.
In space astronomy for instance, the Chandra-X ray Observatory and the Hubble
Space Telescope followed the ”Astronomy and Astrophysics for the 1980s” NASEM
report, the Spitzer Space Telescope issued from the 1991 ”The Decade of
Discovery in Astronomy and Astrophysics” report, and more recently, the Next
Generation Space Telescope, now known as the James Webb Space Telescope
(JWST), was recommended by the 2001 ”Astronomy and Astrophysics in the New
Millennium” report (Gardner et al. 2023). In 2010, the ”New Worlds, New
Horizons” NASEM report recommended to push forward the research in dark energy
with the so-called Nancy Grace Roman Space Telescope (RST), a telescope that
will be launched around $2028$. Its science goal was extended to the field of
exoplanet imaging and characterization, with the first detection of exoplanets
in reflected light. It is now also equiped with a coronagraphic instrument
able to detect and characterize planets down to $200-300$ mas from their host
star and with contrasts down to $10^{-8}-10^{-9}$ ($R\sim 50$ at $700$ nm)
(Bailey et al. 2023). In 2021, a new decadal report was released: the
”Pathways to Discovery in Astronomy and Astrophysics for the 2020s”
recommended the development of the so-called Habitable World Observatory
(HabWorlds). This new large space mission should be able to characterize $25$
Earth-like planets with separations of $20-50$ mas, contrasts down to
$10^{-10}$ at mid to high spectral resolution ($200-5000$ at $500$ nm)
(Vaughan et al. 2023; Roberge et al. 2021; Gaudi et al. 2021, 2020).

With these two future space missions, the scientific requirements reach values
that have never been accessed before, from space and from the ground. As a
comparison, they imply a gain in contrast of a factor $10^{5}$, and a gain in
angular separation of a factor $20$ compared to the JWST constraints. In
addition to the difficulty to access such extreme performance in perfect
conditions, it is highly deteriorated as soon as conditions are non perfect.
As an example, one of the key phases in the JWST launch was the segment
phasing of its primary mirror: $18$ hexagonal segments had to be aligned from
a few microns to $30$ nm RMS of aberration, a complex operation that took one
month to be completed. If HabWorlds, whose primary mirror will probably also
be segmented, is aligned down to $30$ nm RMS, its contrast will be maintained
over $10^{-6}$, far from its requirements. Compared to the JWST or other
telescopes, the technological breakthrough must concern both the absolute
performance and its robustness to realistic conditions.

This proceeding is dedicated to describing methods and tools that are
developed or under development to access high contrasts at small angular
separations (section 2) and, despite non perfect conditions (section 3),
maintain such a performance over time and observation conditions (section 4).

##  2 Performance: coronagraphy and wavefront control for (non) dummies

The access to high contrasts in exoplanet imaging or spectroscopy requires the
assembly of multiple components, set up in pupil planes, focal planes, or out
of pupil planes. The general principle is illustrated in Fig. 1 and is made of
a coronagraph, which can be combined with a wavefront control system, with the
ultimate objective of removing the starlight in the area of interest of the
detector plane.

Figure 1: Scheme of a typical high-contrast imaging system. Purple: components
associated with coronagraphy, Green: components associated with wavefront
control. DM stands for Deformable Mirror and FPM for Focal Plane Mask.

The coronagraph is made of up to three passive components :

$\bullet$ An apodizer (pupil plane) that reshapes the wavefront to modulates
its intensity in the focal plane,

$\bullet$ A Focal-Plane Mask (FPM) that blocks the starlight along the optical
axis,

$\bullet$ A Lyot stop (pupil plane) to remove the high-order star signal, not
stopped by the FPM.

The apodizer and the FPM, if they exist, can be amplitude or phase masks, and
a large variety of combinations are possible depending on the type of masks.
Fig. 2 (left) introduces a few of these combinations, with no mention of the
existence of a Lyot stop. For a more exhaustive list, please refer to Ruane et
al. (2018). Fig. 2 (right) also illustrates this diversity with three cases:
1) a shaped pupil (amplitude apodizer, no FPM, no Lyot stop, see also Kasdin
et al. (2003); Carlotti et al. (2011)) optimized for a 18 segment pupil to dig
a circular $1\lambda/D$-large dark zone localized at $8.3\lambda/D$, 2) an
apodized pupil Lyot coronagraph (amplitude apodizer, FPM, and Lyot stop, see
also Aime et al. (2002); Soummer et al. (2006); N’Diaye et al. (2016)) to
access high contrasts from $3$ to $10\lambda/D$, and 3) an apodized vortex
coronagraph (amplitude apodizer, phase FPM, and amplitude Lyot stop, see also
Carlotti (2013); Fogarty et al. (2017)) to access high contrasts from $3$ to
$10\lambda/D$.

Figure 2: (left) Non exhaustive list of coronagraphs classified by their types
of focal plane and pupil plane masks. (right) Examples of coronagraphs and
their associated coronagraphic images (no aberration added). The shaped pupil
is taken from Leboulleux et al. (2022d) and the apodized pupil Lyot
coronagraph and the apodized vortex coronagraph from Mazoyer et al. (2018a);
N’Diaye et al. (2015); Fogarty et al. (2017). The focal-plane images are in
logarithmic scales.

In addition of a coronagraph, a wavefront control system enables to reach
higher contrasts at small angular separations by reshaping the wavefront with
one or two Deformable Mirrors (DM). One DM is set in a pupil plane for phase
control, but with one single DM, only half of the detector interest zone can
be controled. A second DM can be installed out of pupil plane to access
amplitude control and a symetric dark area. Due to the limited number of
actuators, phase and amplitude are controlled with a small spatial sampling,
which makes this tool usable only for small dark zones at close angular
separations: with a $34$-actuators across the diameter DM, the dark hole
cannot go further than $17\lambda/D$. More details, with the descriptions and
references of the diverse methods, can be found in Jovanovic et al. (2018)
(see also figure 4 for illustration).

##  3 Limitations

Coronagraphs, no matter their ultimate performance, are highly sensitive to
unperfect conditions. Fig. 3 illustrates a few of them for space operations,
in the case of a shaped pupil compatible with a $18$ segment pupil designed to
access an ultimate contrast of $7\times 10^{-9}$ between $6$ and
$13\lambda/D$:

$\bullet$ When the pupil is complexified, for instance here with spiders added
on the architecture: this is the reason why HabWorlds is considered off-axis,
to avoid a complex pupil with spider and central obstruction.

$\bullet$ With global tip-tilt misalignments (or pointing error or jitter):
they are particularly problematic in case of a FPM since the starlight leaks
off the FPM at small angular separations.

$\bullet$ With phasing errors: on HabWorlds, a contrast of $10^{-10}$ imposes
the phasing errors to remain below $10$pm rms, which is not accessible yet in
labs.

$\bullet$ With a mask shift or misalignment.

Figure 3: Examples of aberrations and their impact on the coronagraphic image
with a 18-segment pupil combined with a shaped pupil. From left to right: no
aberration, spiders added on the pupil, tip or jitter (a FPM has been added in
the coronagraphic system), segment phasing errors, apodizer vertical shift.

The high-contrast and high-angular separation performance can then not be
accessed without a robust aberration mitigation. This mitigation can take two
forms:

$\bullet$ Cure: diagnosing the sources of errors (wavefront sensing or WFS)
and correcting them (wavefront sensing or WFC) with adaptive components,

$\bullet$ Prevention: optimizing the instrument to make it insensitive to
errors.

##  4 Robustness: prevention or cure?

###  4.1 Cure

Curing relies on two steps:

$\bullet$ a diagnosis of the aberrations or WFS. Reconstructing the errors,
static or dynamic, is complex and several methodologies and tools have been
developed so far. They are not described in the proceeding and for more
information please refer to Jovanovic et al. (2018).

$\bullet$ a correction of the aberrations or WFC. Several methods have also
been developed, all of them requiring the use of adaptive components (Snik et
al. 2018).

As phase control tools, DMs (Blanco et al. 2022; Bowens-Rubin et al. 2021)
have been used since the beginning of adaptive optics (Chauvin 2018) and can
enable to compensate for the aberrations and recover the performance. One of
its latest application is the combination of two DMs and its use through the
Active Correction of Aperture Discontinuities to compensate for both aperture
complexity (spiders) and phase errors (segment phasing misalignements) on a
space coronagraphic system (Mazoyer et al. 2018a, b). If DMs have so far never
been used in space (except on specific experiments on balloon missions, see
Morgan et al. (2019)), the Nancy Grace Roman Space Telescope will be the first
space telescope equipped of two deformable mirrors (Mennesson et al. 2022). As
an alternative for phase control, one can cite the spatial-light modulator
(SLM) technology (Kühn & Patapis 2022; Kühn et al. 2018) which can be used,
between other applications, as a continuous and low- to high-order
programmable pupil phase mirror, despite being limited by its requirement to
reflect linear polarized light. More recently, investigations have been
ongoing to identify adaptive amplitude components. The use of Digital Micro-
mirror Devices (DMD) is currently under study: the component tested at IPAG,
for instance, is built as a matrix of small square mirrors that can be
controlled in tip-tilt and flipped between two positions, one of them used as
a transmissive direction towards the detector, the other one rejecting the
light out of the optical system. It can then be used to get programmable
binary apodizations (Carlotti et al. 2023; Leboulleux et al. 2022a; Carlotti
et al. 2018).

Fig. 4 illustrates the use of 1) two DMs to mitigate the impact of phasing
errors in the coronagraph dark region, 2) one DMD to recover the contrast in
the coronagraph dark region despite missing segments.

Figure 4: Examples of aberration mitigation with adaptive components: 1) use
of two deformable mirrors to recover the dark hole despite segment phasing
errors with the ACAD-OSM methodology (simulation from Mazoyer et al. (2018a)),
2) use of a DMD to recover the dark hole despite missing segments
(experimental image from Carlotti et al. (2023)).

###  4.2 Prevention

An alternative approach is to create a high-contrast system that is passively
blind to or robust to aberrations by design, meaning optimized to remain
performant (high contrast at small angular separations) despite a range of
aberrations. As aberrations have a high impact on the performance, the
robustness is a key aspect of the design study. Different methods have been
proposed so far to increase the system robustness, combined with different
coronagraphs: to reduce the impact of low-order aberrations on vortex
coronagraphs (Fogarty et al. 2020), scalar vortex coronagraphs (Desai et al.
2022), or apodized pupil Lyot coronagraphs (N’Diaye et al. 2015), or Lyot stop
shifts on apodized pupil Lyot coronagraphs (Nickson et al. 2022).

Let’s focus on one example only, with a fully analytical construction:
Leboulleux et al. (2018); Laginja et al. (2021) introduce the Pair-based
Analytical model for Segmented Telescopes Imaging from Space (PASTIS) to
evaluate the impact of phasing errors on the coronagraphic PSF and
performance. In the PASTIS formula, a specific factor appears, as a weight
factor of all other factors: the PSF of one segment only. If this weight
factor, so the PSF of the segment, increases, then the contrast is easily
deteriorated by segment-level aberrations. On the opposite, if the PSF of the
segment is minimal, then the contrast is less impacted by any segment-level
aberration. This assessment led to the Redundant Apodized Pupil (RAP) concept
Leboulleux et al. (2022b, c): optimizing the segment and multiplying this
apodization to mimic the pupil segmentation enable to obtain a high-contrast
region that remains stable over a large range of segment-level aberrations,
like segment phasing errors. Fig. 5 illustrates this concept and its
robustness for a $10^{-6}$-contrast dark zone between $5.5$ and $12\lambda/D$.

Figure 5: Robustness of RAP design for segment-phasing errors on a Giant
Magellan Telescope-like pupil (7 circular segments): (left) two apodizing
phase plate maps, with $\pi$-shift between black and white areas, the top
apodizer is classically optimized, the bottom one is redundantly optimized;
(right) associated coronagraphic PSFs with increasing segment phasing errors
(figure from Leboulleux et al. (2022b)).

##  5 Conclusions

The field of high-contrast imaging is the environment of many ongoing studies,
that could not be all covered in this proceeding. The list of scientific
requirements, with contrast and angular separations, is here far from
exhaustive, since the planet photon flux, spectroscopy, polarimetry, or
compatibility to post-processing methods were not even mentioned so far. Even
with these requirements left on the side, the future scientific and
technological ambitions for the Nancy Grace Roman Space Telescope and the
Habitable Worlds Observatory demand multiple and complex instrumentation
developments and research. Between others (Laginja et al. 2022; Mazoyer et al.
2019), in France, one can mention the development of several highly performant
high-contrast testbeds, such as the Très Haute Dynamique one (THD2, Potier et
al. (2019)), the Segmented Pupil Experiment for Exoplanet Detection (SPEED,
Martinez et al. (2023)), the Marseille Imaging Testbed for High Contrast
(MITHiC, El Morsy et al. (2022)), or the High Contrast Module testbed (HCM,
Hours et al. (2022)).

These developments are necessary to overcome the limitations of current
instruments and access rocky exoplanets. In space, the most efficient
coronagraphs can currently not go deeper than contrasts of $\sim 10^{-5}$ at
several hundreds of mas, and DMs have never been tested in space telescope,
except for balloon missions. The Nancy Grace Roman Space Telescope and the
Habitable Worlds Observatory are then setting ambitious objectives both in
terms of science targets but also for first technological demonstrations.

###### Acknowledgements.

The author thanks the SF2A organization committees (SOC and LOC) who made this
conference a large success. She is also grateful for the help and references
provided by colleagues, including Marie Ygouf, Elodie Choquet, Johan Mazoyer,
Mamadou N’Diaye, Pierre Baudoz, and Alexis Carlotti.

## References

  * Aime et al. (2002) Aime, C., Soummer, R., & Ferrari, A. 2002, A&A, 389, 334
  * Bailey et al. (2023) Bailey, V. P., Bendek, E., Monacelli, B., et al. 2023, arXiv e-prints, arXiv:2309.08672
  * Blanco et al. (2022) Blanco, L., Behara, N., Valenzuela, J., & Kasper, M. 2022, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 12185, Adaptive Optics Systems VIII, ed. L. Schreiber, D. Schmidt, & E. Vernet, 121856B
  * Bowens-Rubin et al. (2021) Bowens-Rubin, R., Hinz, P., Kuiper, S., & Dillon, D. 2021, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 11823, Techniques and Instrumentation for Detection of Exoplanets X, ed. S. B. Shaklan & G. J. Ruane, 118231R
  * Carlotti (2013) Carlotti, A. 2013, A&A, 551, A10
  * Carlotti et al. (2023) Carlotti, A., Baccar, S., Leboulleux, L., et al. 2023, in Adaptive Optics for Extremely Large Telescopes VII (AO4ELT7)
  * Carlotti et al. (2018) Carlotti, A., Mouillet, D., Correia, J.-J., et al. 2018, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10706, Advances in Optical and Mechanical Technologies for Telescopes and Instrumentation III, ed. R. Navarro & R. Geyl, 107062M
  * Carlotti et al. (2011) Carlotti, A., Vanderbei, R., & Kasdin, N. J. 2011, Optics Express, 19, 26796
  * Chauvin (2018) Chauvin, G. 2018, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10703, Adaptive Optics Systems VI, ed. L. M. Close, L. Schreiber, & D. Schmidt, 1070305
  * Desai et al. (2022) Desai, N., Llop-Sayson, J., Bertrou-Cantou, A., et al. 2022, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 12180, Space Telescopes and Instrumentation 2022: Optical, Infrared, and Millimeter Wave, ed. L. E. Coyle, S. Matsuura, & M. D. Perrin, 121805H
  * El Morsy et al. (2022) El Morsy, M., Vigan, A., Lopez, M., et al. 2022, A&A, 667, A171
  * Fogarty et al. (2020) Fogarty, K., Mawet, D., Mazoyer, J., et al. 2020, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 11443, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, 114433Y
  * Fogarty et al. (2017) Fogarty, K., Pueyo, L., Mazoyer, J., & N’Diaye, M. 2017, AJ, 154, 240
  * Gardner et al. (2023) Gardner, J. P., Mather, J. C., Abbott, R., et al. 2023, PASP, 135, 068001
  * Gaudi et al. (2021) Gaudi, B., Roberge, A., Habex Team, & Luvoir Study Team. 2021, in Bulletin of the American Astronomical Society, Vol. 53, 0301
  * Gaudi et al. (2020) Gaudi, B. S., Seager, S., Mennesson, B., et al. 2020, arXiv e-prints, arXiv:2001.06683
  * Hours et al. (2022) Hours, A., Carlotti, A., Mouillet, D., et al. 2022, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 12185, Adaptive Optics Systems VIII, ed. L. Schreiber, D. Schmidt, & E. Vernet, 121852E
  * Jovanovic et al. (2018) Jovanovic, N., Absil, O., Baudoz, P., et al. 2018, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10703, Adaptive Optics Systems VI, ed. L. M. Close, L. Schreiber, & D. Schmidt, 107031U
  * Kasdin et al. (2003) Kasdin, N. J., Vanderbei, R. J., Spergel, D. N., & Littman, M. G. 2003, ApJ, 582, 1147
  * Kühn et al. (2018) Kühn, J., Patapis, P., Lu, X., & Arikan, M. 2018, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10706, Advances in Optical and Mechanical Technologies for Telescopes and Instrumentation III, ed. R. Navarro & R. Geyl, 107062N
  * Kühn & Patapis (2022) Kühn, J. G. & Patapis, P. 2022, Appl. Opt., 61, 9000
  * Laginja et al. (2022) Laginja, I., Robles, P., Barjot, K., et al. 2022, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 12185, Adaptive Optics Systems VIII, ed. L. Schreiber, D. Schmidt, & E. Vernet, 121853A
  * Laginja et al. (2021) Laginja, I., Soummer, R., Mugnier, L. M., et al. 2021, Journal of Astronomical Telescopes, Instruments, and Systems, 7, 015004
  * Leboulleux et al. (2022a) Leboulleux, L., Carlotti, A., Curaba, S., et al. 2022a, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 12188, Advances in Optical and Mechanical Technologies for Telescopes and Instrumentation, 121884J
  * Leboulleux et al. (2022b) Leboulleux, L., Carlotti, A., & N’Diaye, M. 2022b, A&A, 659, A143
  * Leboulleux et al. (2022c) Leboulleux, L., Carlotti, A., N’Diaye, M., et al. 2022c, A&A, 666, A91
  * Leboulleux et al. (2022d) Leboulleux, L., Carlotti, A., N’Diaye, M., et al. 2022d, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 12188, Advances in Optical and Mechanical Technologies for Telescopes and Instrumentation, 121881S
  * Leboulleux et al. (2018) Leboulleux, L., Sauvage, J.-F., Pueyo, L. A., et al. 2018, Journal of Astronomical Telescopes, Instruments, and Systems, 4, 035002
  * Martinez et al. (2023) Martinez, P., Beaulieu, M., Gouvret, C., Spang, A., & Marcotto, A. 2023, The Messenger, 190, 55
  * Mazoyer et al. (2019) Mazoyer, J., Baudoz, P., Belikov, R., et al. 2019, in Bulletin of the American Astronomical Society, Vol. 51, 101
  * Mazoyer et al. (2018a) Mazoyer, J., Pueyo, L., N’Diaye, M., et al. 2018a, AJ, 155, 7
  * Mazoyer et al. (2018b) Mazoyer, J., Pueyo, L., N’Diaye, M., et al. 2018b, AJ, 155, 8
  * Mennesson et al. (2022) Mennesson, B., Bailey, V. P., Zellem, R., et al. 2022, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 12180, Space Telescopes and Instrumentation 2022: Optical, Infrared, and Millimeter Wave, ed. L. E. Coyle, S. Matsuura, & M. D. Perrin, 121801W
  * Morgan et al. (2019) Morgan, R., Douglas, E., Allan, G. W., et al. 2019, Micromachines, 10, 366
  * N’Diaye et al. (2015) N’Diaye, M., Pueyo, L., & Soummer, R. 2015, ApJ, 799, 225
  * N’Diaye et al. (2016) N’Diaye, M., Soummer, R., Pueyo, L., et al. 2016, ApJ, 818, 163
  * Nickson et al. (2022) Nickson, B. F., Por, E. H., Nguyen, M. M., et al. 2022, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 12180, Space Telescopes and Instrumentation 2022: Optical, Infrared, and Millimeter Wave, ed. L. E. Coyle, S. Matsuura, & M. D. Perrin, 121805K
  * Potier et al. (2019) Potier, A., Baudoz, P., Galicher, R., Huby, E., & Singh, G. 2019, arXiv e-prints, arXiv:1910.09064
  * Roberge et al. (2021) Roberge, A., Fischer, D., Peterson, B., et al. 2021, in Bulletin of the American Astronomical Society, Vol. 53, 332
  * Ruane et al. (2018) Ruane, G., Riggs, A., Mazoyer, J., et al. 2018, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10698, Space Telescopes and Instrumentation 2018: Optical, Infrared, and Millimeter Wave, ed. M. Lystrup, H. A. MacEwen, G. G. Fazio, N. Batalha, N. Siegler, & E. C. Tong, 106982S
  * Snik et al. (2018) Snik, F., Absil, O., Baudoz, P., et al. 2018, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10706, Advances in Optical and Mechanical Technologies for Telescopes and Instrumentation III, ed. R. Navarro & R. Geyl, 107062L
  * Soummer et al. (2006) Soummer, R., Aime, C., Ferrari, A., et al. 2006, in IAU Colloq. 200: Direct Imaging of Exoplanets: Science & Techniques, ed. C. Aime & F. Vakili, 367–372
  * Vaughan et al. (2023) Vaughan, S. R., Gebhard, T. D., Bott, K., et al. 2023, MNRAS, 524, 5477