Whoever looks at inclusive data is met with clear indications of resonance decay contributing to the measured observables. This is put in this paper to a quantitative test by introducing a group of eight mesonic and five baryonic resonances whose production characteristics are sufficiently well known in order to study the impact of their decay on the establishment of the single-inclusive quantities studied in the first parts of this paper. The resonance masses range from $\eta^0$(550) to $f_4$(2050) for the mesons and from $\Delta$(1232) to $N^*$(1680) for the baryons. In order to avoid double counting only two-body decays are included.
As resonance production is relatively well measured in the energy range of the CERN SPS the study is carried out at $\sqrt{s}$ = 17.2 GeV corresponding to the NA49 experiment.

As a first result it turns out that 77% of all inclusive $\pi^-$ are accounted for by resonance two or three-body ($\eta$ and $\omega$) decays – which is a lower limit as the list of resonances is incomplete. This result is not new as the importance of resonance decay has been claimed decades ago. It is new to the point that both mesonic and baryonic resonances are involved and their decays traced in parallel.

In a first step the basic kinematic features of resonance decay are pointed out in particular concerning the resonance transverse momentum and – for strong decays – the resonance mass distribution. For this aim the well-measured $\Delta^{++}$(1232) is used to also bring out the effect of asymmetric masses.

In a second step the decay contribution to most of the inclusive distributions presented in this paper are quantified.

On the level of double-differential cross sections a surprisingly exact reproduction of the essential features of the $p_T$ and $x_F$ distributions is achieved. The deviations of up to 20% from the data are studied in their $p_T$ and $x_F$ dependence and brought into relation to further expected decay features like cascading decays.

Concerning single-differential, integrated quantities it may be stated that their features are reproduced with surprising precision.This is true both for mean $p_T$ and its $x_F$ dependence and for $dn/dx_F$ as a function of $x_F$.

The study is extended to $p_T$ values up to 3 GeV/c at central rapidity. It is shown that resonance decay reproduces the measured cross sections also in this sector which has been ascribed to hard parton scattering as an agreed consensus invoking perturbative QCD over the past decades.

As a general consequence it becomes clear that the term of "soft hadronic physics" must be redefined as far as the application of parton dynamics and "perturbative QCD" is concerned.

§ FUTURE STUDIES: EXPERIMENTAL AND ACCELERATOR CONSTRAINTS

In this paper a considerable effort has been made to bring into perspective the experimental and phenomenological information concerning one single final state particle in soft proton-proton interactions and this by covering the full available range of collision energy, the complete production phase space and, of course, particle identification.

Looking back at this effort, the authors have to confess to a feeling of frustration. Not only would this study have to be repeated for the full range of secondary particles and types of collision concerning different hadron beams on both protons and neutron targets. In addition the nuclear sector should not be disregarded as it presents a vast laboratory for the study of multiple hadronic collisions. The actual status of the field gives rise to a number of questions:

* How is it possible that non-perturbative QCD, as the by far biggest sector of the standard model, has been more or less completely abandoned by experiment as well as theory?

* How is it possible that in spite of the non-deniable progress made both in detector and in accelerator technology there are still experimental results produced which do not come up in quality and precision to work done five or six decades ago?

* How is it possible that the problem of parton dynamics and its purported contribution to final state inclusive results is still treated as a non-touchable certainty where experimental results, when obtained with proper precision and reliability, point just to the opposite?

* How is it possible that – extending the range of hadronic interactions to the nuclear sector – a possible phase transition to a QCD plasma state is considered as existing when no indisputable experimental proof has ever been put forward?

* Why no real attempts have been made to have a look beyond the purely inclusive level in order to open up another source of quantitative understanding underlying the observed phenomena on the single inclusive level?

For the time being it is apparent that it is on the experimental level that further progress is relying as the actual ideas based on statistical thermodynamics on the one hand and parton dynamics on the other hand seem to be so deep rooted that no real movement towards putting the one or the other to a decisive test are visible. This in a situation where progress on the experimental side should allow a decisive improvement if certain constraints both concerning detector design and construction as well as accelerator systems are properly taken into account. Part of these constraints have hopefully become apparent in the present paper.

* Phase space coverage

New detector systems should be designed such that the full production phase space is covered uniformly from the far backward to the far forward direction with precision tracking. It has been a mistake to concentrate since decades more or less exclusively on the very central part of the collision.

* Particle identification

Particle identification should cover the full phase space including calorimetry for photon/electron and neutral hadron detection.

* Systematic uncertainties

The systematic uncertainties should be on the percent level. This is the real problem – as has become apparent in the above comparison of the "classic" experiments using bubble chambers with more or less all "modern" approaches using different kinds of spectrometer layouts or so-called "detector facilities".

* Versatility of the strong interaction

The tremendous versatility of the strong interaction in terms of initial state conditions defining the collision itself as well as the choice of secondary particles should be fully exploited.

* Order of approach

Research should start with the so-called "elementary" collisions using the full spectrum of possible beam particles on proton targets. This should also contain lepton beams which are sometimes called "noble probes" as compared to "dirty hadrons". A definite high precision comparison is still lacking.

* The nuclear sector

This sector should be worked up successively from light to heavy nuclei. Here the deuteron offers a (non-trivial) "neutron" target. Again all kinds of projectile hadrons should be used.

* Impact parameter

Hadron-nucleus collision should be studied with a precise impact parameter measurement defining the number of intranuclear projectile collisions which gives access to a new sector of hadronic physics.

* For all the above points, full attention should be given to the precision measurement of hadronic resonances.

* Nucleus-nucleus collisions

The interest here should not be exclusively concentrated on "Heavy Ion" collisions. Full attention should also be paid to peripheral interactions.

This list is not aimed at "discovery". In the past too much attention has been paid to the perturbative sector concentrating the attention upon more and more remote sectors hence necessitating extreme accelerator luminosities whereas the soft regime is accessible with convenient interaction rates.

From the above (non-complete) list of experimental constraints and tasks several consequences follow immediately:

* Proton colliders cannot fulfil the wide spectrum of possibilities, neither the homogeneous coverage of production phase space nor the versatility of projectile/target combinations.

* Fixed-target operation is on the contrary well suited to cover the complete range of tasks. For a first generation of experiments, the interaction energy may be kept conveniently low in order to cope with high precision tracking and the necessities of complete particle identification.

In view of this the CERN SPS complex offers an available, at present under-exploited, environment.

[20]
[1] R. P. Feynman, Phys. Rev. Lett. 23 (1969) 1415
[2] G. Neuhofer et al., Phys. Lett. 37B (1971) 438
[3] M. Breidenbach et al., Phys. Lett. 39B (1972) 654
[4] G. Barbiellini et al., Phys. Lett. 39B (1972) 663
[5] G. Barr et al., Eur. Phys. J. C49 (2007) 919
[6] G. Alberi and G. Goggi, Phys. Rep. 74 (1981) 1
[7] L. Stodolsky, Mod. Phys. Lett. A32 (2017) 1730028
[8] E. R. Gellert, LBL-749 (1972)
[9] V. Blobel et al., Nuc. Phys. B69 (1974) 454
[10] D. B. Smith, UCRL-20632 (1971)
[11] A. N. Diddens and K. Schlüpmann, Landolt-Börnstein – Group I, Vol. 6 (1972) 79–164
[12] H. Bøggild et al., Nucl. Phys. B57 (1973) 77
[13] W. H. Sims et al., Nucl. Phys. B41 (1972) 317
[14] E. E. Zabrodin et al., Phys. Rev. D52 (1995) 1316
[15] V. V. Ammosov et al., N. Cim. 40A (1977) 237
[16] C. Bromberg et al., Nucl. Phys. B107 (1976) 82
[17] C. Alt et al., Eur. Phys. J. C45 (2006) 343
[18] M. G. Albrow et al., Nucl. Phys. B56 (1973) 333
[19] B. Alper et al., Nucl. Phys. B100 (1975) 237
[20] P. Capiluppi et al., Nucl. Phys. B79 (1974) 189
[21] K. Guettler et al., Phys. Lett. 64B (1976) 111
[22] J. Singh et al., Nucl. Phys. B140 (1978) 189
[23] M. G. Albrow et al., Phys. Lett. 42B (1972) 279
[24] B. Alper et al., Nucl. Phys. B87 (1975) 19
[25] P. Capiluppi et al., Nucl. Phys. B70 (1974) 1
[26] A. C. Melissinos et al., Phys. Rev. 128 (1962) 2373
[27] C. W. Akerlof et al., Phys. Rev. D3 (1971) 645
[28] D. Dekkers et al., Phys. Rev. 137 (1965) B962
[29] J. V. Allaby et al., CERN 70-12 (1970)
[30] U. Amaldi et al., Nucl. Phys. B86 (1975) 403
[31] E. W. Beier et al., Phys. Rev. D18 (1978) 2235
[32] B. D. Anderson et al., Phys. Rev. Lett. 19 (1967) 198
[33] V. V. Abramov et al., Nucl. Phys. B173 (1980) 348
[34] A. E. Brenner et al., Phys. Rev. D26 (1982) 1497
[35] J. R. Johnson et al., Phys. Rev. D17 (1978) 1292
[36] D. Antreasyan et al., Phys. Rev. D19 (1979) 764
[37] N. Abgrall et al., Eur. Phys. J. C74 (2014) 2794
[38] A. Aduszkiewicz et al., Eur. Phys. J. C77 (2017) 671
[39] S. Afanasiev et al., Nucl. Instr. Meth. A430 (1999) 210
[40] J. Adams et al., Phys. Lett. B637 (2006) 161
[41] B. I. Abelev et al., Phys. Rev. C79 (2009) 034909
[42] A. Adare et al., Phys. Rev. C83 (2011) 064903
[43] I. Arsene et al., Phys. Rev. Lett. 98 (2007) 252001
[44] H. Yang, PhD Dept. of Physics, Bergen Univ.(2007)
[45] K. Aamodt et al., Eur. Phys. J. C71 (2011) 1655
[46] B. Abelev et al., Phys. Lett. B736 (2014) 196
[47] S. Acharya et al., Phys. Rev. C101 (2020) 044907
[48] J. Adam et al., Eur. Phys. J. C75 (2015) 226
[49] S. Acharya et al., arXiv:2005.11120 [nucl-ex]
[50] S. Chatrchyan et al., Eur. Phys. J. C72 (2012) 2164
[51] A. M. Sirunyan et al., Phys.Rev. D96 (2017) 112003
[52] T. Anticic et al., Eur. Phys. J. C68 (2010) 1
[53] T. Anticic et al., Eur. Phys. J. C65 (2010) 9
[54] C. Alt et al., Eur. Phys. J. C49 (2007) 897
[55] E. Bierman et al., Phys. Rev. 147 (1966) 922
[56] R. L. Eisner et al., Nucl. Phys. B123 (1977) 361
[57] B. Y. Oh and G. A. Smith, Nucl. Phys. B49 (1972) 13
[58] K. Jaeger et al., Phys. Rev. D11 (1975) 1756
[59] K. Alpgård et al., Nucl. Phys. B103 (1976) 234
[60] V. Blobel et al., Nucl. Phys. B135 (1978) 379
[61] H. Fesefeldt et al., Nucl. Phys. B147 (1979) 317
[62] V. V. Ammosov et al., Nucl. Phys. B115 (1976) 269
[63] J. W. Chapman et al., Phys. Lett. 47B (1973) 465
[64] D. Brick et al., Nucl. Phys. B164 (1980) 1
[65] K. Jaeger et al., Phys. Rev. D11 (1975) 2405
[66] A. Sheng et al., Phys. Rev. D11 (1975) 1733
[67] M. Asai et al., Z. Phys. C27 (1985) 11
[68] R. D. Kass et al., Phys. Rev. D20 (1979) 605
[69] H. Kichimi et al., Phys. Rev. D20 (1979) 37
[70] F. W. Büsser et al., Phys. Lett. 61B (1976) 309
[71] S. Erhan et al., Phys. Lett. 85B (1979) 447
[72] B. Rensch, Thesis Heidelberg Univ. RX-1096 (1985)
[73] A. Aduszkiewicz et al., Eur. Phys. J. C76 (2016) 198
[74] R. Hagedorn, Riv. Nuovo Cim. 6 (1983) 1
[75] H. Grässler et al., Nucl. Phys. B132 (1978) 1
[76] G. Jancso et al., Nucl. Phys. B124 (1977) 1
[77] http://spshadrons.web.cern.ch
[78] I. Arsene et al., Phys. Rev. Lett. 98 (2007) 252001
[79] O. Adriani et al., Phys. Rev. D86 (2012) 092001
[80] O. Adriani et al., Phys. Rev. D94 (2016) 032007
[81] J. Benecke et al., Phys. Rev. 188 (1969) 2159
[82] R. K. Adair, Phys. Rev. 172 (1968) 1370
[83] S. J. Lindenbaum and R. M. Sternheimer, Phys. Rev. 105 (1957) 1874
[84] M. L. Good and W. D. Walker, Phys. Rev. 120 (1960) 1857
[85] J. D. Bjorken, Phys. Rev. 179 (1969) 1547
[86] S. M. Berman, J. D. Bjorken and J. B. Kogut, Phys. Rev. D4 (1971) 3388
[87] R. D. Field and R. P. Feynman, Phys. Rev. D15 (1977) 2590
[88] R. P. Feynman, R. D. Field and G. C. Fox, Phys. Rev. D18 (1978) 3320
[89] B. Baatar et al., Eur. Phys. J. C73 (2013) 2364
[90] O. Chvala et al., Eur. Phys. J. C73 (2013) 2329
[91] D. Varga, PhD Thesis (2005) Eötvös Loránd University, Budapest
[92] S. Chekanov et al., Nucl. Phys. B658 (2003) 3
[93] S.Chekanov et al., Nucl. Phys. B776 (2007) 1
[94] M. Bozzo et al., Phys. Lett. B136 (1984) 217
[95] M. Bardadin-Otwinowska et al., Phys. Lett. 21 (1966) 351
[96] G. W. van Apeldoorn et al., Nucl. Phys. B91 (1975) 1
[97] E. Fermi, Prog. Theor. Phys. 5 (1950) 570
[98] S. Z. Belenskij and L. D. Landau, Nuovo Cim. Suppl. 3S10 (1956) 15
[99] R. Hagedorn, Nuovo Cim. Suppl. 3 (1965) 147
[100] R. Hagedorn and J. Ranft, Nuovo Cim. Suppl. 6 (1968) 169
[101] C. Tsallis, J. Stat. Phys. 52 (1988) 479
[102] F. Bueyuekkilic and D. Demirhan, Phys. Lett. A181 (1993) 24
[103] R. Hagedorn, Nuovo Cim. Vol LII A, N.4 (1967), CERN-TH-827 (1967)
[104]  A S. Parvan and T. Bhattacharyya, arXiv:1904.02947v
[105] J. Whitmore, Proc. 19th Int. Conf. High Energy Physics, Tokyo (1978)
[106] V. V .Ammosov et al., Nucl. Phys. 24 (1976) 30
[107] J. Erwin et al., Phys. Rev. Lett. 35 (1975) 980
[108] J. P. de Brion et al., Phys. Rev. Lett. 34 (1975) 910
[109] D. Brick et al., Phys. Rev. D21 (1980) 632
[110] S. J. Barish et al., Phys. Rev. D12 (1975) 1260
[111] F. T. Dao et al., Phys. Rev. Lett. 30 (1973) 34
[112] D  Drijard et al., Z. Phys. C21 (1984) 321
[113] M. Aguilar-Benitez et al., Z. Phys.C50 (1991) 405
[114] J. Bracinik, Thesis Comenius University Bratislava  MFF-UK (2000)
[115] M. Kreps, Thesis Comenius Unversity Bratislava (2004)
[116] M. Jacob and R. Slansky, Phys. Rev. D5 (1972) 1847
[117] J. D. Jackson, Nuovo Cim. 34 (1964) 1644
[118] H. G. Fischer, private communication
[119] Particle Data Group, PTEP 2020 (2020) 083C01
[120] H. G. Fischer et al., Acta Physica Polonica B33 (2002) 1473
[121] H. G. Fischer et al., https://spshadrons.web.cern.ch/docs.html, document S8(2005),
CERN/SPSC 2005-35
[122] W. Ochs, Nucl. Phys. B118 (1977) 397
[123] K. P. Das and R. C. Hwa, Phys. Lett. 68B (1977) 459
[124] M. Della Negra et al., Nucl. Phys. B127 (1977) 1
[125] H. G. Fischer and W. Geist, Z. Phys. C19 (1983) 159


# AMP: Adaptive Masked Proxies for Few-Shot Segmentation

Mennatullah Siam
University of Alberta
<EMAIL_ADDRESS>Boris N. Oreshkin
Element AI
<EMAIL_ADDRESS>Martin Jagersand
University of Alberta
<EMAIL_ADDRESS>
###### Abstract

Deep learning has thrived by training on large-scale datasets. However, in
robotics applications sample efficiency is critical. We propose a novel
adaptive masked proxies method that constructs the final segmentation layer
weights from few labelled samples. It utilizes multi-resolution average
pooling on base embeddings masked with the label to act as a positive proxy
for the new class, while fusing it with the previously learned class
signatures. Our method is evaluated on PASCAL-$5^{i}$ dataset and outperforms
the state-of-the-art in the few-shot semantic segmentation. Unlike previous
methods, our approach does not require a second branch to estimate parameters
or prototypes, which enables it to be used with 2-stream motion and appearance
based segmentation networks. We further propose a novel setup for evaluating
continual learning of object segmentation which we name incremental PASCAL
(iPASCAL) where our method outperforms the baseline method. Our code is
publicly available at https://github.com/MSiam/AdaptiveMaskedProxies.

##  1 Introduction

Children are able to adapt their knowledge and learn about their surrounding
environment with limited samples [18]. One of the main bottlenecks in the
current deep learning methods is their dependency on the large-scale training
data. However, it is intractable to collect one large-scale dataset that
contains all the required object classes for different environments. This
motivated the emergence of few-shot learning methods [12, 38, 32, 26, 27].
These early works were primarily focused on solving few-shot image
classification tasks, where a support set consists of a few images and their
class labels. The earliest attempt to solve the few-shot segmentation task
seems to be the approach proposed by Shaban et al. [28] that predicts the
parameters of the final segmentation layer. This and other previous methods
require the training of an additional branch to guide the backbone
segmentation network. The additional network introduces extra computational
burden. On top of that, existing approaches cannot be trivially extended to
handle the continuous stream of data containing annotations for both novel and
previously learned classes.

Figure 1: Multi-resolution adaptive imprinting in AMP.

To address these shortcomings, we propose a novel sample efficient adaptive
masked proxies method, which we call AMP. It constructs weights of the final
segmentation layer via multi-resolution imprinting. AMP does not rely on a
second guidance branch, as shown in Figure 1. Following the terminology of
[19], a proxy is a representative signature of a given class. In the few-shot
segmentation setup, the support set contains pixel-wise class labels for each
support image. Therefore, the response of the backbone fully convolutional
network (FCN) to a set of images from a given class in the support set can be
masked by segmentation labels and then average pooled to create a proxy for
this class. This forms what we call a normalized masked average pooling layer
(NMAP in Fig. 1). The computed proxies are used to set the 1x1 convolutional
filters for the new classes, forming the process known as weight imprinting
[23]. Multi-resolution weight imprinting is proposed to improve the
segmentation accuracy of our method.

We further consider the continual learning setup in which a few-shot algorithm
may be presented with a sequence of support sets (continuous semantic
segmentation scenario). In connection with this scenario, we propose to adapt
the previously learned class weights with the new proxies from each incoming
support set. Imprinting only the weights for the positive class, i.e. the
newly added class, is insufficient as new samples will incorporate new
information about other classes as well. For example, learning a new class for
boat will also entail learning new information about the background class,
which should include sea. To address this, a novel method for updating the
weights of the previously learned classes without back-propagation is
proposed. The adaptation part of our method is inspired by the classical
approaches in learning adaptive correlation filters [1, 7]. Correlation
filters date back to 1980s [8]. More recently, the fast object tracking method
[1] relied on hand crafted features to form the correlation filters and
adapted them using a running average. In our method the adaptation of the
previously learned weights is based on a similar approach, yielding the
ability to process the continuous stream of data containing novel and existing
classes. This opens the door toward leveraging segmentation networks to
continually learn semantic segmentation in a sample efficient manner.

To sum up, AMP is shown to provide sample efficiency in three scenarios: (1)
few-shot semantic segmentation, (2) video object segmentation and (3)
continuous semantic segmentation. Unlike previous methods, AMP can easily
operate with any pre-trained network without the need to train a second
branch, which entails fewer parameters. In the video object segmentation
scenario we show that our method can be used with a 2-stream motion and
appearance network without any additional guidance branch. AMP is flexible and
still allows coupling with back-propagation using the support image-label
pair. The proxy weight imprinting steps can be interleaved with the back-
propagation steps to boost the adaptation process. AMP is evaluated on
PASCAL-$5^{i}$ [28], DAVIS benchmark [22], FBMS [20] and our proposed iPASCAL
setup. The novel contributions of this paper can be summarized as follows.

  * •

Normalized masked average pooling layer that efficiently computes a class
signature from the backbone FCN response without relying on an additional
branch.

  * •

Multi-resolution imprinting scheme that imprints the proxies from several
resolutions of the backbone FCN to increase accuracy.

  * •

Novel adaptation mechanism that updates the weights of known classes based on
the new proxies.

  * •

Empirical results that demonstrate that our method is state-of-the-art on
PASCAL-$5^{i}$, and on DAVIS’16.

  * •

iPASCAL, a new version of PASCAL-VOC to evaluate the continuous semantic
segmentation.

##  2 Related Work

###  2.1 Few-shot Classification

In few-shot classification, the model is provided with a support set and a
query image. The support set contains a few labelled samples that can be used
to train the model, while the query image is used to test the final model. The
setup is formulated as $k$-shot $n$-way, where $k$ denotes the number of
samples per class, while $n$ denotes the number of classes in the support set.
An early approach to solve the few-shot learning problem relied on Bayesian
methodology [6]. More recently, Vinyals et al. proposed matching networks
approach that learns an end-to-end differentiable nearest neighbour [38].
Following that, Snell et al. proposed prototypical networks based on the
assumption that there exists an embedding space in which points belonging to
one class cluster around their corresponding centroid [32]. Qiao et al.
proposed a parameter predictor method [24]. Finally, a method for computing
imprinted weights was proposed by Qi et al. [23].

###  2.2 Few-shot Semantic Segmentation

Unlike the classification scenario that assumes the availability of image
level class labels, the few-shot segmentation relies on pixel-wise class
labels for support images. A popular dataset used to evaluate few-shot
segmentation is PASCAL-$5^{i}$ [28]. The dataset is sub-divided into 4 folds
each containing 5 classes. A fold contains labelled samples from 5 classes
that are used for evaluating the few-shot learning method. The rest 15 classes
are used for training. Shaban et al. proposed a 2-branch method [28], where
the second branch predicts the parameters for the final segmentation layer.
The baselines proposed by Shaban et al. [28] included nearest neighbour,
siamese network, and naive fine-tuning. Rakelly et al. proposed a 2-branch
method where the second branch acts as a conditioning branch instead [25].
Finally, Dong et al. inspired from prototypical networks, designed another
2-branch method to learn prototypes for the few-shot segmentation problem [4].
Clearly, most of the previously proposed methods require an extra branch
trained in a simulated few-shot setting. They cannot be trivially extended to
continue adaptation whilst processing a continuous stream of data with
multiple classes.

Figure 2: AMP using the NMAP Layer. For simplicity it shows the imprinting on
the final layer solely. Nonetheless, our scheme is applied on multiple
resolution levels.

In a concurrent work, Zhang et al. [41] proposed a single branch network
deriving guidance features from masked average pooling layer. This is similar
to our NMAP layer. Zhang et al. [41] use the output of their pooling layer to
compute a guidance to the base network. AMP uses NMAP output to imprint the
1x1 convolutional layer weights. AMP has the following advantages: (i) it
allows the adaptation of imprinted weights in continuous data stream, (ii) it
can be seamlessly coupled with any pre-trained networks, including 2-stream
networks for video object segmentation.

##  3 AMP: Adaptive Masked Proxies

Our approach, which we call AMP, is rooted deeply in the concept of weight
imprinting [23]. The imprinting process was initially proposed in the context
of classification [23]. The method used the normalized responses of the base
feature extractor as weights of the final fully connected layer. In this
context, the normalized response of the feature extractor for a given class is
called a proxy. The justification behind such learning scheme is based on the
relation between metric learning, proxy-NCA loss and softmax cross-entropy
loss [19]. 1x1 convolutional layers are equivalent to fully connected layers.
Hence we propose to utilize base segmentation network activations as proxies
to imprint the 1x1 convolutional filters of the final segmentation layer. When
convolved with the query image, the imprinted proxy activates pixels maximally
similar to its class signature.

However, it is not trivial to perform weight imprinting in semantic
segmentation, unlike in classification. First, in the classification setup the
output embedding vector corresponds to a single class and hence can be used
directly for imprinting. By contrast, a segmentation network outputs 3D
embeddings, which incorporate features for a multitude of different classes,
both novel and previously learned. Second, unlike classification, multi-
resolution support is essential in segmentation.

We propose the following novel architectural components to address the
challenges outlined above. First, in Section 3.1 and in Section 3.2 we propose
the proxy masking and adaptation methods to handle multi-class segmentation.
Second, in Section 3.3 we propose a multi-resolution weight imprinting scheme
to maintain the segmentation accuracy during imprinting. The contribution of
each method to the overall accuracy is further motivated experimentally in
Section 4.2.

###  3.1 Normalized Masked Average Pooling

We propose to address the problem of imprinting the 3D segmentation base
network embeddings that contain responses from multiple classes in a single
image by masking the embeddings prior to averaging and normalization. We
encapsulate this function in a NMAP layer (refer to Figures 1 and 2). To
construct a proxy for one target class, the NMAP layer bilinearly upsamples
segmentation base network outputs and masks them via the pixel-wise labels for
the target class available in the support set. This is followed by average
pooling and normalization as follows:

$P^{r}_{l}=\frac{1}{k}\sum_{i=1}^{k}{\frac{1}{N}\sum_{x\in
X}{F^{ri}(x)Y^{i}_{l}(x)}},$ (1a) $\hat{P^{r}_{l}}=\frac{P^{r}_{l}}{\lVert
P^{r}_{l}\rVert_{2}}.$ (1b)

Here $Y^{i}_{l}$ is a binary mask for $i^{th}$ image with the novel class $l$,
$F^{ri}$ is the corresponding output feature maps for $i^{th}$ image and $rth$
resolution. $X$ is the set of all possible spatial locations and $N$ is the
number of pixels that are labelled as foreground for class $l$. The normalized
output from the masked average pooling layer $\hat{P}^{r}_{l}$ can be further
used as proxies representing class $l$ and resolution $r$. In the case of a
novel class the proxy can be utilized directly as filter weights. In the case
of few-shot learning, the average of all the NMAP processed features for the
samples provided in the support set for a given class is used as its proxy.

###  3.2 Adaptive Proxies

The NMAP layer solves the problem of processing a single support set. However,
in practice many of the applications require the ability to process a
continuous stream of support sets. This is the case in continuous semantic
segmentation and video object segmentation scenarios. In this context the
learning algorithm is presented with a sequence of support sets. Each incoming
support set may provide information on both the new class and the previously
learned classes. It is valuable to utilize both instead of solely imprinting
the new class weights. At the same time, in the case of the previously learned
classes, e.g. background, it is not wise to simply override what the network
learned from the large-scale training either. A good example illustrating the
need for updating the negative classes is the addition of class boat. It is
obvious that the background class needs to be updated to match the sea
background, especially if the images with sea background are not part of the
large scale training dataset.

To take advantage of the information available in the continuous stream of
data, we propose to adapt class proxies with the information obtained from
each new support set. We propose the following exponentially smoothed adaptive
scheme with update rate $\alpha$:

$\hat{W}^{r}_{l}=\alpha\hat{P^{r}_{l}}+(1-\alpha)W^{r}_{l}.$ (2)

Here $\hat{P^{r}_{l}}$ is the normalized masked proxy for class $l$,
$W^{r}_{l}$ is the previously learned 1x1 convolutional filter at resolution
$r$, $\hat{W}^{r}_{l}$ is the updated $W^{r}_{l}$. The update rate can be
either treated as as a hyper-parameter or learned.

The adaptation mechanism is applied differently in the few-shot setup and in
the continual learning setup. In the few-shot setup, the support set contains
segmentation masks for each new class foreground and background. The
adaptation process is performed on the background class weights from the large
scale training. The proxies for the novel classes are derived directly from
the NMAP layer via imprinting with no adaptation. In the continual learning
setup, the proxies for all the classes learned up to the current task are
available when a new support set is processed. Thus, we adapt all the proxies
learned in all the previous tasks for which samples are available in the
support set of the current task.

Figure 3: Multi-resolution imprinting using proxies from different resolution
levels.

###  3.3 Multi-resolution Imprinting Scheme

In the classification scenario, in which imprinting was originally proposed,
the resolution aspect is not naturally prominent. In contrast, in the
segmentation scenario, resolution is naturally important to obtain very
accurate segmentation mask predictions. On top of that, we argue that
imprinting the outputs of several resolution levels and fusing the probability
maps from those in the final probability map can be used to improve overall
segmentation accuracy. This is illustrated in Fig. 3, showing the output
heatmaps from 1x1 convolution using our proposed proxies as imprinted weights
at three different resolutions, $\hat{P^{1}_{l}}$, $\hat{P^{2}_{l}}$,
$\hat{P^{3}_{l}}$. Clearly, the coarse resolution captures blobs necessary for
global alignment, while the fine resolution provides the granular details
required for an accurate segmentation.

Figure 4: Visualization for the T-SNE [17] embeddings for the generated masked
proxies. Layers L1, L2, L3 denote the smaller to higher resolution feature
maps.

This idea is further supported by the T-SNE [17] plot of the proxies learned
in the proposed NMAP layer at different resolutions depicted in Fig. 4. It
shows the 5 classes belonging to fold 0 in PASCAL-$5^{i}$ at 3 resolutions
imprinted by our AMP model. A few things catch attention in Fig. 4. First,
clustering is different at different resolutions. Fusing probability maps at
different resolutions may therefore be advantageous from statistical
standpoint, as slight segmentation errors at different resolutions may cancel
each other. Second, the class-level clustering is not necessarily tightest at
the highest resolution level: mid-resolution layer L2 seems to provide the
tightest clustering. This may seem counter-intuitive. Yet, this is perfectly
in line with the latest empirical results in weakly-supervised learning (see
[2] and related work). For example, [2] clearly demonstrates that
convolutional networks store most of the class level information in the middle
layers, and mid-resolution features result in the best transfer learning
classification results.

###  3.4 Base Network Architectures

The backbone architecture used in our segmentation network is a VGG-16 [31]
that is pre-trained on ImageNet [3]. Similar to the FCN8s architecture [16]
skip connections are used to benefit from higher resolution feature maps, and
a 1x1 convolution layers are used to map from the feature space to the label
space. Unlike FCN8s we utilize bilinear interpolation layers with fixed
weights for upsampling. This is to simplify the imprinting of weights based on
the support set (transposed convolutions are hard to imprint). We also rely on
an extension to the above base network using dilated convolution [40], which
we call DFCN8s. The last two pooling layers are replaced by dilated
convolution with dilation factors 2 and 4 respectively. This increases the
receptive field without affecting the resolution. Finally, a more compact
version of the network with two final convolutional layers removed is denoted
as Reduced-DFCN8s. The final classification layer, and the two 1x1
convolutional layers following dilated convolutions in the case of DFCN8s and
the Reduced-DFCN8s are the ones imprinted.

In the video object segmentation scenario we use a 2-stream wide-resnet [39]
architecture. Each stream has 11 residual blocks followed by multiplying the
output activation from both motion and appearance. The motion is presented to
the model as optical flow based on Liu et al. [15] and converted to RGB using
a color wheel. The flexibility of our method enables it to work with different
architectures without the overhead of designing another branch to provide
guidance, predicted parameters or prototypes.

Table 1: mIoU for 1-way 1-shot segmentation on PASCAL-$5^{i}$. FT: Fine-tuning. AMP-1 and AMP-2: our method using DFCN8s and Reduced-DFCN8s, respectively. Red, Blue: best and second best methods. co-FCN evaluation is from [41]. | 1-NN [28] | Siamese [28] | FT [28] | OSLSM [28] | co-FCN [25] | AMP-1 (ours) | AMP-2 (ours)
---|---|---|---|---|---|---|---
Fold 0 | 25.3 | 28.1 | 24.9 | 33.6 | 36.7 | 37.4 | 41.9
Fold 1 | 44.9 | 39.9 | 38.8 | 55.3 | 50.6 | 50.9 | 50.2
Fold 2 | 41.7 | 31.8 | 36.5 | 40.9 | 44.9 | 46.5 | 46.7
Fold 3 | 18.4 | 25.8 | 30.1 | 33.5 | 32.4 | 34.8 | 34.7
Mean | 32.6 | 31.4 | 32.6 | 40.8 | 41.1 | 42.4 | 43.4
Table 2: mIoU for 1-way 5-shot segmentation on PASCAL-$5^{i}$. FT: Fine-tuning. AMP-2 + FT(2): our method with 2 fine-tuning iterations, respectively. Red, Blue: best and second best methods. co-FCN evaluation is from [41]. | 1-NN [28] | LogReg [28] | OSLSM [28] | co-FCN [25] | AMP-2 (ours) | AMP-2 + FT(2) (ours)
---|---|---|---|---|---|---
Fold 0 | 34.5 | 35.9 | 35.9 | 37.5 | 40.3 | 41.8
Fold 1 | 53.0 | 51.6 | 58.1 | 50.0 | 55.3 | 55.5
Fold 2 | 46.9 | 44.5 | 42.7 | 44.1 | 49.9 | 50.3
Fold 3 | 25.6 | 25.6 | 39.1 | 33.9 | 40.1 | 39.9
Mean | 40.0 | 39.3 | 43.9 | 41.4 | 46.4 | 46.9

###  3.5 Training and Evaluation Methodology

Few-shot segmentation. We use the same setup as Shaban et al. [28]. The
initial training phase relies on a large scale dataset $D_{train}$ including
semantic label maps for classes in $L_{train}$. During the test phase, a
support set and a query image are sampled from $D_{test}$ containing novel
classes with labels in $L_{test}$, where $L_{train}\cap L_{test}=\emptyset$.
The support set contains pairs $S={(I_{i},Y_{i}(l))}_{i=1}^{k}$, where $I_{i}$
is the $i^{th}$ image in the set and $Y_{i}(l)$ is the corresponding binary
mask. The binary mask $Y_{i}(l)$ is constructed with novel class $l$ labelled
as foreground while the rest of the pixels are considered background. As
before, $k$ denotes the number of images provided in the support set. It is
worth noting that during training only images that include at least one pixel
belonging to $L_{train}$ are included in $D_{train}$ for large-scale training.
If some images have pixels labelled as classes belonging to $L_{test}$ they
are ignored and not used in the back-propagation. Our model does not need to
be trained in the few-shot regime by sampling a support set and a query image.
It is trained in a normal fashion with image-label pairs.

Continuous Semantic Segmentation. In continuous semantic segmentation
scenario, we propose the setup based on PASCAL VOC [5], following the class
incremental learning scenario described in [37]. We call the proposed setup
incremental PASCAL (iPASCAL). It is designed to assess sample efficiency of a
method in the continual learning setting. The classes in the dataset are split
into $L_{train}$ and $L_{incremental}$ with 10 classes each, where
$L_{train}\cap L_{incremental}=\emptyset$ . The classes belonging to the
$L_{train}$ are used to construct the training dataset $D_{train}$ and pre-
train the segmentation network. Unlike the static setting in the few-shot
case, the continuous segmentation mode provides the image-label pairs
incrementally with different encountered tasks. The tasks are in the form of
triplets $(t_{i},(X_{i},Y_{i}))$, where $(X_{i},Y_{i})$ represent the overall
batch of images and labels from task $t_{i}$. Each task $t_{i}$ introduces two
novel classes to learn in its batch. That batch contains samples with at least
one pixel belonging to these two novel classes. The labels per task $t_{i}$
include the two novel classes belonging to that task, and the previously
learned classes in the encountered tasks $t_{0},...,t_{i-1}$.

##  4 Experimental Results

We evaluate the sample efficiency of the proposed AMP method in three
different scenarios: (1) few-shot segmentation, (2) video object segmentation,
and (3) continuous semantic segmentation. In the few-shot segmentation
scenario we evaluate on pascal-$5^{i}$ [28] (see Section 4.1). An ablation
study is performed to demonstrate the improvement resulting from multi-
resolution imprinting and proxy adaptation in Section 4.2. The study also
compares weight imprinting coupled with back-propagation against back-
propagation on randomly generated weights. Section 4.4 demonstrates the
benefit of AMP in the context of continuous semantic segmentation on the
proposed incremental PASCAL VOC evaluation framework, iPASCAL. We further
evaluate AMP in the online adaptation scenario on DAVIS [22] and FBMS [20]
benchmarks for video object segmentation (see Section 4.3). We use mean
intersection over union (mIoU) [28] as evaluation metric unless explicitly
stated otherwise. mIoU denotes the average of the per-class IoUs per fold. Our
training and evaluation code is based on the semantic segmentation work [29]
and is made publicly available
111https://github.com/MSiam/AdaptiveMaskedProxies.

###  4.1 Few-Shot Semantic Segmentation

The setup for training and evaluation on PASCAL-$5^{i}$ is as follows. The
base network is trained using RMSProp [9] with learning rate $10^{-6}$ and L2
regularization weight 5x$10^{-4}$. For each fold, models are pretrained on 15
train classes and evaluated on remaining 5 classes, unseen during pretraining.
The few-shot evaluation is performed on 1000 randomly sampled tasks, each
including a support and a query set, similar to OSLSM setup [28]. A hyper-
parameter random search is conducted over the $\alpha$ parameter, the number
of iterations, and the learning rate. The search is conducted by training on
10 classes from the training set and evaluating on the other 5 classes of the
training set. Thus ensuring all the classes used are outside the fold used in
the evaluation phase. The $\alpha$ parameter selected is 0.26. In the case of
performing fine-tuning, the selected learning rate is 7.6x$10^{-5}$ with 2
iterations for the 5-shot case.

Figure 5: Qualitative evaluation on PASCAL-$5^{i}$ 1-way 1-shot. The support
set and prediction on the query image are shown.

Tables 1 and 2 show the mIoU for the 1-shot and 5-shot segmentation,
respectively, on PASCAL-$5^{i}$ (mIoU is computed on the foreground class as
in [28]). Our method is compared to OSLSM [28] as well as other baseline
methods for few-shot segmentation. AMP outperforms the baseline fine-tuning
[28] method by 10.8% in terms of mIoU, without the need for extra back-
propagation iterations by directly using the adaptive masked proxies. AMP
outperforms OSLSM [28] in both the 1-shot and the 5-shot cases. Unlike OSLSM,
our method does not need to train an extra guidance branch. This advantage
provides the means to use AMP with a 2-stream motion and appearance based
network as shown in Section 4.3. On top of that, AMP outperforms co-FCN method
[25].

Table 3 reports our results in comparison to the state-of-the-art using the
evaluation framework of [25] and [4]. In this framework the mIoU is computed
as the mean of the foreground and background IoU averaged over folds. AMP
outperforms the baseline FG-BG [4] in the 1-shot and 5-shot cases. When our
method is coupled with two iterations of back-propagation through the last
layers solely it outperforms co-FCN [25] in the 5-shot case by 3%.

Qualitative results on PASCAL-$5^{i}$ are demonstrated in Figure 5 that shows
both the support set image-label pair, and segmentation for the query image
predicted by AMP. Importantly, segmentation produced by AMP does not seem to
depend on the saliency of objects. In some of the query images, multiple
potential objects can be categorized as salient, but AMP learns to segment
what best matches the target class.

Table 3: Quantitative results for 1-way, 1-shot and 5-shot segmentation on PASCAL-$5^{i}$ dataset, following evaluation in [4]. FT: Fine-tuning for 2 iterations in 1-shot and 5-shot setting. Red, Blue: best and second best methods. Method | 1-Shot | 5-Shot
---|---|---
FG-BG [4] | 55.1 | 55.6
OSLSM [28] | 55.2 | -
co-FCN [25] | 60.1 | 60.8
PL+SEG [4] | 61.2 | 62.3
AMP-2 (ours) | 61.9 | 62.1
AMP-2 + FT (ours) | 62.2 | 63.8
Table 4: Ablation study of the different design choices for the imprinting scheme. Adaptation: $\alpha$ parameter is non-zero. Multi-res: performing multi-resolution imprinting. Imp: imprinting weights using our proxies. FT: fine-tuning. Method | Adaptation | Multi-res. | N-Shot | mIoU
---|---|---|---|---
FT only | ✗ | ✓ | 5 | 28.7
Imp. | ✓ | ✓ | 5 | 40.3
Imp. + FT | ✓ | ✓ | 5 | 41.8
Imp. | ✗ | ✓ | 1 | 13.6
Imp. | ✓ | ✗ | 1 | 34.8
Imp. | ✓ | ✓ | 1 | 41.9
Table 5: Quantitative comparison between unsupervised methods and the adaptive masked imprinting scheme on DAVIS’16. Measure | FSeg [10] | LVO [36] | MOTAdapt [30] | ARP [13] | PDB [33] | AMP + CRF (Ours)
---|---|---|---|---|---|---
$\mathcal{J}$ | Mean | 70.7 | 75.9 | 77.2 | 76.2 | 77.2 | 78.9
Recall | 83.5 | 89.1 | 87.8 | 91.1 | 90.1 | 91.6
Decay | 1.5 | 7.0 | 5.0 | 7.0 | 0.9 | 4.7
$\mathcal{F}$ | Mean | 65.3 | 72.1 | 77.4 | 70.6 | 74.5 | 78.4
Recall | 73.8 | 83.4 | 84.4 | 83.5 | 84.4 | 87.3
Decay | 1.8 | 1.3 | 3.3 | 7.9 | 0.2 | 2.7
Table 6: Quantitative results on FBMS dataset (test set). Measure | FST [21] | CVOS [34] | CUT [11] | MPNet-V[35] | LVO[36] | MotAdapt [30] | AMP (ours)
---|---|---|---|---|---|---|---
$\mathcal{P}$ | 76.3 | 83.4 | 83.1 | 81.4 | 92.1 | 80.7 | 82.7
$\mathcal{R}$ | 63.3 | 67.9 | 71.5 | 73.9 | 67.4 | 77.4 | 75.7
$\mathcal{F}$ | 69.2 | 74.9 | 76.8 | 77.5 | 77.8 | 79.0 | 79.0

###  4.2 Ablation Study

We perform an ablation study to demonstrate the effectiveness of different
components in AMP. Results are reported in Table 4. For our final method, it
corresponds to the evaluation provided in Tables 1 and 2 on fold 0, following
Shaban et al. [28]. First, AMP clearly outperforms naïve fine-tuning using
randomly generated weights by 11.6%. Second, AMP can be effectively combined
with the fine-tuning of imprinted weights to further improve performance. This
is ideal for a continuous data stream processing. Third, AMP’s proxy
adaptation component is effective: no adaptation with $\alpha$ set to 0,
degrades accuracy by 28.3% in the 1-shot scenario. Finally, multi-resolution
imprinting is effective: not performing multi-resolution imprinting degrades
mIoU in the 1-shot scenario. We conclude that simply imprinting the weights
only for the new class is not optimal. Imprinting has to be coupled with the
proposed adaptation and multi-resolution schemes to be effective in the
segmentation scenario.

###  4.3 Video Object Segmentation

To assess AMP in the video object segmentation scenario, we use it to adapt
2-stream segmentation networks based on pseudo-labels and evaluate on the
DAVIS-2016 benchmark [22]. Here our base network is a 2-stream Wide ResNet
model similar to [30]. We make the model adapt to the appearance changes that
the object undergoes in the video sequence using the proposed proxy adaptation
scheme with $\alpha$ parameter set to 0.001. The adaptation mechanism operates
on top of the masked proxies derived from the segmentation probability maps
output from the model itself, since the model has learned background-
foreground segmentation already. Therefore, we call this ”self adaptation” as
it is unsupervised video object segmentation. Since we do not employ manual
segmentation masks, we compare our results against the state-of-the-art
unsupervised methods that utilize motion and appearance based models. Table 5
shows the mIoU over the validation set for AMP and the baselines. Our method
when followed with fully connected conditional random fields [14] post
processing outperforms the state of the art (the CRF post-processing is
commonly applied by most methods evaluated on DAVIS’16).

Table 6 shows our self adaptation results on FBMS dataset where it outperforms
all methods except for MotAdapt [30], which it is on-par with. These results
uncover one of the weaknesses of our method: it is unable to operate with high
dilation rates since it relies on masked proxies. High dilation rates can lead
to interference between background and foreground features in AMP. Another
AMP’s weakness is that it may face difficulties segmenting a specific
instance, since it uses a proxy per class that aims to generalize across
different instances.

###  4.4 Continuous Semantic Segmentation

Figure 6: N-way evaluation on iPASCAL. Naive #M: fine-tuning with M iteration
per sample. Imprint A: our method is used with $\alpha$ for the task classes
set to A.

To demonstrate the benefit of AMP in the continuous semantic segmentation
scenario, we conducted experiments on iPASCAL. iPASCAL provides triplets for
the task, the corresponding images and semantic labels. For each task,
semantic labels include labels of new classes encountered in the current task
as well as the labels of classes encountered in the previous tasks (please see
Section 3.5 for more details on the setup definition). Figure 6 compares naïve
fine-tuning from random weights against AMP without any fine-tuning, in terms
of mIoU (average over 5 runs). Multiple runs are evaluated with different
seeds that control random assignment of unseen classes in new tasks. The mIoU
is reported per task on all the classes learned up to the current task. Fine-
tuning was conducted using RMSProp with the best learning rate from the 1-shot
setup 9.06x$10^{-5}$. Fine-tuning is applied to the last layers responsible
for pixel-wise classification, while the feature extraction weights are kept
fixed. We are focusing on improving sample efficiency by imprinting the
weights of the final layer, therefore we perform the fine-tuning on the final
weights only. Figure 6 demonstrates that in the continual learning scenario,
weight imprinting via AMP is more effective than fine-tuning, which suffers
from over-fitting that is very hard to overcome.

It is worth noting that the current evaluation setting is a $n$-way where $n$
increases with 2 additional classes with each encountered task resulting in
10-way evaluation in the last task. This explains the difference between the
mIoU in Table 1 and Figure 6, which we attribute to the fact that $n$-way
classification is more challenging than 1-way.

##  5 Conclusion

In this paper we proposed a sample efficient method to segment unseen classes
via multi-resolution imprinting of adaptive masked proxies (AMP). AMP
constructs the final segmentation layer weights from few labelled support set
samples by imprinting the masked multi-resolution response of the base feature
extractor and by fusing it with the previously learned class signatures. AMP
is empirically validated to be superior in the few-shot segmentation on
PASCAL-$5^{i}$ with 5.5% in 5-shot case. It is also validated on video object
segmentation on DAVIS’16 as well as on the proposed iPASCAL.

## References

  * [1] David S. Bolme, J. Ross Beveridge, Bruce A. Draper, and Yui Man Lui.  Visual object tracking using adaptive correlation filters.  In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 2544–2550. IEEE, 2010.
  * [2] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 132–149, 2018.
  * [3] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248–255. Ieee, 2009.
  * [4] Nanqing Dong and Eric P. Xing.  Few-shot semantic segmentation with prototype learning.  In BMVC, volume 3, page 4, 2018.
  * [5] Mark Everingham, S.M. Ali Eslami, Luc Van Gool, Christopher K.I. Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes challenge: A retrospective.  International journal of computer vision, 111(1):98–136, 2015.
  * [6] Li Fei-Fei, Rob Fergus, and Pietro Perona.  One-shot learning of object categories.  IEEE transactions on pattern analysis and machine intelligence, 28(4):594–611, 2006.
  * [7] João F. Henriques, Rui Caseiro, Pedro Martins, and Jorge Batista.  High-speed tracking with kernelized correlation filters.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3):583–596, 2015.
  * [8] Charles F. Hester and David Casasent.  Multivariant technique for multiclass pattern recognition.  Applied Optics, 19(11):1758–1761, 1980.
  * [9] Geoff Hinton.  Neural Networks for Machine Learning, Lecture Notes: overview of mini-batch gradient descent.  URL: https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf.
  * [10] Suyog Dutt Jain, Bo Xiong, and Kristen Grauman.  Fusionseg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos.  arXiv preprint arXiv:1701.05384, 2017.
  * [11] Margret Keuper, Bjoern Andres, and Thomas Brox.  Motion trajectory segmentation via minimum cost multicuts.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3271–3279, 2015.
  * [12] Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov.  Siamese neural networks for one-shot image recognition.  In ICML Deep Learning Workshop, volume 2, 2015.
  * [13] Yeong Jun Koh and Chang-Su Kim.  Primary object segmentation in videos based on region augmentation and reduction.
  * [14] Philipp Krähenbühl and Vladlen Koltun.  Efficient inference in fully connected crfs with gaussian edge potentials.  In Advances in neural information processing systems, pages 109–117, 2011.
  * [15] Ce Liu.  Beyond pixels: exploring new representations and applications for motion analysis.  PhD thesis, Massachusetts Institute of Technology, 2009.
  * [16] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [17] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-SNE.  Journal of machine learning research, 9(Nov):2579–2605, 2008.
  * [18] Ellen M. Markman.  Categorization and naming in children: Problems of induction.  MIT Press, 1989.
  * [19] Yair Movshovitz-Attias, Alexander Toshev, Thomas K. Leung, Sergey Ioffe, and Saurabh Singh.  No fuss distance metric learning using proxies.  In Proceedings of the IEEE International Conference on Computer Vision, pages 360–368, 2017.
  * [20] Peter Ochs, Jitendra Malik, and Thomas Brox.  Segmentation of moving objects by long term video analysis.  IEEE transactions on pattern analysis and machine intelligence, 36(6):1187–1200, 2014.
  * [21] Anestis Papazoglou and Vittorio Ferrari.  Fast object segmentation in unconstrained video.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1777–1784, 2013.
  * [22] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung.  A benchmark dataset and evaluation methodology for video object segmentation.  In Computer Vision and Pattern Recognition, 2016.
  * [23] Hang Qi, Matthew Brown, and David G Lowe.  Learning with imprinted weights.  arXiv preprint arXiv:1712.07136, 2017.
  * [24] Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan Yuille.  Few-shot image recognition by predicting parameters from activations.  arXiv preprint arXiv:1706.03466, 2, 2017.
  * [25] Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alyosha Efros, and Sergey Levine.  Conditional networks for few-shot semantic segmentation.  2018\.
  * [26] Sachin Ravi and Hugo Larochelle.  Optimization as a model for few-shot learning.  In ICLR, 2017.
  * [27] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap.  Meta-learning with memory-augmented neural networks.  In ICML, pages 1842–1850, 2016.
  * [28] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots.  One-shot learning for semantic segmentation.  arXiv preprint arXiv:1709.03410, 2017.
  * [29] Meet P. Shah.  Semantic segmentation architectures implemented in PyTorch.  https://github.com/meetshah1995/pytorch-semseg, 2017.
  * [30] Mennatullah Siam, Chen Jiang, Steven Lu, Laura Petrich, Mahmoud Gamal, Mohamed Elhoseiny, and Martin Jagersand.  Video segmentation using teacher-student adaptation in a human robot interaction (hri) setting.  arXiv preprint arXiv:1810.07733, 2018.
  * [31] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [32] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  In Advances in Neural Information Processing Systems, pages 4077–4087, 2017.
  * [33] Hongmei Song, Wenguan Wang, Sanyuan Zhao, Jianbing Shen, and Kin-Man Lam.  Pyramid dilated deeper convlstm for video salient object detection.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 715–731, 2018.
  * [34] Brian Taylor, Vasiliy Karasev, and Stefano Soatto.  Causal video object segmentation from persistence of occlusions.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4268–4276, 2015.
  * [35] Pavel Tokmakov, Karteek Alahari, and Cordelia Schmid.  Learning motion patterns in videos.  arXiv preprint arXiv:1612.07217, 2016.
  * [36] Pavel Tokmakov, Karteek Alahari, and Cordelia Schmid.  Learning video object segmentation with visual memory.  arXiv preprint arXiv:1704.05737, 2017.
  * [37] Gido M. van de Ven and Andreas S. Tolias.  Three scenarios for continual learning.  CoRR, abs/1904.07734, 2019.
  * [38] Oriol Vinyals, Charles Blundell, Tim Lillicrap, and Daan Wierstra.  Matching networks for one shot learning.  In Advances in Neural Information Processing Systems, pages 3630–3638, 2016.
  * [39] Zifeng Wu, Chunhua Shen, and Anton van den Hengel.  Wider or deeper: Revisiting the resnet model for visual recognition.  arXiv preprint arXiv:1611.10080, 2016.
  * [40] Fisher Yu and Vladlen Koltun.  Multi-scale context aggregation by dilated convolutions.  arXiv preprint arXiv:1511.07122, 2015.
  * [41] Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas Huang.  SG-One: Similarity guidance network for one-shot semantic segmentation.  arXiv preprint arXiv:1810.09091, 2018.