* (9) M. Bershtein, G. Bonelli, M. Ronzani, and A. Tanzini, “Exact results for $\mathcal{N}$ = 2 supersymmetric gauge theories on compact toric manifolds and equivariant Donaldson invariants,” JHEP 07 (2016) 023, arXiv:1509.00267 [hep-th].
  * (10) M. Bershtein, G. Bonelli, M. Ronzani, and A. Tanzini, “Gauge theories on compact toric surfaces, conformal field theories and equivariant Donaldson invariants,” J. Geom. Phys. 118 (2017) 40–50, arXiv:1606.07148 [hep-th].
  * (11) G. Bonelli, F. Fucito, J. F. Morales, M. Ronzani, E. Sysoeva, and A. Tanzini, “Gauge theories on compact toric manifolds,” Lett. Math. Phys. 111 no. 3, (2021) 77, arXiv:2007.15468 [hep-th].
  * (12) S. M. Hosseini, I. Yaakov, and A. Zaffaroni, “Topologically twisted indices in five dimensions and holography,” JHEP 11 (2018) 119, arXiv:1808.06626 [hep-th].
  * (13) G. Festuccia, J. Qiu, J. Winding, and M. Zabzine, “Twisting with a Flip (the Art of Pestunization),” Commun. Math. Phys. 377 no. 1, (2020) 341–385, arXiv:1812.06473 [hep-th].
  * (14) G. Festuccia, J. Qiu, J. Winding, and M. Zabzine, “Transversally Elliptic Complex and Cohomological Field Theory,” J. Geom. Phys. 156 (2020) 103786, arXiv:1904.12782 [hep-th].
  * (15) G. Festuccia, A. Gorantis, A. Pittelli, K. Polydorou, and L. Ruggeri, “Cohomological localization of $\mathcal{N}$ = 2 gauge theories with matter,” JHEP 09 (2020) 133, arXiv:2005.12944 [hep-th].
  * (16) N. Nekrasov, “Localizing gauge theories,” XIVth International Congress on Mathematical Physics (2003) 645–654.
  * (17) H. Nakajima and K. Yoshioka, “Instanton counting on blowup. 1.,” Invent. Math. 162 (2005) 313–355, arXiv:math/0306198.
  * (18) H. Nakajima and K. Yoshioka, “Lectures on instanton counting,” in CRM Workshop on Algebraic Structures and Moduli Spaces.  11, 2003.  arXiv:math/0311058.
  * (19) L. Gottsche, H. Nakajima, and K. Yoshioka, “Instanton counting and Donaldson invariants,” J. Diff. Geom. 80 no. 3, (2008) 343–390, arXiv:math/0606180.
  * (20) P. M. Crichigno, D. Jain, and B. Willett, “5d Partition Functions with A Twist,” JHEP 11 (2018) 058, arXiv:1808.06744 [hep-th].
  * (21) S. Pasquetti, “Factorisation of N = 2 Theories on the Squashed 3-Sphere,” JHEP 04 (2012) 120, arXiv:1111.6905 [hep-th].
  * (22) C. Beem, T. Dimofte, and S. Pasquetti, “Holomorphic Blocks in Three Dimensions,” JHEP 12 (2014) 177, arXiv:1211.1986 [hep-th].
  * (23) J. Källén and M. Zabzine, “Twisted supersymmetric 5D Yang-Mills theory and contact geometry,” JHEP 05 (2012) 125, arXiv:1202.1956 [hep-th].
  * (24) J. Källén, J. Qiu, and M. Zabzine, “The perturbative partition function of supersymmetric 5D Yang-Mills theory with matter on the five-sphere,” JHEP 08 (2012) 157, arXiv:1206.6008 [hep-th].
  * (25) Y. Imamura, “Perturbative partition function for squashed $S^{5}$,” PTEP 2013 no. 7, (2013) 073B01, arXiv:1210.6308 [hep-th].
  * (26) G. Lockhart and C. Vafa, “Superconformal Partition Functions and Non-perturbative Topological Strings,” JHEP 10 (2018) 051, arXiv:1210.5909 [hep-th].
  * (27) H.-C. Kim, J. Kim, and S. Kim, “Instantons on the 5-sphere and M5-branes,” arXiv:1211.0144 [hep-th].
  * (28) J. Qiu and M. Zabzine, “5D Super Yang-Mills on $Y^{p,q}$ Sasaki-Einstein manifolds,” Commun. Math. Phys. 333 no. 2, (2015) 861–904, arXiv:1307.3149 [hep-th].
  * (29) J. Qiu and M. Zabzine, “Factorization of 5D super Yang-Mills theory on $Y^{p,q}$ spaces,” Phys. Rev. D 89 no. 6, (2014) 065040, arXiv:1312.3475 [hep-th].
  * (30) J. Qiu, L. Tizzano, J. Winding, and M. Zabzine, “Gluing Nekrasov partition functions,” Commun. Math. Phys. 337 no. 2, (2015) 785–816, arXiv:1403.2945 [hep-th].
  * (31) L. F. Alday, P. Benetti Genolini, M. Fluder, P. Richmond, and J. Sparks, “Supersymmetric gauge theories on five-manifolds,” JHEP 08 (2015) 007, arXiv:1503.09090 [hep-th].
  * (32) G. Festuccia, J. Qiu, J. Winding, and M. Zabzine, “$\mathcal{N}=2$ supersymmetric gauge theory on connected sums of $S^{2}\times S^{2}$,” JHEP 03 (2017) 026, arXiv:1611.04868 [hep-th].
  * (33) J. Qiu and M. Zabzine, “Review of localization for 5d supersymmetric gauge theories,” J. Phys. A 50 no. 44, (2017) 443014, arXiv:1608.02966 [hep-th].
  * (34) F. Benini and A. Zaffaroni, “A topologically twisted index for three-dimensional supersymmetric theories,” JHEP 07 (2015) 127, arXiv:1504.03698 [hep-th].
  * (35) C. Closset, S. Cremonesi, and D. S. Park, “The equivariant A-twist and gauged linear sigma models on the two-sphere,” JHEP 06 (2015) 076, arXiv:1504.06308 [hep-th].
  * (36) J. Lundin and L. Ruggeri, “SYM on quotients of spheres and complex projective spaces,” JHEP 03 (2022) 204, arXiv:2110.13065 [hep-th].
  * (37) C. P. Boyer and K. Galicki, Sasakian Geometry.  Oxford University Press, Oxford, 2008.
  * (38) W. Fulton, Introduction to Toric Varieties.  Princeton University Press, Oxford, 1993.
  * (39) D. McDuff and D. Salamon, Introduction to Symplectic Topology.  Oxford University Press, Oxford, third ed., 2017.
  * (40) E. Lerman, “Contact toric manifolds,” J. Symplectic Geom. 1 no. 4, (2001) 785, arXiv:math/0107201.
  * (41) J. Sparks, “Sasaki-Einstein Manifolds,” Surveys Diff. Geom. 16 (2011) 265–324, arXiv:1004.2461 [math.DG].
  * (42) C. P. Boyer and K. Galicki, “A note on toric contact geometry,” Journal of Geometry and Physics 35 no. 4, (2000) 288–298, arXiv:math/9907043.
  * (43) D. Martelli, J. Sparks, and S.-T. Yau, “The Geometric dual of a-maximisation for Toric Sasaki-Einstein manifolds,” Commun. Math. Phys. 268 (2006) 39–65, arXiv:hep-th/0503183.
  * (44) M. Cvetic, H. Lu, D. N. Page, and C. N. Pope, “New Einstein-Sasaki spaces in five and higher dimensions,” Phys. Rev. Lett. 95 (2005) 071101, arXiv:hep-th/0504225.
  * (45) J. P. Gauntlett, D. Martelli, J. Sparks, and D. Waldram, “Sasaki-Einstein metrics on S**2 x S**3,” Adv. Theor. Math. Phys. 8 no. 4, (2004) 711–734, arXiv:hep-th/0403002.
  * (46) D. Martelli and J. Sparks, “Toric geometry, Sasaki-Einstein manifolds and a new infinite class of AdS/CFT duals,” Commun. Math. Phys. 262 (2006) 51–89, arXiv:hep-th/0411238.
  * (47) C. P. Boyer, “Completely Integrable Contact Hamiltonian Systems and Toric Contact Structures on $S^{2}\times S^{3}$,” SIGMA 7 (2011) 058, arXiv:1101.5587 [math.SG].
  * (48) G. Festuccia and N. Seiberg, “Rigid Supersymmetric Theories in Curved Superspace,” JHEP 06 (2011) 114, arXiv:1105.0689 [hep-th].
  * (49) C. H. Taubes, Differential Geometry: Bundles, Connections, Metrics and Curvature.  Oxford University Press, 2011.
  * (50) N. Kurokawa and S. ya Koyama, “Multiple sine functions,” Forum Math. 15 (2003) 839–876.
  * (51) A. Narukawa, “The modular properties and the integral representations of the multiple elliptic gamma functions,” Advances in Mathematics 189 no. 2, (2004) 247–267, arXiv:math/0306164 [math].
  * (52) N. Hama and K. Hosomichi, “Seiberg-Witten Theories on Ellipsoids,” JHEP 09 (2012) 033, arXiv:1206.6359 [hep-th]. [Addendum: JHEP 10, 051 (2012)].
  * (53) R. Mauch and L. Ruggeri, “Index of the transversally elliptic complex in Pestunization,” J. Phys. A 55 no. 29, (2022) 295401, arXiv:2112.10658 [hep-th].
  * (54) J. Schmude, “Localisation on Sasaki-Einstein manifolds from holomorphic functions on the cone,” JHEP 01 (2015) 119, arXiv:1401.3266 [hep-th].
  * (55) P. Jefferson, S. Katz, H.-C. Kim, and C. Vafa, “On Geometric Classification of 5d SCFTs,” JHEP 04 (2018) 103, arXiv:1801.04036 [hep-th].
  * (56) E. Witten, “Some comments on string dynamics,” in STRINGS 95: Future Perspectives in String Theory, pp. 501–523.  7, 1995.  arXiv:hep-th/9507121.
  * (57) A. Strominger, “Open p-branes,” Phys. Lett. B 383 (1996) 44–47, arXiv:hep-th/9512059.
  * (58) N. Seiberg, “Nontrivial fixed points of the renormalization group in six-dimensions,” Phys. Lett. B 390 (1997) 169–171, arXiv:hep-th/9609161.
  * (59) G. Festuccia and M. Zabzine, “S-duality and supersymmetry on curved manifolds,” JHEP 09 (2020) 128, arXiv:2007.12001 [hep-th].
  * (60) P. Ferrero, J. P. Gauntlett, J. M. Pérez Ipiña, D. Martelli, and J. Sparks, “D3-Branes Wrapped on a Spindle,” Phys. Rev. Lett. 126 no. 11, (2021) 111601, arXiv:2011.10579 [hep-th].
  * (61) M. Inglese, D. Martelli, and A. Pittelli, “The Spindle Index from Localization,” arXiv:2303.14199 [hep-th].
  * (62) F. Faedo and D. Martelli, “D4-branes wrapped on a spindle,” JHEP 02 (2022) 101, arXiv:2111.13660 [hep-th].
  * (63) F. Faedo, A. Fontanarossa, and D. Martelli, “Branes wrapped on orbifolds and their gravitational blocks,” arXiv:2210.16128 [hep-th].
  * (64) K. Polydorou, A. Rocén, and M. Zabzine, “7D supersymmetric Yang-Mills on curved manifolds,” JHEP 12 (2017) 152, arXiv:1710.09653 [hep-th].
  * (65) A. Rocén, “7D supersymmetric Yang-Mills on a 3-Sasakian manifold,” JHEP 11 (2018) 024, arXiv:1808.06917 [hep-th].
  * (66) N. Iakovidis, J. Qiu, A. Rocén, and M. Zabzine, “7D supersymmetric Yang-Mills on hypertoric 3-Sasakian manifolds,” JHEP 06 (2020) 026, arXiv:2003.12461 [hep-th].
  * (67) A. Hatcher, Algebraic Topology.  Cambridge University Press, 2002.
  * (68) A. Putman, “The picard group of the moduli space of curves with level structures,” Duke Math. J. 161 no. 4, (2012) , arXiv:0908.0555.


# Cost Aggregation Is All You Need for Few-Shot Segmentation

Sunghwan Hong∗
Korea University  Seokju Cho
Yonsei University Equal contribution Jisu Nam
Korea University  Seungryong Kim
Korea University Corresponding author

###### Abstract

We introduce a novel cost aggregation network, dubbed Volumetric Aggregation
with Transformers (VAT), to tackle the few-shot segmentation task by using
both convolutions and transformers to efficiently handle high dimensional
correlation maps between query and support. In specific, we propose our
encoder consisting of volume embedding module to not only transform the
correlation maps into more tractable size but also inject some convolutional
inductive bias and volumetric transformer module for the cost aggregation. Our
encoder has a pyramidal structure to let the coarser level aggregation to
guide the finer level and enforce to learn complementary matching scores. We
then feed the output into our affinity-aware decoder along with the projected
feature maps for guiding the segmentation process. Combining these components,
we conduct experiments to demonstrate the effectiveness of the proposed
method, and our method sets a new state-of-the-art for all the standard
benchmarks in few-shot segmentation task. Furthermore, we find that the
proposed method attains state-of-the-art performance even for the standard
benchmarks in semantic correspondence task although not specifically designed
for this task. We also provide an extensive ablation study to validate our
architectural choices. The trained weights and codes are available at:
https://seokju-cho.github.io/VAT/.

##  1 Introduction

Semantic segmentation is one of the fundamental Computer Vision tasks which
aims to label each pixel in an image with a corresponding class. With the
advent of deep networks and the availability of large-scale datasets with
ground-truth segmentation annotations, substantial progress has been made in
this task [34, 44, 4, 5, 60]. However, as such advances can be attributed to
abundant pixel-wise segmentation maps made by manual annotation, which is
often labor-intensive, few-shot segmentation task [48, 54] has been introduced
to address this, where only a handful of support samples are provided to make
a mask prediction for a query, which mitigates the reliance on the labeled
data.

Figure 1: Our VAT reformulates few-shot segmentation task to semantic
correspondence task. VAT not only sets state-of-the-art in few-shot
segmentation (top), but also attains highly competitive performance for
semantic correspondence (bottom).

For a decade, numerous methods for few-shot segmentation have been proposed
[67, 76, 32, 30, 38, 37, 77, 75, 72, 74, 25, 78, 2], and most methods follow a
learning-to-learn paradigm [48] to avoid the risk of overfitting due to
insufficient training data. Since the prediction for query image should be
conditioned on support images and corresponding masks, the key to this task is
how to effectively utilize provided support samples. Although formulated in
various ways, most early efforts [57, 8, 32, 76] attempted to utilize a
prototype extracted from support samples. However, such approaches disregard
pixel-wise relationships between support and query features or spatial
structure of features, which may lead to sub-optimal results.

In light of this, we argue that few-shot segmentation task can be reformulated
as semantic correspondence that aims to find pixel-level correspondences
across semantically similar images, which poses some challenges from large
intra-class appearance and geometric variations [11, 12, 41]. Recent
approaches for this task [49, 22, 50, 52, 40, 42, 31, 63, 39] carefully
designed their models analogously to the classical matching pipeline [53, 46],
i.e., feature extraction, cost aggregation and flow estimation. Especially,
the latest works focused on cost aggregation stage [52, 40, 51, 20, 31, 26,
39, 6] and showed outstanding performance.

Taking similar approaches, recent few-shot segmentation methods [79, 67, 74,
29, 81] also attempted to leverage pixel-wise information by refining features
either by using cross-attention [81] or graph attention [79, 67, 74]. However,
as proven in semantic correspondence literature [39, 6], without aggregating
the matching scores, and solely relying on raw correlation maps between
features may suffer from the challenges posed due to ambiguities generated by
repetitive patterns or background clutters [49, 22, 24, 63, 15]. To address
this, one of the latest work, HSNet [38], attempts to aggregate the matching
scores with 4D convolutions, but it lacks an ability to consider an
interaction among the matching scores due to the inherent nature of
convolutions.

In this paper, we introduce a novel cost aggregation network, dubbed
Volumetric Aggregation with Transformers (VAT), to tackle the few-shot
segmentation task by using both convolutions and transformers to efficiently
handle high dimensional correlation maps between query and support.
Specifically, within our encoder, we propose volumetric embedding module
consisting of a series of 4D convolutions to not only transform high-
dimensional correlation maps into more tractable size, but also inject some
convolutional bias to aid the subsequent processing by transformers [73]. The
output then undergoes volumetric transformer module for the cost aggregation
with 4D swin transformer. With these combined, our encoder processes the input
in a pyramidal manner to expedite the learning by letting the aggregated
correlation maps at coarser level play as a guidance to finer level, enforcing
to learn complementary matching scores. Subsequent to encoder, our affinity-
aware decoder refines the aggregated costs with the help from appearance
affinity and makes a prediction.

We demonstrate the effectiveness of our method on several benchmarks [54, 28,
27]. Although not specifically designed for semantic correspondence task, our
work attains state-of-the-art performance on all the benchmarks for few-shot
segmentation and achieves highly competitive results even for semantic
correspondence, showing its superiority over the recently proposed methods. We
also include a detailed ablation study to justify our choices.

##  2 Related Work

#### Few-shot Segmentation.

Inspired by few-shot learning paradigm [48, 57], which aims to learn-to-learn
a model for a novel task with only a limited number of samples, few-shot
segmentation has received considerable attention. Following the success of
[54], prototypical networks [57] and numerous other works [8, 43, 55, 68, 32,
76, 30, 75, 77, 59, 82, 25] proposed to utilize a prototype extracted from
support samples, which is used to refine the query features to contain the
relevant support information. In addition, inspired by [80] that observed the
use of high-level features leads to a performance drop, [62] proposed to
utilize high-level features by computing a prior map which takes maximum score
within a correlation map. Many variants [59, 78] extended this idea of
utilizing prior maps to guide the feature learning.

Figure 2: Overall network architecture. Our networks consist of feature
extraction and cost computation, pyramidal transformer encoder, and affinity-
aware transformer decoder. Given query and support images, we first extract
all the intermediate features extracted from backbone network and compute
multi-level correlation maps. They then undergo the encoder to aggregate the
matching scores with transformers in a pyramidal fashion. The decoder finally
predicts a mask label for a query image.

However, as methods based on prototypes or prior maps have apparent
limitations, e.g., disregarding pixel-wise relationships between support and
query features por spatial structure of feature maps, numerous recent works
[79, 67, 38, 74, 29] attempted to fully utilize a correlation map to leverage
the pixel-wise relationships between source and query features. Specifically,
[79, 67, 74] use graph attention, [38] proposes efficient 4D convolutions to
fully exploit the multi-level features, and [29] formulates the task as
optimal transport problem. However, these approaches either do not provide a
means to aggregate the matching scores or lack an ability to consider
interactions of matching scores.

As concurrent works, [81] utilizes transformers and proposes to use a cycle-
consistent attention mechanism to refine the feature maps to be more
discriminative, without considering aggregation of matching scores. [59]
propose global and local enhancement module to refine the features using
transformers and convolutions in the decoder, respectively. [37] focuses
solely on the transformer-based classifier by freezing the encoder and
decoder, and adapting only the classifier to the task. Unlike them, we take
different approach, focusing on aggregation of the high dimensional
correlation maps in a novel and efficient way.

#### Semantic Correspondence.

The objective of semantic correspondence is to find correspondences between
semantically similar images with additional challenges posed by large intra-
class appearance and geometric variations [31, 6, 39]. This is highly similar
to few-shot segmentation setting in that few-shot segmentation also aims to
label the objects of same class with large intra-class variations, and thus
the recent works in both tasks have been taking similar approaches. The latest
approaches [52, 40, 51, 20, 31, 26, 39, 6] in semantic correspondence focused
on cost aggregation stage to find reliable correspondences, and proved its
importance. Among those, [39] proposed to use 4D convolutions for cost
aggregation while it showed apparent limitations which include limited
receptive fields of convolutions. CATs [6] resolve this issue and sets a new
state-of-the-art by leveraging transformers [65] to aggregate the cost volume.
However, it suffers from high computation due to high dimensional nature of
correlation map. In this paper, we propose to resolve the aforementioned
issues.

##  3 Methodology

###  3.1 Problem Formulation

The goal of few-shot segmentation is to segment an object of unseen classes
from a query image given only a few annotated examples [54]. To mitigate the
overfitting caused by insufficient training data, we follow the common
protocol called episodic training [66]. Let us denote training and test sets
as $\mathcal{D}_{\mathrm{train}}$ and $\mathcal{D}_{\mathrm{test}}$,
respectively, where object classes of both sets do not overlap. Under $K$-shot
setting, multiple episodes are formed from both sets, each consisting of a
support set $\mathcal{S}=\\{(x_{s}^{k},m_{s}^{k})\\}_{k=1}^{\mathrm{K}}$,
where $(x_{s}^{k},m_{s}^{k})$ is $k$-th support image and its corresponding
mask pair and a query sample $\mathcal{Q}=(x_{q},m_{q})$, where $x_{q}$ and
$m_{q}$ are a query image and mask, respectively. During training, our model
takes a sampled episode from $\mathcal{D}_{\mathrm{train}}$, and learn a
mapping from $\mathcal{S}$ and $x_{q}$ to a prediction $m_{q}$. At inference,
our model predicts $\hat{m}_{q}$ for randomly sampled $\mathcal{S}$ and
$x_{q}$ from $\mathcal{D}_{\mathrm{test}}$.

###  3.2 Motivation and Overview

The key to few-shot segmentation is how to effectively utilize provided
support samples for a query image. While conventional methods [62, 59, 81, 77,
25] attempted to utilize global- or part-level prototypes extracted from
support features, recent methods [79, 67, 38, 74, 29, 81] attempted to
leverage pixel-wise relationships between query and support. One of the latest
work, HSNet [38], attempts to aggregate the matching scores with 4D
convolutions, but it lacks an ability to consider interactions among the
matching scores due to the inherent nature of convolutions.

Figure 3: Overview of volumetric embedding and transformer modules. Our VEM
aims to ease the subsequent processing of correlation maps by projecting the
correlation maps to reduce the spatial dimensions. The output of VEM then
undergoes VTM for cost aggregation.

To overcome these, we present a novel design, dubbed volumetric aggregation
with transformers (VAT), to effectively integrate information present in all
pixel-wise matching costs between query and support with transformers [65].
VAT is an encoder-decoder architecture. To compensate for huge complexity of
standard transformers [65] that prevents from directly applying to correlation
maps, for the encoder, we present volume embedding module to effectively
reduce the number of tokens while injecting some convolutional inductive bias
[73] and volume transformer module based on swin transformer [33]. We design
our encoder in a pyramidal fashion to let the output from coarser level cost
aggregation to guide the finer level. We then present our decoder that
utilizes appearance affinity to resolve the ambiguities in the correlation map
and expedite the learning process.

###  3.3 Feature Extraction and Cost Computation

We first extract features from query and support images and compute initial
cost between them. In specific, we follow [38] to exploit rich semantics
present in different feature levels, and build multi-level correlation maps.
Given query and support images, $x_{q}$ and $x_{s}$, we use CNNs [14, 56] to
produce a sequence of $L$ feature maps,
$\\{(F_{q}^{l},F_{s}^{l})\\}^{L}_{l=1}$, where $F_{q}^{l}$ and $F_{s}^{l}$
denote query and support feature maps at $l$-th level, respectively. We
utilize a support mask, $m_{s}$, to encode segmentation information and filter
out the background information as done in [25, 38, 78]. We obtain a masked
support feature such that $\hat{F}^{l}_{s}={F}^{l}_{s}\ \odot\
\psi^{l}{(m_{s})}$, where $\odot$ denotes Hadamard product and
$\psi^{l}(\cdot)$ denotes a function that resizes the given tensor followed by
expansion along channel dimension of $l$-th layer.

Given a pair of feature maps, $F^{l}_{q}$ and $F^{l}_{s}$, we compute a
correlation map using the inner product between L-2 normalized features such
that

$\mathcal{C}^{l}(i,j)=\mathrm{ReLU}\left(\frac{F^{l}_{q}(i)\cdot\hat{F}^{l}_{s}(j)}{\|F^{l}_{q}(i)\|\|\hat{F}^{l}_{s}(j)\|}\right),$
(1)

where $i$ and $j$ denote 2D spatial positions of feature maps, respectively.
As done in [38], we collect correlation maps computed from all the
intermediate features of same spatial size and stack them to obtain a
hypercorrelation
$\mathcal{C}_{p}=\\{\mathcal{C}^{l}\\}_{l\in\mathcal{L}_{p}}\in\mathbb{R}^{h_{q}\times
w_{q}\times h_{s}\times w_{s}\times|\mathcal{L}_{p}|}$, where $h_{q},w_{q}$
and $h_{s},w_{s}$ are height and width of feature maps of query and support,
respectively, and $\mathcal{L}_{p}$ is subset of CNN layer indices
$\\{1,...,L\\}$ at some pyramid layer $p$, indicating the correlation maps of
identical spatial size.

###  3.4 Pyramidal Transformer Encoder

In this section, we show how to effectively aggregate the hypercorrelation
with proposed Transformers-based architecture and how to extend this to
pyramidal architecture.

#### Volume Embedding Module.

Aggregating the hypercorrelation by considering all the query and support
spatial dimensions, i.e., $h_{q}\times w_{q}\times h_{s}\times w_{s}$, as
tokens requires extremely large computation, which has to be resolved to fully
exploit all the pixel-wise interactions present in the correlation maps.
Perhaps the most straightforward way to reduce the resolutions is to use 4D
spatial pooling across query and support spatial dimensions, but this strategy
risks losing some information. As an alternative, one can split the
hypercorrelation into non-overlapping tensors and embed with a large learnable
kernel similarly to a patch embedding in ViT [9], but this demands a
substantial amount of resources due to the curse of dimensionality.
Furthermore, unlike image classification task [9] which finds a global
representation of an image, segmentation aiming for dense prediction needs to
consider overlapping neighborhood information.

To alleviate these limitations, as illustrated in Fig. 3, we introduce Volume
Embedding Module (VEM) to not only reduce the computation by effectively
decreasing the number of tokens, but also inject some convolutional inductive
bias [73] to help the subsequent transformer-based model to improve the
ability of learning interactions among the hypercorrelation. Concretely, we
sequentially reduce support and query spatial dimensions by applying 4D
spatial max-pooling, overlapping 4D convolutions, RELU, and Group
Normalization (GN), where we project the multi-level similarity vector at each
4D position, i.e., projecting a vector size of $|\mathcal{L}_{p}|$ to a
arbitrary fixed dimension denoted as $D$. Considering receptive fields of VEM
as 4D window size, i.e., $m\times m\times m\times m$, we build a tensor
$\mathcal{M}\in\mathbb{R}^{\hat{h}_{q}\times\hat{w}_{q}\times\hat{h}_{s}\times\hat{w}_{s}\times
D}$, where $\hat{h}$ and $\hat{w}$ are the processed sizes. Note that
different size of output can be made for source and target spatial dimensions
by varying the hyperparameters.

Overall, we define such a process as following:

$\mathcal{M}_{p}=\mathrm{VEM}(\mathcal{C}_{p}).\vspace{-5pt}$ (2)

#### Volumetric Transformer Module.

Although VEM reduces the number of tokens to some extent, directly applying
standard transformers [9] that has quadratic complexity with respect to number
of tokens is still challenging. Our Volumetric Transformer Module (VTM)
tackles this by extending swin transformer [33]. By setting a local 4D window,
the computation for computing self-attention can be significantly reduced,
without significant drop in performance. While [69, 71, 36] could also handle
a long sequence of tokens with linear complexity, we argue that as proven in
optical flow and semantic correspondence literature [58, 51] that neighboring
pixels tend to have similar correspondences, computing self-attention over
local regions and allowing cross-window interaction can help to find reliable
correspondences, which in turn yields better segmentation performance in our
framework. It should be noted that use of 4D convolutions [38] also considers
local consensus, but it lacks an ability to consider pixel-wise interactions
due to the use of fixed kernels during convolution while transformers
attentively explore pixel-wise interactions, which we validate our choice in
later section.

In specific, for the design of VTM, we extend the original 2D version of swin
transformer [33]. As illustrated in Fig. 4, we first evenly partition query
and support spatial dimensions of $\mathcal{M}_{p}$ into non-overlapping sub-
hypercorrelations $\mathcal{M}^{\prime}_{p}\in\mathbb{R}^{n\times n\times
n\times n\times D}$. We compute self-attention within each partitioned sub-
hypercorrelation. Subsequently, we shift the windows by displacement of
$(\lfloor\frac{n}{2}\rfloor,\lfloor\frac{n}{2}\rfloor,\lfloor\frac{n}{2}\rfloor,\lfloor\frac{n}{2}\rfloor)$
pixels from the previously partitioned windows, which we perform self-
attention within the newly created windows. Then as done in original swin
transformer [33], we simply roll it back to its original form without adopting
any complex implementation. In computing self-attention, we use relative
position bias and take the values from an expanded parameterized bias matrix,
following [17, 18, 33]. We leave other components in swin transformer blocks
unchanged, e.g., Layer Normalization (LN) [1] and MLP layers.

In addition, to stabilize the learning, we enforce our networks to estimate
the residual matching scores as complementary details. We add residual
connection in order to expedite the learning process [14, 6, 83], accounting
for fact that at the initial phase when the input $\mathcal{M}_{p}$ is fed,
erroneous matching scores are inferred due to randomly-initialized parameters
of transformers, which could complicate the learning process as the networks
need to learn the complete matching details from random matching scores.

To summarize, the overall process is defined as:

$\mathcal{A}_{p}=\mathrm{VTM}(\mathcal{M}_{p})=\mathcal{T}(\mathcal{M}_{p})+\mathcal{M}_{p},$
(3)

where $\mathcal{T}$ denotes transformer module.

Figure 4: Illustration of the shifted 4D window Our VTM computes self-
attention within the partitioned windows in high dimensional space, and it
also provides a means to consider inter-window interactions by shifting the 4D
window.

#### Pyramidal Processing.

Analogous to [38, 59], we also utilize the coarse-to-fine approach through a
pyramidal processing as illustrated in Fig. 2. Note that although it was
claimed that utilizing the high-level features has a negative effect on
performance [80], e.g., conv5$\\_$x, numerous recent works [81, 39, 6, 38] in
both semantic matching and few-shot segmentation task demonstrated that
leveraging multi-level features results in performance boost by large margin.
Motivated by this, we also use pyramidal hypercorrelation.

For the coarse-to-fine approach, we let the finer level aggregated correlation
map $\mathcal{A}_{p}$ to be guided by the aggregated correlation map of
previous (or deeper) levels $\mathcal{A}_{p+1}$. Concretely, aggregated
correlation map $\mathcal{A}_{p+1}$ is up-sampled, which is denoted as
$\mathrm{up}(\mathcal{A}_{p+1})$, and added to next level’s correlation map
$\mathcal{A}_{p}$ to play as a guidance. This process is repeated until the
finest level prior to decoder. This differs from [83, 38], which independently
aggregates each correlation map and fuses later, that at each level, they have
to learn the matching details from scratch while we are guided by the previous
level scores, which dramatically boosts the performance. The pyramidal process
is defined as:

$\mathcal{A}_{p}=\mathrm{VTM}(\mathcal{M}_{p}+\mathrm{up}(\mathcal{A}_{p+1})),$
(4)

where $\mathrm{up}(\cdot)$ denotes a bilinear upsampling.

###  3.5 Affinity-Aware Transformer Decoder

Given the hypercorrelation processed by pyramidal aggregation, we propose to
additionally utilize the appearance embedding obtained from query feature maps
to effectively decode the hypercorrelation to a query mask. From the
perspective of correlation maps, this setting can help finding more accurate
correspondence as the appearance affinity information helps to filter out the
erroneous matching scores, as proven in stereo matching literature, e.g., Cost
Volume Filtering (CVF) [16, 58]. Note that HSNet [38] only decodes the
matching costs with a series of 2D convolutions, thus often suffering from
ambiguities in the matching costs.

For the design of our decoder, we first take the average over support
dimensions of $\mathcal{A}_{p}$, which is then concatenated with appearance
embedding from query feature maps and processed by swin transformer [33]
followed by bilinear interpolation. The process is defined as following:

$\hat{\mathcal{A^{\prime}}}=\mathrm{Decoder}([\hat{\mathcal{A}},\mathcal{P}(F_{q})]),$
(5)

where $\hat{\mathcal{A}}\in\mathbb{R}^{\hat{h}_{q}\times\hat{w}_{q}\times D}$
extracted by average-pooling $\mathcal{A}_{p}$ in its support spatial
dimensions, $\mathcal{P}(\cdot)$ is linear projection,
$\mathcal{P}(F_{q})\in\mathbb{R}^{\hat{h}_{q}\times\hat{w}_{q}\times h}$, and
$[\,\cdot,\cdot\,]$ denotes concatenation. We sequentially refine the output
when immediately after bilinear upsampling to maximize preserving fine details
and integrating appearance information. $\hat{\mathcal{A}}$ is bilinearly
upsampled and undergoes Eq 5 until the projection head which outputs the
predicted mask $\hat{m}_{q}$.

###  3.6 Extension to $K$-Shot Setting

For $K>$ 1, given $K$ pairs of support image and mask
$\\{(x_{s}^{i},m_{s}^{i})\\}_{i=1}^{\mathrm{K}}$ and a query image $x_{q}$,
our model forward-passes $K$ times to obtain $K$ different query mask
$\hat{m}^{k}_{q}$. We sum up all the $K$ predictions, and find the maximum
number of predictions labelled as foreground across all the spatial locations.
If the output divided by $k$ is above threshold $\tau$, we label it as
foreground, otherwise background.

Backbone feature | Methods | 1-shot | 5-shot | # learnable
---|---|---|---|---
$5^{0}$ | $5^{1}$ | $5^{2}$ | $5^{3}$ | mIoU | FB-IoU | $5^{0}$ | $5^{1}$ | $5^{2}$ | $5^{3}$ | mIoU | FB-IoU | params
ResNet50 [14] | PANet [68] | 44.0 | 57.5 | 50.8 | 44.0 | 49.1 | - | 55.3 | 67.2 | 61.3 | 53.2 | 59.3 | - | 23.5M
PFENet [62] | 61.7 | 69.5 | 55.4 | 56.3 | 60.8 | 73.3 | 63.1 | 70.7 | 55.8 | 57.9 | 61.9 | 73.9 | 10.8M
ASGNet [25] | 58.8 | 67.9 | 56.8 | 53.7 | 59.3 | 69.2 | 63.4 | 70.6 | 64.2 | 57.4 | 63.9 | 74.2 | 10.4M
CWT [37] | 56.3 | 62.0 | 59.9 | 47.2 | 56.4 | - | 61.3 | 68.5 | 68.5 | 56.6 | 63.7 | - | -
RePRI [2] | 59.8 | 68.3 | 62.1 | 48.5 | 59.7 | - | 64.6 | 71.4 | 71.1 | 59.3 | 66.6 | - | -
HSNet [38] | 64.3 | 70.7 | 60.3 | 60.5 | 64.0 | 76.7 | 70.3 | 73.2 | 67.4 | 67.1 | 69.5 | 80.6 | 2.6M
CyCTR [81] | 67.8 | 72.8 | 58.0 | 58.0 | 64.2 | - | 71.1 | 73.2 | 60.5 | 57.5 | 65.6 | - | -
| VAT (ours) | 67.6 | 71.2 | 62.3 | 60.1 | 65.3 | 77.4 | 72.4 | 73.6 | 68.6 | 65.7 | 70.0 | 80.9 | 3.2M
ResNet101 [14] | FWB [43] | 51.3 | 64.5 | 56.7 | 52.2 | 56.2 | - | 54.8 | 67.4 | 62.2 | 55.3 | 59.9 | - | 43.0M
DAN [67] | 54.7 | 68.6 | 57.8 | 51.6 | 58.2 | 71.9 | 57.9 | 69.0 | 60.1 | 54.9 | 60.5 | 72.3 | -
PFENet [62] | 60.5 | 69.4 | 54.4 | 55.9 | 60.1 | 72.9 | 62.8 | 70.4 | 54.9 | 57.6 | 61.4 | 73.5 | 10.8M
ASGNet [25] | 59.8 | 67.4 | 55.6 | 54.4 | 59.3 | 71.7 | 64.6 | 71.3 | 64.2 | 57.3 | 64.4 | 75.2 | 10.4M
CWT [37] | 56.9 | 65.2 | 61.2 | 48.8 | 58.0 | - | 62.6 | 70.2 | 68.8 | 57.2 | 64.7 | - |
RePRI [2] | 59.6 | 68.6 | 62.2 | 47.2 | 59.4 | - | 66.2 | 71.4 | 67.0 | 57.7 | 65.6 | - | -
HSNet [38] | 67.3 | 72.3 | 62.0 | 63.1 | 66.2 | 77.6 | 71.8 | 74.4 | 67.0 | 68.3 | 70.4 | 80.6 | 2.6M
CyCTR [81] | 69.3 | 72.7 | 56.5 | 58.6 | 64.3 | 72.9 | 73.5 | 74.0 | 58.6 | 60.2 | 66.6 | 75.0 | -
| VAT (ours) | 68.4 | 72.5 | 64.8 | 64.2 | 67.5 | 78.8 | 73.3 | 75.2 | 68.4 | 69.5 | 71.6 | 82.0 | 3.3M

Table 1: Performance on PASCAL-5i [54] in mIoU and FB-IoU. Numbers in bold
indicate the best performance and underlined ones are the second best.

Backbone feature | Methods | 1-shot | 5-shot
---|---|---|---
$20^{0}$ | $20^{1}$ | $20^{2}$ | $20^{3}$ | mean | FB-IoU | $20^{0}$ | $20^{1}$ | $20^{2}$ | $20^{3}$ | mean | FB-IoU
ResNet50 [14] | PMM [76] | 29.3 | 34.8 | 27.1 | 27.3 | 29.6 | - | 33.0 | 40.6 | 30.3 | 33.3 | 34.3 | -
RPMM [76] | 29.5 | 36.8 | 28.9 | 27.0 | 30.6 | - | 33.8 | 42.0 | 33.0 | 33.3 | 35.5 | -
PFENet [62] | 36.5 | 38.6 | 34.5 | 33.8 | 35.8 | - | 36.5 | 43.3 | 37.8 | 38.4 | 39.0 | -
ASGNet [25] | - | - | - | - | 34.6 | 60.4 | - | - | - | - | 42.5 | 67.0
RePRI [2] | 32.0 | 38.7 | 32.7 | 33.1 | 34.1 | - | 39.3 | 45.4 | 39.7 | 41.8 | 41.6 | -
HSNet [38] | 36.3 | 43.1 | 38.7 | 38.7 | 39.2 | 68.2 | 43.3 | 51.3 | 48.2 | 45.0 | 46.9 | 70.7
CyCTR [81] | 38.9 | 43.0 | 39.6 | 39.8 | 40.3 | - | 41.1 | 48.9 | 45.2 | 47.0 | 45.6 | -
| VAT (ours) | 39.0 | 43.8 | 42.6 | 39.7 | 41.3 | 68.8 | 44.1 | 51.1 | 50.2 | 46.1 | 47.9 | 72.4

Table 2: Performance on COCO-20i [28] in mIoU and FB-IoU.

Backbone feature | Methods | mIoU
---|---|---
1-shot | 5-shot
ResNet50 [14] | FSOT [29] | 82.5 | 83.8
HSNet [38] | 85.5 | 87.8
| VAT (ours) | 89.5 | 90.3
ResNet101 [14] | DAN [67] | 85.2 | 88.1
HSNet [38] | 86.5 | 88.5
| VAT (ours) | 90.0 | 90.6

Table 3: Mean IoU comparison on FSS-1000 [27].

##  4 Experiments

###  4.1 Implementation Details

For backbone feature extractor, we use ResNet50 and ResNet101 [14] pre-trained
on ImageNet [7], which are frozen during training, following [38, 80]. We set
the threshold $\tau$ to $0.5$. We use data augmentation used in [6, 3] for
training. We use AdamW [35] with learning rate set to $5\mathrm{e}-4$. We set
$D$ to $128$. We use feature maps from conv3$\\_$x ($p=3$), conv4$\\_$x
($p=4$) and conv5$\\_$x ($p=5$) for cost computation. We use last layers from
conv2$\\_$x, conv3$\\_$x and conv4$\\_$x for appearance affinity when trained
on FSS-1000 [27] and conv4$\\_$x is excluded when trained on PASCAL-5i [54]
and COCO-20i [28]. The aforementioned hyperparameters are set with cross-
validation. More details can be found in supplementary material.

| Components | mIoU
---|---|---
| 1-shot | 5-shot
(I) | Baseline | 78.7 | 80.4
(II) | \+ VEM | 84.7 | 85.9
(III) | \+ VTM | 86.9 | 88.2
(IV) | \+ residual connection | 87.0 | 88.4
(V) | \+ appearance affinity | 90.0 | 90.6

Table 4: Ablation study of VAT.

Different aggregators | FSS-1000 [27]
---|---
mIoU (%)
Standard transformer [65] | OOM
Center-pivot 4D convolutions [38] | 88.1
Linear transformer [21] | 87.7
Fastformer [71] | 87.8
Volumetric transformer (ours) | 90.0

Table 5: Ablation study of cost aggregators. OOM: Out of Memory.

Backbones feature | FSS-1000 [27]
---|---
mIoU ($\%$)
1-shot | 5-shot
ResNet50 [14] | 89.5 | 90.3
ResNet101 [14] | 90.0 | 90.6
PVT [70] | 89.5 | 89.9
Swin transformer [33] | 89.1 | 89.4

Table 6: Ablation study of different feature backbone.

###  4.2 Experimental Settings

In this section, we conduct comprehensive experiments for few-shot
segmentation, by evaluating our approach through comparisons to recent state-
of-the-art methods including PMM [76], RPMM [76], FWB [43], OSLSM [54], PANet
[68], CANet [80], PFENet [62] DAN [67], RePRI [2], SAGNN [74], FSOT [29],
CyCTR [81], CWT [37], ASGNet [25], and HSNet [38]. In Section 4.3, we provide
a detailed analysis on our segmentation results evaluated on several
benchmarks, and we conduct an extensive ablation study to justify our
architectural choices and include an analysis of each component in Section
4.4.

#### Datasets.

We evaluate our approach on three standard few-shot segmentation datasets,
PASCAL-5i [54], COCO-20i [28], and FSS-1000 [27]. PASCAL-5i is created from
images from PASCAL VOC 2012 [10] and extra mask annotations [13], Originally,
20 object classes are available, but for cross-validation, as done in OSLM
[54], they are evenly divided into 4 folds $i\in\\{0,1,2,3\\}$, and this makes
each fold contains 5 classes. COCO-20i contains 80 object classes, and as done
for PASCAL-5i, the dataset is evenly divided into 4 folds, which results 20
classes for each fold. FSS-1000 is a more diverse dataset consisting of 1000
object classes. Following [27], we divide 1000 categories into 3 splits for
training, validation and testing, which consist of 520, 240 and 240 classes,
respectively. For PASCAL-5i and COCO-20i, we follow the common evaluation
practice [38, 62, 32] and standard cross-validation protocol; for each $i$-th
fold, the target fold $i$ is used for evaluation and other folds are used for
training.

#### Evaluation Metric.

Following common practice [80, 62, 38, 81], we adopt mean intersection over
union (mIoU) and foreground-background IoU (FB-IoU) as our evaluation metric.
The mIoU averages over all IoU values for all object classes such that
$\mathrm{mIoU}=\frac{1}{C}\sum_{c=1}^{C}\mathrm{IoU}_{c}$, where $C$ is the
number of classes in each fold, e.g., $C=20$ for COCO-20i. FB-IoU disregards
the object classes and averages over foreground and background IoU,
$\mathrm{IoU}_{\mathrm{F}}$ and $\mathrm{IoU}_{\mathrm{B}}$, such that
$\mathrm{FBIoU}=\frac{1}{2}(\mathrm{IoU}_{\mathrm{F}}+\mathrm{IoU}_{\mathrm{B}})$.
As stated in [80], we mainly focus on mIoU since it better accounts for
generalization power of a model than FB-IoU.

###  4.3 Segmentation Results

Table 1 summarizes quantitative results on the PASCAL-5i [54]. We denote the
type of backbone feature extractor and number of parameters for comparison.
For PASCAL-5i, we tested with two backbone networks, ResNet50 and ResNet101
[14]. The proposed method outperforms other methods for almost all the folds
with respect to both mIoU and FB-IoU. Consistent with this, VAT also attains
state-of-the-art performance on COCO-20i [28] for both 1-shot and 5-shot.
Interestingly, for the most recently introduced dataset deliberately created
for few-shot segmentation task, FSS-1000 [27], VAT outperforms HSNet [38] and
FSOT [29] by a large margin, almost 4$\%$ increase in mIoU compared to HSNet
with ResNet50. Overall, VAT sets a new state-of-the-art for all the benchmarks
and this is confirmed at both Table 3 and Table 3. The qualitative results are
shown in Fig. 5.

Another remarkable point is that, we find that even with larger number of
learnable parameter than [38], our method still outperforms. As shown in Table
1, the models with larger number of parameters tend to perform worse. It is
well known that the number of parameters has inverse relation to
generalization power [61]. However, VAT successfully avoids this by focusing
on cost aggregation, showing that simply reducing the number of parameters may
not be the answer.

Figure 5: Qualitative results on FSS-1000 [27].

###  4.4 Ablation study

In this section, we show an ablation analysis to justify the architectural
choices we made, explore impact of feature backbone and investigate whether
VAT is transferable to semantic correspondence task. Throughout this section,
all the experiments are conducted on FSS-1000 [27] datasets with ResNet-101 as
backbone feature extractor unless specified. Each ablation experiment is
conducted under same experimental setting for a fair comparison. For semantic
correspondence ablation study, we use percentage of correct keypoints (PCK) as
our evaluation metric.

#### Effectiveness of each component in VAT.

We consider these as core components: volume embedding module, volumetric
transformer module, residual connection and appearance affinity. We define the
baseline for this ablation study as a model that replaces VEM with simple 4D
max-pooling, skips VTM to show the importance of cost aggregation, excludes
residual connection around cost aggregation module and disregards appearance
affinity. We evaluate this by adding components progressively.

As summarized in Table 6, each component helps to boost the performance.
Especially, from (I) to (III) and (IV) to (V), we observe significant
improvement. We find that leveraging appearance affinity helps to find
accurate correspondences, which in turn yields better segmentation
performance, and VEM followed by VTM not only eases the computational burden
but also boosts the performance.

#### Is VTM better than other aggregators?

As summarized in Table 6, we provide an ablation study to justify the use of
our pyramidal transformers for cost aggregation. To this end, it is perhaps
necessary to compare with other methods including 4D convolutions [52, 38] and
efficient transformers [71, 69]. As 4D convolution as in [38] also aggregates
the matching scores in a local manner similar to swin transformer [33], we
conjecture that it should perform similarly or even better. We investigate
this and report the results to confirm that VTM exceeds other aggregators.

For a fair comparison, we only replace VTM with other aggregators and leave
all the other components in our architecture unchanged. We observe that our
proposed method outperforms other aggregators by a large margin. Note that the
use of standard transformers for cost aggregation is inapplicable due to high
dimensional space of hypercorrelation. Interestingly, although center-pivot 4D
convolutions [38] also focus on locality as swin transformer [33], the
performance gap indicates that the ability to attentively consider pixel-wise
interactions during the self-attention computation is critical. Another
interesting point is that linear transformer [21] and fastformer [71] that
exploit from global receptive fields of transformers achieve similar
performance. This confirms that for the cost aggregation, our approach may be
more suitable.

Methods | SPair-71k [41] | PF-PASCAL [12] | PF-WILLOW [11]
---|---|---|---
$\alpha_{\text{bbox}}=0.1$ |  $\alpha_{\text{img}}$ = 0.1 |  $\alpha_{\text{bbox}}$ = 0.1
SCOT [31] | 35.6 | 85.4 | 76.0
CHM [39] | 46.3 | 91.6 | 79.4
MMNet [83] | 50.4 | 91.6 | -
PMNC [23] | 50.4 | 90.6 | -
CATs [6] | 49.9 | 92.6 | 79.2
VAT (ours) | 54.2 | 92.3 | 81.0
|  |  |

Table 7: Quantitative results on SPair-71k [41], PF-PASCAL [12] and PF-WILLOW
[11].

#### Does different feature extractor matter?

Conventional few-shot segmentation methods only utilized CNN-based feature
backbones [14] for extracting features. [80] observed that high-level features
contains semantics of objects which could lead to overfitting and not suitable
to use for the task of few-shot segmentation. Then the question naturally
arises. What about transformer-based backbone networks? As addressed in many
works [47, 9], CNN and transformers see images differently, which means that
the kinds of backbone networks may affect the performance significantly, but
this has never been explored in this task. We thus exploit several well-known
vision transformer architectures to explore the potential differences that
probably exist.

The results are summarized in Table 6. We were surprised to find that both
convolution- and transformer-based backbone networks attain similar
performance. We conjecture that although it has been widely studied that
convolutions and transformers see differently [47], as they are pre-trained on
the same dataset [7], the representations learned by models are almost alike.
Note that we only utilized backbones with pyramidal structure, and the results
may differ if other backbone networks are used, which we leave this
exploration for the future work.

#### Can VAT also perform well in semantic correspondence task?

To tackle the few-shot segmentation task, we reformulated it as finding
semantic correspondences under conditions of several challenges posed by large
intra-class variations and geometric deformations. This means that the
proposed method should be able to perform well in semantic correspondence task
at least to some extent. Here, we compare our method with other state-of-the-
art methods in semantic correspondence task and investigate the key aspect: To
what extent and how well can VAT perform in semantic correspondence task given
similar formulation?

For this ablation study, we made minor modifications to our model: i.e.,
output spatial resolution and embedded features. We refer the readers to
either Appendix or github page: https://seokju-cho.github.io/VAT/ for details.
Note that this is possible as our projection head also outputs the same
dimension as if a flow field is inferred. Following common protocol [40, 19,
39, 6], we use standard benchmarks for this task, which we trained our model
on training split of PF-PASCAL [12] when evaluated on test split of PF-PASCAL
[12] and PF-WILLOW [11], and trained on SPair-71k [41] when evaluated on
SPair-71k [41]. We fine-tuned the feature backbone for fair comparison. As
shown in Table 7 and Fig. 6, although not specifically designed for semantic
correspondence task, VAT either sets a new state-of-the-art [41, 11] or attain
the second highest PCK [12] for this task. This results show that cost
aggregation is a prime importance in both few-shot segmentation and semantic
correspondence.

(a) CATs [6]

(b) Ours

Figure 6: Qualitative results on PF-PASCAL [12].

##  5 Conclusion

In this paper, we have proposed, for the first time, to aggregate the high
dimensional correlation maps with both convolutions and transformers for few-
shot segmentation. For our pyramidal encoder, to handle the high computation
caused by direct aggregation of correlation maps, we introduced volume
embedding module, and for the cost aggregation, we proposed volumetric
transformer module. Then the output undergoes the proposed affinity-aware
decoder for prediction. We have shown that although the proposed method is not
specifically designed for semantic correspondence task, it attains state-of-
the-art performance for all the standard benchmarks for both few-shot
segmentation and semantic correspondence. Also, we have conducted extensive
ablation study.

## References

  * [1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [2] Malik Boudiaf, Hoel Kervadec, Ziko Imtiaz Masud, Pablo Piantanida, Ismail Ben Ayed, and Jose Dolz.  Few-shot segmentation without meta-learning: A good transductive inference is all you need?  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
  * [3] Alexander Buslaev, Vladimir I Iglovikov, Eugene Khvedchenya, Alex Parinov, Mikhail Druzhinin, and Alexandr A Kalinin.  Albumentations: fast and flexible image augmentations.  Information, 2020.
  * [4] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence, 2017\.
  * [5] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proceedings of the European conference on computer vision (ECCV), 2018.
  * [6] Seokju Cho, Sunghwan Hong, Sangryul Jeon, Yunsung Lee, Kwanghoon Sohn, and Seungryong Kim.  Cats: Cost aggregation transformers for visual correspondence.  In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
  * [7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009.
  * [8] Nanqing Dong and Eric P Xing.  Few-shot semantic segmentation with prototype learning.  In BMVC, 2018.
  * [9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [10] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International journal of computer vision, 2010.
  * [11] Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce.  Proposal flow.  In CVPR, 2016.
  * [12] Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce.  Proposal flow: Semantic correspondences from object proposals.  IEEE transactions on pattern analysis and machine intelligence, 2017\.
  * [13] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik.  Simultaneous detection and segmentation.  In European conference on computer vision. Springer, 2014.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
  * [15] Sunghwan Hong and Seungryong Kim.  Deep matching prior: Test-time optimization for dense correspondence.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  * [16] Asmaa Hosni, Christoph Rhemann, Michael Bleyer, Carsten Rother, and Margrit Gelautz.  Fast cost-volume filtering for visual correspondence and beyond.  PAMI, 2012.
  * [17] Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei.  Relation networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3588–3597, 2018.
  * [18] Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin.  Local relation networks for image recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3464–3473, 2019.
  * [19] Shuaiyi Huang, Qiuyue Wang, Songyang Zhang, Shipeng Yan, and Xuming He.  Dynamic context correspondence network for semantic alignment.  In ICCV, 2019.
  * [20] Sangryul Jeon, Dongbo Min, Seungryong Kim, Jihwan Choe, and Kwanghoon Sohn.  Guided semantic flow.  In ECCV. Springer, 2020.
  * [21] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret.  Transformers are rnns: Fast autoregressive transformers with linear attention.  In International Conference on Machine Learning, pages 5156–5165. PMLR, 2020.
  * [22] Seungryong Kim, Dongbo Min, Bumsub Ham, Sangryul Jeon, Stephen Lin, and Kwanghoon Sohn.  Fcss: Fully convolutional self-similarity for dense semantic correspondence.  In CVPR, 2017.
  * [23] Jae Yong Lee, Joseph DeGol, Victor Fragoso, and Sudipta N. Sinha.  Patchmatch-based neighborhood consensus for semantic correspondence.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  * [24] Junghyup Lee, Dohyung Kim, Jean Ponce, and Bumsub Ham.  Sfnet: Learning object-aware semantic correspondence.  In CVPR, 2019.
  * [25] Gen Li, Varun Jampani, Laura Sevilla-Lara, Deqing Sun, Jonghyun Kim, and Joongkyu Kim.  Adaptive prototype learning and allocation for few-shot segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8334–8343, 2021.
  * [26] Shuda Li, Kai Han, Theo W Costain, Henry Howard-Jenkins, and Victor Prisacariu.  Correspondence networks with adaptive neighbourhood consensus.  In CVPR, 2020.
  * [27] Xiang Li, Tianhan Wei, Yau Pun Chen, Yu-Wing Tai, and Chi-Keung Tang.  Fss-1000: A 1000-class dataset for few-shot segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  * [28] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, 2014.
  * [29] Weide Liu, Chi Zhang, Henghui Ding, Tzu-Yi Hung, and Guosheng Lin.  Few-shot segmentation with optimal transport matching and message flow.  arXiv preprint arXiv:2108.08518, 2021.
  * [30] Weide Liu, Chi Zhang, Guosheng Lin, and Fayao Liu.  Crnet: Cross-reference networks for few-shot segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  * [31] Yanbin Liu, Linchao Zhu, Makoto Yamada, and Yi Yang.  Semantic correspondence as an optimal transport problem.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  * [32] Yongfei Liu, Xiangyi Zhang, Songyang Zhang, and Xuming He.  Part-aware prototype network for few-shot semantic segmentation.  In European Conference on Computer Vision, pages 142–158. Springer, 2020.
  * [33] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  arXiv preprint arXiv:2103.14030, 2021.
  * [34] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, 2015.
  * [35] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [36] Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang, and Li Zhang.  Soft: Softmax-free transformer with linear complexity.  arXiv preprint arXiv:2110.11945, 2021.
  * [37] Zhihe Lu, Sen He, Xiatian Zhu, Li Zhang, Yi-Zhe Song, and Tao Xiang.  Simpler is better: Few-shot semantic segmentation with classifier weight transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  * [38] Juhong Min, Dahyun Kang, and Minsu Cho.  Hypercorrelation squeeze for few-shot segmentation.  arXiv preprint arXiv:2104.01538, 2021.
  * [39] Juhong Min, Seungwook Kim, and Minsu Cho.  Convolutional hough matching networks for robust and efficient visual correspondence.  arXiv preprint arXiv:2109.05221, 2021.
  * [40] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho.  Hyperpixel flow: Semantic correspondence with multi-layer neural features.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
  * [41] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho.  Spair-71k: A large-scale benchmark for semantic correspondence.  arXiv preprint arXiv:1908.10543, 2019.
  * [42] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho.  Learning to compose hypercolumns for visual correspondence.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16. Springer, 2020.
  * [43] Khoi Nguyen and Sinisa Todorovic.  Feature weighting and boosting for few-shot segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 622–631, 2019.
  * [44] Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han.  Learning deconvolution network for semantic segmentation.  In Proceedings of the IEEE international conference on computer vision, 2015.
  * [45] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * [46] James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman.  Object retrieval with large vocabularies and fast spatial matching.  In CVPR. IEEE, 2007.
  * [47] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy.  Do vision transformers see like convolutional neural networks?  arXiv preprint arXiv:2108.08810, 2021.
  * [48] Sachin Ravi and Hugo Larochelle.  Optimization as a model for few-shot learning.  2016\.
  * [49] Ignacio Rocco, Relja Arandjelovic, and Josef Sivic.  Convolutional neural network architecture for geometric matching.  In CVPR, 2017.
  * [50] Ignacio Rocco, Relja Arandjelović, and Josef Sivic.  End-to-end weakly-supervised semantic alignment.  In CVPR, 2018.
  * [51] Ignacio Rocco, Relja Arandjelović, and Josef Sivic.  Efficient neighbourhood consensus networks via submanifold sparse convolutions.  In ECCV, 2020.
  * [52] Ignacio Rocco, Mircea Cimpoi, Relja Arandjelović, Akihiko Torii, Tomas Pajdla, and Josef Sivic.  Neighbourhood consensus networks.  arXiv preprint arXiv:1810.10510, 2018.
  * [53] Daniel Scharstein and Richard Szeliski.  A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.  International journal of computer vision, 2002.
  * [54] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots.  One-shot learning for semantic segmentation.  arXiv preprint arXiv:1709.03410, 2017.
  * [55] Mennatullah Siam, Boris Oreshkin, and Martin Jagersand.  Adaptive masked proxies for few-shot segmentation.  arXiv preprint arXiv:1902.11123, 2019.
  * [56] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [57] Jake Snell, Kevin Swersky, and Richard S Zemel.  Prototypical networks for few-shot learning.  arXiv preprint arXiv:1703.05175, 2017.
  * [58] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.  Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume.  In CVPR, 2018.
  * [59] Guolei Sun, Yun Liu, Jingyun Liang, and Luc Van Gool.  Boosting few-shot semantic segmentation with transformers.  arXiv preprint arXiv:2108.02266, 2021.
  * [60] Andrew Tao, Karan Sapra, and Bryan Catanzaro.  Hierarchical multi-scale attention for semantic segmentation.  arXiv preprint arXiv:2005.10821, 2020.
  * [61] Igor V. Tetko, David J. Livingstone, and Alexander I. Luik.  Neural network studies, 1. comparison of overfitting and overtraining.  J. Chem. Inf. Comput. Sci., 35:826–833, 1995.
  * [62] Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia.  Prior guided feature enrichment network for few-shot segmentation.  IEEE Transactions on Pattern Analysis & Machine Intelligence, 2020\.
  * [63] Prune Truong, Martin Danelljan, and Radu Timofte.  Glu-net: Global-local universal network for dense flow and correspondences.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6258–6268, 2020.
  * [64] Prune Truong, Martin Danelljan, Luc Van Gool, and Radu Timofte.  Learning accurate dense correspondences and when to trust them.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
  * [65] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, 2017.
  * [66] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  Advances in neural information processing systems, 2016.
  * [67] Haochen Wang, Xudong Zhang, Yutao Hu, Yandan Yang, Xianbin Cao, and Xiantong Zhen.  Few-shot semantic segmentation with democratic attention networks.  In European Conference on Computer Vision, 2020.
  * [68] Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng.  Panet: Few-shot image semantic segmentation with prototype alignment.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
  * [69] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma.  Linformer: Self-attention with linear complexity.  arXiv preprint arXiv:2006.04768, 2020.
  * [70] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  arXiv preprint arXiv:2102.12122, 2021.
  * [71] Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang, and Xing Xie.  Fastformer: Additive attention can be all you need.  arXiv preprint arXiv:2108.09084, 2021.
  * [72] Zhonghua Wu, Xiangxi Shi, Guosheng Lin, and Jianfei Cai.  Learning meta-class memory for few-shot semantic segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 517–526, 2021.
  * [73] Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick.  Early convolutions help transformers see better.  arXiv preprint arXiv:2106.14881, 2021.
  * [74] Guo-Sen Xie, Jie Liu, Huan Xiong, and Ling Shao.  Scale-aware graph neural network for few-shot semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5475–5484, 2021.
  * [75] Guo-Sen Xie, Huan Xiong, Jie Liu, Yazhou Yao, and Ling Shao.  Few-shot semantic segmentation with cyclic memory network.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  * [76] Boyu Yang, Chang Liu, Bohao Li, Jianbin Jiao, and Qixiang Ye.  Prototype mixture models for few-shot semantic segmentation.  In European Conference on Computer Vision. Springer, 2020.
  * [77] Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, and Yang Gao.  Mining latent classes for few-shot segmentation.  arXiv preprint arXiv:2103.15402, 2021.
  * [78] Bingfeng Zhang, Jimin Xiao, and Terry Qin.  Self-guided and cross-guided learning for few-shot segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
  * [79] Chi Zhang, Guosheng Lin, Fayao Liu, Jiushuang Guo, Qingyao Wu, and Rui Yao.  Pyramid graph networks with connection attentions for region-based one-shot semantic segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9587–9595, 2019.
  * [80] Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen.  Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5217–5226, 2019.
  * [81] Gengwei Zhang, Guoliang Kang, Yunchao Wei, and Yi Yang.  Few-shot segmentation via cycle-consistent transformer.  arXiv preprint arXiv:2106.02320, 2021.
  * [82] Hui Zhang and Henghui Ding.  Prototypical matching and open set rejection for zero-shot semantic segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  * [83] Dongyang Zhao, Ziyang Song, Zhenghao Ji, Gangming Zhao, Weifeng Ge, and Yizhou Yu.  Multi-scale matching networks for semantic correspondence.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3354–3364, 2021.

Appendix

In this document, we provide detailed implementation details, more
quantitative results on semantic correspondence benchmarks, including
SPair-71k [41], PF-PASCAL [12], and PF-WILLOW [11], and more qualitative
results on all the benchmarks we used.

## Appendix A. Implementation details

#### Implementation Details.

We first extract backbone features using ResNet50 [14] and ResNet101 [14] from
conv3$\\_$x, conv4$\\_$x and conv5$\\_$x. This feature extracting scheme
results in 3 pyramidal layers. To compose VEM, we stack a series of $3\times
3\times 3\times 3$ convolutions followed by ReLU and Group Normalization. The
coarsest level correaltion map has a size of $8\times 8\times 8\times 8$ after
undergoing VEM. Finer levels, $p=4$ and $p=3$, have volume embeddings of size
$16\times 16\times 8\times 8$ and $32\times 32\times 8\times 8$, respectively.
We set the number of heads in VTM as 4. At $p=5$ level, we set the depth of
VTM as 4 and rest are set to 2. Window size $M$ is set to 4 at all levels of
VTM. We project the feature maps of query at last layer of level $p=4,p=3$ and
$p=2$ to have channels 64, 32 and 16 from 1024, 512 and 256, respectively. We
implemented our work with PyTorch [45]. We used learning rate of 5e-4. We used
augmentation introduced in [3] for training as shown in Table 5. We adopted
early-stopping for all the training, which we picked the model with best
validation score. We set the batch size as 16.

#### Components of VAT.

For the ablation study of components of VAT, we defined the baseline model
which substitutes VEM with 4D max-pooling. As VTM is a transformer module that
takes an input and outputs the same shape to input, it can simply be skipped.
Residual connections are simply removed to test its efficiency, and appearance
affinity is excluded at the decoder. This experiment is conducted in a
sequential manner, where each component is added progressively.

#### Different Cost Aggregators.

For the ablation study of cost aggregators, we used codes released by authors
of works, including standard transformer [65], center-pivot 4D convolutions
[38], linear transformer [21], and fastformer [71]. We simply replaced VTM
with those in order to make a fair comparison.

#### Different backbone networks.

Ablation study on different feature backbone follows similar procedure, as PVT
[70] and swin transformer [33] have pyramidal architecture that outputs
features of size and dimensions similar to ResNet [14].

Methods | aero. | bike | bird | boat | bott. | bus | car | cat | chai. | cow | dog | hors. | mbik. | pers. | plan. | shee. | trai. | tv | all
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CNNGeo [49] | 23.4 | 16.7 | 40.2 | 14.3 | 36.4 | 27.7 | 26.0 | 32.7 | 12.7 | 27.4 | 22.8 | 13.7 | 20.9 | 21.0 | 17.5 | 10.2 | 30.8 | 34.1 | 20.6
WeakAlign [50] | 22.2 | 17.6 | 41.9 | 15.1 | 38.1 | 27.4 | 27.2 | 31.8 | 12.8 | 26.8 | 22.6 | 14.2 | 20.0 | 22.2 | 17.9 | 10.4 | 32.2 | 35.1 | 20.9
NC-Net [52] | 17.9 | 12.2 | 32.1 | 11.7 | 29.0 | 19.9 | 16.1 | 39.2 | 9.9 | 23.9 | 18.8 | 15.7 | 17.4 | 15.9 | 14.8 | 9.6 | 24.2 | 31.1 | 20.1
HPF [40] | 25.2 | 18.9 | 52.1 | 15.7 | 38.0 | 22.8 | 19.1 | 52.9 | 17.9 | 33.0 | 32.8 | 20.6 | 24.4 | 27.9 | 21.1 | 15.9 | 31.5 | 35.6 | 28.2
SCOT [31] | 34.9 | 20.7 | 63.8 | 21.1 | 43.5 | 27.3 | 21.3 | 63.1 | 20.0 | 42.9 | 42.5 | 31.1 | 29.8 | 35.0 | 27.7 | 24.4 | 48.4 | 40.8 | 35.6
DHPF [42] | 38.4 | 23.8 | 68.3 | 18.9 | 42.6 | 27.9 | 20.1 | 61.6 | 22.0 | 46.9 | 46.1 | 33.5 | 27.6 | 40.1 | 27.6 | 28.1 | 49.5 | 46.5 | 37.3
CHM [39] | 49.6 | 29.3 | 68.7 | 29.7 | 45.3 | 48.4 | 39.5 | 64.9 | 20.3 | 60.5 | 56.1 | 46.0 | 33.8 | 44.3 | 38.9 | 31.4 | 72.2 | 55.5 | 46.3
MMNet [83] | 55.9 | 37.0 | 65.0 | 35.4 | 50.0 | 63.9 | 45.7 | 62.8 | 28.7 | 65.0 | 54.7 | 51.6 | 38.5 | 34.6 | 41.7 | 36.3 | 77.7 | 62.5 | 50.4
PMNC [23] | 54.1 | 35.9 | 74.9 | 36.5 | 42.1 | 48.8 | 40.0 | 72.6 | 21.1 | 67.6 | 58.1 | 50.5 | 40.1 | 54.1 | 43.3 | 35.7 | 74.5 | 59.9 | 50.4
CATs [6] | 52.0 | 34.7 | 72.2 | 34.3 | 49.9 | 57.5 | 43.6 | 66.5 | 24.4 | 63.2 | 56.5 | 52.0 | 42.6 | 41.7 | 43.0 | 33.6 | 72.6 | 58.0 | 49.9
VAT (ours) | 56.5 | 37.8 | 73.0 | 38.7 | 50.9 | 58.2 | 40.8 | 70.5 | 20.4 | 72.6 | 61.1 | 57.8 | 45.6 | 48.1 | 52.4 | 39.7 | 77.7 | 71.4 | 54.2

Table 1:  Per-class quantitative evaluation on SPair-71k [41] benchmark.

Methods | PF-PASCAL [12] | PF-WILLOW [11]
---|---|---
PCK @ $\alpha_{\text{img}}$ | PCK @ $\alpha_{\text{bbox}}$
0.1 | 0.15 | 0.1 | 0.15
CNNGeo [49] | 69.5 | 80.4 | 69.2 | 77.8
WeakAlign [50] | 74.8 | 84.0 | 70.2 | 79.9
SFNet [24] | 81.9 | 90.6 | 74.0 | 84.2
NC-Net [52] | 78.9 | 86.0 | 67.0 | 83.7
DCC-Net [19] | 82.3 | 90.5 | 73.8 | 86.5
HPF [40] | 84.8 | 92.7 | 74.4 | 85.6
GSF [20] | 87.8 | 95.9 | 78.7 | 90.2
DHPF [42] | 90.7 | 95.0 | 77.6 | 89.1
SCOT [31] | 85.4 | 92.7 | 76.0 | 87.1
CHM [39] | 91.6 | 94.9 | 79.4 | 87.5
MMNet [83] | 91.6 | 95.9 | - | -
PMNC [23] | 90.6 | - | - | -
CATs [6] | 92.6 | 96.4 | 79.2 | 90.3
VAT (ours) | 92.3 | 96.1 | 81.0 | 91.4

Table 2: Quantitative evaluation on standard benchmarks [41, 11, 12].

Different aggregators | Memory | Run-time
---|---|---
(GB) | (ms)
Standard transformer [65] | OOM | N/A
Center-pivot 4D convolutions [38] | 3.5 | 52.7
Linear transformer [21] | 3.5 | 56.8
Fastformer [71] | 3.5 | 122.9
Volumetric transformer (ours) | 3.8 | 57.3

Table 3: Memory and Run-time comparison between different aggregators. The
memory and run-time are measured by simply replacing the aggregator module
within VAT.

| Memory | Run-time
---|---|---
(GB) | (ms)
HSNet [38] | 2.2 | 51.7
VAT (ours) | 3.8 | 57.3

Table 4: Memory and Run-time comparison to HSNet [38].

#### VAT in Semantic Correspondence.

For the experiments to validate effectiveness of VAT on semantic
correspondence, we used the three most widely used datasets in this task,
which include SPair-71k [41], PF-PASCAL [12] and PF-WILLOW [11]. We made minor
modifications to our model: we excluded bilinear upsampling in the affinity-
aware decoder, which the original output shape is $128\times 128$ but with the
modification, now the output shape is $32\times 32$ as two x2 bilinear
upsampling is excluded. We used embedded features from conv4$\\_$ and
conv5$\\_$x, which is different from the training setting for the experiments
on few-shot segmentation datasets. Note that we do not utilize segmentation
mask for this task. We only use keypoints provided in the datasets for
training similarly to [6]. We used the same data augmentation scheme [6, 3] as
Table 5. We used learning rate of 3e-6 for the feature backbone and 3e-4 for
VAT. We used batch size of 8. For the evaluation, we followed the standard
protocol [40, 42, 31, 20, 6, 39].

| Augmentation type | Probability
---|---|---
(I) | ToGray | 0.2
(II) | Posterize | 0.2
(III) | Equalize | 0.2
(IV) | Sharpen | 0.2
(V) | RandomBrightnessContrast | 0.2
(VI) | Solarize | 0.2
(VII) | ColorJitter | 0.2

Table 5: Augmentation type

## Appendix B. Failure Cases

Overall, we observe that our method may lack an ability to preserve fine-
details and address multi-correspondence as shown in Figure 1. Given an object
in the support image and multiple corresponding objects in query image, VAT
sometimes seem to fail finding correspondence. From these, perhaps, it could
be argued that the proposed method is better at encoding matching information
and effectively aggregating them than others, but it might lack an ability to
better find multi-objects or preserve the fine-details. Better means to
consider both factors either by introducing some confidence module [64] or
preserve fine-details when increasing the resolutions of the predicted mask
during the process in decoder can be a promising direction future work.

## Appendix C. Memory and Run-time

We additionally provide memory and run-time comparison to other aggregators
and HSNet [38]. The results are obtained using a single NVIDIA GeForce RTX
3090 GPU and Intel Core i7-10700 CPU. As shown in Table 3, we observe that VAT
is relatively slower and comsumes more memory than other aggregators. About
0.3 GB of more memory consumption and 5 ms slower run-time occur with the
proposed VAT in return for the performance. From the perspective of
aggregators, 0.3 GB and 5 ms gap seem quite trivial to us.

We also provide direct comparison to the current state-of-the-art method,
HSNet [38], in Table 4. We observe that memory consumption gap is quite large,
which is an apparent limitation to the proposed method. Run-time gap is quite
trivial, enabling real-time inference. Note that standard transformer can not
be used due to OOM. In light of this, although we managed to propose a method
to aggregate a raw correlation map without arbitrarily changing its
resolutions prior to feeding into networks, we observe relatively large memory
consumption gap to HSNet [38], which suggests a promising direction for future
work, i.e., even more efficient method while enabling to process at higher
spatial resolutions.

Figure 1: Failure cases.

## Appendix D. More Results

For few-shot segmentation, VAT clearly sets new state-of-the-art for all the
benchmarks, demonstrating its effectiveness. Also as VAT outperforms other
SOTA methods by large margin for semantic correspondence, we argue that the
cost aggregation is indeed a prime importance, even for few-shot segmentation
as well. We argue that this performance boost in few-shot segmentation can be
attributed to VAT’s ability to find accurate semantic correspondences when
tackling the segmentation task. In light of this, we are suggesting a paradigm
shift for the few-shot segmentation task.

#### Quantitative Results on Semantic Correspondence.

As shown in Table 1 and Table 2, we provide more quantitative results on
SPair-71k [41], PF-PASCAL [12], and PF-WILLOW [11] in comparison to other
semantic correspondence methods, including CNNGeo [49], WeakAlign [50], NC-Net
[52], HPF [40], SFNet [24], DCC-Net [19], GSF [20], SCOT [31], DHPF [42], CHM
[39], MMNet [83], PMNC [23] and CATs [6].

#### Qualitative Results.

As shown in Figure 2, Figure 3, Figure 4, Figure 5 and Figure 6, we provide
qualitative results on all the benchmarks, which includes PASCAL-5i [54],
COCO-20i [28], FSS-1000 [27], PF-PASCAL [12], PF-WILLOW [11] and SPair-71k
[41].

Figure 2: Qualitative results on PASCAL-5i [54].

Figure 3: Qualitative results on COCO-20i [28].

Figure 4: Qualitative results on FSS-1000 [27].

Figure 5: Qualitative results on PF-PASCAL [12] (left) and PF-WILLOW [11]
(right).

Figure 6: Qualitative results on SPair-71k [41].