# CAT-Seg : Cost Aggregation for Open-Vocabulary Semantic Segmentation

Seokju Cho∗,1 Heeseong Shin∗,1 Sunghwan Hong1 Seungjun An1 Seungjun Lee1,
Anurag Arnab2 Paul Hongsuck Seo2 Seungryong Kim1
1Korea University 2Google Research
{seokju_cho, hsshin98, sung_hwan, dkstmdwns, 9penguin9,
<EMAIL_ADDRESS>
{aarnab<EMAIL_ADDRESS>
###### Abstract

Existing works on open-vocabulary semantic segmentation have utilized large-
scale vision-language models, such as CLIP, to leverage their exceptional
open-vocabulary recognition capabilities. However, the problem of transferring
these capabilities learned from image-level supervision to the pixel-level
task of segmentation and addressing arbitrary unseen categories at inference
makes this task challenging. To address these issues, we aim to attentively
relate objects within an image to given categories by leveraging relational
information among class categories and visual semantics through aggregation,
while also adapting the CLIP representations to the pixel-level task. However,
we observe that direct optimization of the CLIP embeddings can harm its open-
vocabulary capabilities. In this regard, we propose an alternative approach to
optimize the image-text similarity map, i.e. the cost map, using a novel cost
aggregation-based method. Our framework, namely CAT-Seg, achieves state-of-
the-art performance across all benchmarks. We provide extensive ablation
studies to validate our choices. Project page: https://ku-cvlab.github.io/CAT-
Seg/. ††footnotetext: ∗Equal contribution

Figure 1: Teaser. For open-vocabulary semantic segmentation, we propose to
aggregate a matching cost derived from dense image and text embeddings of CLIP
[44], resulting in state-of-the-art performance across all benchmarks.

(a) mIoU scores of seen classes (b) mIoU scores of unseen classes

(c) Qualitative comparisons

Figure 2: Comparison between feature and cost aggregation. To validate our
framework, we consider two approaches: direct optimization of CLIP embeddings
through feature aggregation and indirect optimization through cost
aggregation. (a): Both approaches achieves performance gains for seen classes
from fine-tuning the CLIP image encoder. (b): Feature aggregation fails to
generalize to unseen classes, while cost aggregation achieves a large
performance gains, highlighting the effectiveness of this approach for open-
vocabulary segmentation. (c): Qualitative results where (IV) successfully
segments the previously unseen class, e.g., ‘birdcage’, whereas (III) fails.

##  1 Introduction

Open-vocabulary semantic segmentation aims to label each pixel in an image
with class categories given as textual descriptions, which may include classes
that were not seen during training. To address this task, recent works [32,
17, 14, 33, 56] typically attempted to leverage large-scale pre-trained
vision-language models, e.g., CLIP [44] and ALIGN [26], for their remarkable
open-vocabulary recognition capabilities achieved from extensive training
process involving large image-caption pairs scaling from millions to billions
[47]. However, transferring the knowledge of these models trained with image-
level supervision to the pixel-level prediction task is highly challenging
[56] due to the difference in granularity between image and pixel.

To address this challenge, recent methods [33, 14, 56] generate class-agnostic
region proposals and use these regions as inputs to the CLIP encoder. Despite
their appreciable performance, these proposals not only exhibit bias towards
the training dataset [33] but are also processed independently by CLIP
disregarding the global context within the image. Alternatively, MaskCLIP [63]
modifies the last pooling layer within the CLIP encoder [44] to obtain dense
image embeddings. However, this approach can introduce substantial noise
without a means for refinement, which may lead to erroneous predictions, as
shown in Fig. 1. While it is a common practice to fine-tune the encoder to
obtain task-specific features [36, 20, 6] for mitigating this issue, it has
been shown that using the conventional fine-tuning approach harms the
capabilities of CLIP [63].

Moreover, existing methods [32, 14, 17, 33] often do not account for the
additional challenges presented by open-vocabulary setups. In contrast to
conventional segmentation settings [36, 20, 19, 58, 64], where relational
information among predefined classes can be used to better understand the
scene during inference [64, 19, 28, 59], open-vocabulary setups assume
arbitrary classes at inference time, preventing the models from learning such
relational information without an explicit means for consideration.

To this end, given that the encoders of CLIP have been trained on a large-
scale data that enables remarkable open-vocabulary capabilities [32, 17, 63,
56, 33, 14], we aim to effectively transfer their knowledge for the pixel-
level prediction task, and attentively relate objects within the image to
given class categories provided as textual descriptions by joint aggregation
of both modalities. Furthermore, we aim to fine-tune the CLIP image encoder to
enable the model to adapt to the pixel-level prediction task at hand. However,
due to the aforementioned issue that direct optimization degrades the open-
vocabulary capabilities [63], it is important to explore an alternative to
optimize the image encoder of CLIP without hurting its pre-trained knowledge.

Upon revisiting the training process of CLIP, we find that the cost volume,
often defined as cosine-similarity map, constructed between the image and text
embeddings [44] is used for loss computation. From this, based on an insight
that preserving the use of cost volume could serve as an alternative approach,
we propose to construct an image-text matching cost to explicitly represent
their relations, and introduce a cost aggregation-based framework to process
the cost volume as shown in Fig. 1. We demonstrate that this approach can
effectively transfer the knowledge of CLIP without diminishing its open-
vocabulary capabilities as shown in Fig. 2.

Our network, namely CAT-Seg, consists of three components: first, we construct
a cost volume using image and text embeddings; second, we employ a Transformer
[52] based module that decomposes the aggregation process into spatial and
class aggregation, aided by an additional technique called embedding guidance,
to effectively aggregate the cost volume; and finally, we use a decoder to
process the aggregated cost volume while capturing fine details. In addition,
we introduce an efficient approach for training our network, where we only
fine-tune the attention layers within the Transformer architecture. Our
approach is not only efficient, but also more effective compared to the
conventional method of fine-tuning all parameters.

In experiments, we evaluate CAT-Seg on various datasets, including ADE20K
[62], PASCAL VOC [16] and PASCAL-Context [41]. CAT-Seg achieves state-of-the-
art results, with a 22% improvement in mIoU over the previous best result on
ADE20K with 150 categories and a 73% improvement on PASCAL-Context with 459
categories. We also provide extensive ablation studies and analyses.

(a) Raw cost of “cat” (b) Aggregated cost of “cat”

(c) Raw cost of “guitar” (d) Aggregated cost of “guitar”

Figure 3: Visualization of cost volumes. The proposed cost aggregation
framework effectively refines the noisy costs in (a) and (c), as exemplified
in (b) and (d). Note that we overlay the costs with the input image for
visualization.
Figure 4: Overall architecture of CAT-Seg. Our network consists of a cost
computation, aggregation module consisting of spatial aggregation and inter-
class aggregation and an upsampling decoder.

##  2 Related Work

#### Open-vocabulary semantic segmentation.

Classical approaches to the task [61, 2, 55] attempt to learn visual
embeddings that align with pre-defined text embeddings [40, 39]. However, the
domain difference, as well as the limited vocabulary of the words have been
the major bottlenecks. To address this, LSeg [32] leveraged CLIP for learning
pixel-level visual embeddings aligned with the text embeddings of CLIP.
Alternatively, OpenSeg [17] proposed to identify local regions within the
image and correlate with the text embeddings with class-agnostic region
proposals.

More recently, ZegFormer [14] and ZSseg [56] proposed two-stage frameworks.
Typically, they first learn to predict class-agnostic region proposals similar
to [17], and feed them to CLIP for final predictions. To better recognize
these regions, OVSeg [33] collects region-text pairs to fine-tune the CLIP
encoder. A notable work, MaskCLIP [63] attempts to obtain pixel-level
embeddings from CLIP that can be directly utilized for segmentation. However,
based on their findings [63] that conventional fine-tuning impedes the open-
vocabulary capability of CLIP, they instead propose to freeze the CLIP encoder
and apply non-learnable methods to mitigate its noise. Unlike the
aforementioned works, we introduce a cost aggregation method to benefit from
fine-tuning the CLIP encoder and obtain accurate pixel-level predictions.

#### Cost aggregation.

Cost aggregation is a popular technique adopted for the process of
establishing correspondence between visually or semantically similar images
[30, 18, 57, 10, 22] by reducing the impact of errors and inconsistencies in
the matching process. A matching cost, an input to cost aggregation, is
typically constructed between dense features extracted from a pair of images
[45], and often cosine-similarity [34, 45] is used. In matching literature,
numerous works [30, 5, 18, 57, 48, 23, 25, 11] have proposed cost aggregation
modules and demonstrated its importance, owing to its favorable generalization
ability [48, 34]. In this work, we leverage the cost volume constructed
between image and text embeddings from CLIP encoders to promote accurate
segmentation through cost aggregation.

##  3 Methodology

###  3.1 Motivation and Overview

Given an image $I$ and a set of candidate class categories
$\\{T(1),...,T({N_{\mathcal{C}}})\\}\in\mathcal{C}$, where $T(n)$ denotes
textual description of $n$-th category and $N_{\mathcal{C}}$ is the number of
classes, open-vocabulary semantic segmentation aims to assign a class label to
each pixel in $I$. Different from classical semantic segmentation tasks [36,
20, 64, 19, 28, 58, 59], open-vocabulary segmentation is additionally
challenged by varying $\mathcal{C}$ at inference, which includes classes that
were not observed during training.

In this paper, we aim to transfer the open-vocabulary capabilities of CLIP
[44] to enable our model to reason about relations between image semantics and
arbitrary class labels, addressing the challenges of open-vocabulary semantic
segmentation. To achieve this goal, we propose to jointly aggregate the image
and text embeddings of CLIP and further enhance the aggregation by fine-tuning
the image encoder. While it is a common approach to transfer the knowledge
within large-scale models to tackle downstream tasks through fine-tuning the
parameters [20, 38], existing approaches in this task have failed to
effectively transfer their knowledge [63] due to several reasons. This
includes the granularity gap between image-level supervision and pixel-level
prediction, and catastrophic forgetting caused by direct optimization through
feature embeddings [63].

To address the issues, we leverage insights from the training process of CLIP
[44] which uses a cosine-similarity map, _i.e_. cost map. Our pilot experiment
showed that using cost volume derived from dense image embeddings and text
embeddings of CLIP as an alternative is effective, as demonstrated in Fig. 2,
and thus we introduce cost volume to our framework. Subsequently, we propose
an aggregation module that incorporates both spatial and class aggregation to
effectively capture relational information among inputs while promoting
accurate matching between text and objects in the image, thereby reducing
errors and inconsistencies in the matching costs, as shown in Fig. 3. Finally,
we introduce an efficient fine-tuning approach that not only provides improved
performance, but also enhances accessibility. All these combined, we present
CAT-Seg, Cost AggregaTion approach for open-vocabulary semantic Segmentation.

###  3.2 Cost Computation and Embedding

To obtain dense CLIP features, we follow [63] and modify the last layer of the
image encoder. Given the modified CLIP image encoder $\Phi^{V}(\cdot)$ and the
frozen text encoder $\Phi^{L}(\cdot)$, we extract the dense image features
$D^{V}=\Phi^{V}(I)$ and the text features $D^{L}=\Phi^{L}(T)$, respectively.
We use the image and text features $D^{V}(i)$ and $D^{L}(n)$, where $i$
denotes 2D spatial positions of feature embedding and $n$ denotes an index for
a class, to compute a cost volume $C\in\mathbb{R}^{(H\times W)\times
N_{\mathcal{C}}}$ by cosine similarity [45]. Formally, this is defined as:

${C}(i,n)=\frac{D^{V}(i)\cdot D^{L}(n)}{\|D^{V}(i)\|\|D^{L}(n)\|}.$ (1)

To enhance the processing of cost in high dimensional feature space, we feed
the cost volume to a single convolution layer that processes each cost slice
$C(:,n)\in\mathbb{R}^{(H\times W)\times 1}$ independently to obtain initial
cost volume embedding $F\in\mathbb{R}^{(H\times W)\times N_{\mathcal{C}}\times
d_{F}}$, where $d_{F}$ is the cost embedding dimension, as shown in Fig. 4.

###  3.3 Cost Aggregation

Based on the insights from above, we use the constructed cost volume embedding
and feed it to aggregation modules. As it has been shown in previous studies
that learning based on a matching cost within a regularized representation,
such as the cosine-similarity map, can improve generalization ability and
robustness to unseen domains [4, 48, 34], we find this approach valid.

Within our cost aggregation module, as shown in Fig. 4, two separate
aggregation strategies, _i.e_. spatial and class aggregation, are designed to
model interrelations across pixels and classes. This considers the distinct
characteristic of each modality, such as spatial smoothness in an image or the
permutation invariance among classes. Specifically, we first perform spatial
aggregation and then class aggregation takes place, and interleave both
aggregations $N_{B}$ times. In addition, we facilitate the cost aggregation
process with embedding guidance that provide contextual information from two
modalities. In the following, we explain each in detail.

#### Spatial aggregation.

Here, we reason about the spatial relations based on pixel-wise similarities
computed between image and text embeddings. For this, we adopt Transformer
[52, 35] over CNNs for its adaptability to input tokens [12], while also
having global [52] or semi-global [35, 22] receptive fields, which is more
favorable to our goal to learn relations among all tokens. In practice, we
employ Swin Transformer [35] for computational efficiency. We define this
process as:

${F}^{\prime}(:,n)=\mathcal{T}^{\mathrm{sa}}(F(:,n)),$ (2)

where $F(:,n)\in\mathbb{R}^{(H\times W)\times d_{F}}$, and
$\mathcal{T}^{\mathrm{sa}}(\cdot)$ denotes a pair of two consecutive Swin
transformer block for spatial aggregation, where the first block features
self-attention within a local window, followed by the second block with self-
attention within shifted window. Note that we treat $d_{F}$ as channel
dimensions for each token, and attention is computed within individual classes
separately. Intuitively, we perform spatial aggregation for each class to
locate the features that will guide to accurate segmentation outputs.

#### Class aggregation.

Subsequent to spatial aggregation, class aggregation is designed to explicitly
capture relationships between different class categories. However, this task
presents two challenges that need to be addressed: the variable number of
categories $\mathcal{C}$ and their unordered input arrangement. To address
these challenges, we employ a Transformer [52] model without position
embedding for aggregation. This approach enables the handling of sequences of
arbitrary length and provides the model with permutation invariance to inputs.
This process is defined as:

$\displaystyle{F}^{\prime\prime}(i,:)=\mathcal{T}^{\mathrm{ca}}({F}^{\prime}(i,:)),$
(3)

where $F^{\prime}(i,:)\in\mathbb{R}^{N_{\mathcal{C}}\times d_{F}}$, and
$\mathcal{T}^{\mathrm{ca}}(\cdot)$ denotes a transformer block for class
aggregation. Although we can employ the same Swin Transformer [35] as for the
spatial aggregation, we instead employ a linear transformer [29] as we do not
need to consider spatial structure of the input tokens in this aggregation.
Also, it offers a linear computational complexity with respect to the number
of the tokens, allowing efficient computation.

#### Embedding guidance.

As a means to enhance cost aggregation process, we additionally leverage
feature embeddings to provide spatial structure or contextual information of
the inputs. Intuitively, we aim to guide the process with feature embeddings,
based on the assumption that visually or semantically similar input tokens,
e.g., color or category, have similar matching costs, inspired by cost volume
filtering [24, 49] in stereo matching literature [46]. Specifically, the
visual embeddings $E^{V}$ and the text embedding $E^{L}$ are extracted from
the additional feature backbone and CLIP text encoder, respectively, followed
by a linear projection and concatenation to the input sequence. When
extracting $E^{V}$, we avoid using the CLIP image encoder because of the
potential hindrance to generalization ability when directly utilized, as
discussed in Fig. 2. Accordingly, we redefine Eq. 2 and Eq. 3 as:

$\begin{split}&{F}^{\prime}(:,n)=\mathcal{T}^{\mathrm{sa}}([F(:,n);\mathcal{P}^{V}(E^{V})]),\\\
&{F}^{\prime\prime}(i,:)=\mathcal{T}^{\mathrm{ca}}([F^{\prime}(i,:);\mathcal{P}^{L}(E^{L})]),\end{split}$
(4)

where $[\cdot]$ denotes concatenation, $\mathcal{P}^{V}$ and $\mathcal{P}^{L}$
denote linear projection layer, $E^{V}\in\mathbb{R}^{(H\times W)\times
d_{E}}$, and $E^{L}\in\mathbb{R}^{N_{\mathcal{C}}\times d_{E}}$, where $d_{E}$
denotes the feature dimension. Notably, we only provide feature embeddings to
query and key as we find this is sufficient for embedding guidance.

###  3.4 Upsampling Decoder

Given the aggregated cost volume, we aim to generate the final segmentation
mask that captures fine-details via upsampling. The simplest approach would be
using handcrafted upsamplers, _i.e_. bilinear upsampling, but we propose to
conduct further aggregation within the decoder with light-weight convolution
layers. Additionally, we provide feature embeddings from the feature backbone
that act as an effective guide to filter out the noises in the cost volume and
exploit the higher-resolution spatial structure for preserving fine-details.

Specifically, we leverage the multi-level feature embeddings readily extracted
in embedding guidance for cost aggregation. Following the approach presented
in [22], we employ bilinear upsampling on the cost volume and concatenate it
with the corresponding level of feature map, followed by a convolutional
layer. We iterate this process $N_{U}$ times, generating a high-resolution
output that we feed into the prediction head for final inference.

###  3.5 Efficient Fine-Tuning of CLIP

During training, we train our model in an end-to-end manner, including the
CLIP image encoder. However, fine-tuning the encoder, which can scale up to
hundreds of millions of parameters, can be computationally expensive and
memory-intensive. Moreover, since our objective is to effectively transfer the
open-vocabulary capability of CLIP, performance should not be compromised as
well. To this end, we freeze the MLP layers in the encoder, also known as
Feed-Forward Networks (FFNs), and only optimize the layers responsible for
spatial interaction, such as the attention layers and positional embeddings
[52], based on the insight that tuning these layers is sufficient for
transferring image-level representations to pixel-level. With this approach,
we achieve higher efficiency and improved performance over full fine-tuning.

Methods | VLM | Feature backbone | Training dataset | Additional dataset | A-847 | PC-459 | A-150 | PC-59 | PAS-20 | $\textnormal{PAS-20}^{b}$
---|---|---|---|---|---|---|---|---|---|---
SPNet [55] | - | ResNet-101 | PASCAL VOC | - | - | - | - | 24.3 | 18.3 | -
ZS3Net [2] | - | ResNet-101 | PASCAL VOC | - | - | - | - | 19.4 | 38.3 | -
LSeg [32] | ViT-B/32 | ResNet-101 | PASCAL VOC-15 | - | - | - | - | - | 47.4 | -
LSeg+ [17] | ALIGN | ResNet-101 | COCO-Stuff | - | 2.5 | 5.2 | 13.0 | 36.0 | - | 59.0
ZegFormer [14] | ViT-B/16 | ResNet-101 | COCO-Stuff-156 | - | 4.9 | 9.1 | 16.9 | 42.8 | 86.2 | 62.7
ZegFormer$\dagger$ [14] | ViT-B/16 | ResNet-101 | COCO-Stuff | - | 5.6 | 10.4 | 18.0 | 45.5 | 89.5 | 65.5
ZSseg [56] | ViT-B/16 | ResNet-101 | COCO-Stuff | - | 7.0 | - | 20.5 | 47.7 | 88.4 | -
OpenSeg [17] | ALIGN | ResNet-101 | COCO Panoptic [31] | Localized Narrative | 4.4 | 7.9 | 17.5 | 40.1 | - | 63.8
OVSeg [33] | ViT-B/16 | ResNet-101c | COCO-Stuff | COCO Caption | 7.1 | 11.0 | 24.8 | 53.3 | 92.6 | -
CAT-Seg (ours) | ViT-B/16 | ResNet-101 | COCO-Stuff | - | 8.4 (+1.3) | 16.6 (+5.6) | 27.2 (+2.4) | 57.5 (+4.2) | 93.7 (+1.1) | 78.3 (+12.8)
LSeg [32] | ViT-B/32 | ViT-L/16 | PASCAL VOC-15 | - | - | - | - | - | 52.3 | -
OpenSeg [17] | ALIGN | Eff-B7 [50] | COCO Panoptic [31] | Localized Narrative | 8.1 | 11.5 | 26.4 | 44.8 | - | 70.2
OVSeg [33] | ViT-L/14 | Swin-B | COCO-Stuff | COCO Caption | 9.0 | 12.4 | 29.6 | 55.7 | 94.5 | -
| ViT-L/14 | Swin-B | COCO-Stuff | - | 10.8 (+1.8) | 20.4 (+8.0) | 31.5 (+1.9) | 62.0 (+6.3) | 96.6 (+2.1) | 81.8 (+11.6)
| ViT-H/14* | Swin-B | COCO-Stuff | - | 12.4 (+3.4) | 20.1 (+7.7) | 34.4 (+4.8) | 61.2 (+5.5) | 96.7 (+2.2) | 80.2 (+10.0)
CAT-Seg (ours) | ViT-G/14* | Swin-B | COCO-Stuff | - | 13.3 (+4.3) | 21.4 (+9.0) | 36.2 (+6.6) | 61.5 (+5.8) | 97.1 (+2.6) | 81.4 (+11.2)

Table 1: Quantitative evaluation. The best-performing results are presented in
bold, while the second-best results are underlined. Improvements over the
second-best methods are highlighted in green. mIoU is adopted for evaluation
metric. $\dagger$: Re-implemented to train with full COCO-Stuff [3] dataset.
*: Model trained on LAION-2B [47] dataset introduced in [9].

| Components | A-847 | PC-459 | A-150 | PC-59 | PAS-20 | $\textnormal{PAS-20}^{b}$
---|---|---|---|---|---|---|---
(I) | Feature Agg. | 3.8 | 10.9 | 19.1 | 53.5 | 96.2 | 74.2
(II) | Cost Agg. | 9.2 | 14.6 | 27.4 | 58.6 | 96.8 | 78.2
(III) |  (II) \+ Transformer | 9.9 | 15.1 | 28.5 | 57.0 | 96.2 | 76.4
(IV) |  (II) \+ Spatial agg. | 9.4 | 15.1 | 28.1 | 58.3 | 97.0 | 78.5
(V) |  (IV) \+ Inter-class agg. | 10.6 | 16.1 | 30.9 | 61.7 | 96.8 | 81.4
(VI) |  (V) \+ Embedding guidance | 10.8 | 20.4 | 31.5 | 62.0 | 96.6 | 81.8

Table 2: Ablation study for CAT-Seg.

| Methods | A-847 | PC-459 | A-150 | PC-59 | PAS-20 | $\textnormal{PAS-20}^{b}$ | #param. | Memory
---|---|---|---|---|---|---|---|---|---
| (M) | (GiB)
(I) | Freeze | 4.8 | 8.7 | 20.8 | 52.5 | 84.5 | 76.7 | 0 | 1.9
(II) | VPT [27] | 8.3 | 13.7 | 29.4 | 59.5 | 90.8 | 81.0 | 2.3 | 2.0
(III) | Full F.T. | 9.4 | 18.5 | 30.2 | 61.0 | 96.8 | 81.5 | 290.0 | 5.4
(IV) | Ours | 10.8 | 20.4 | 31.5 | 62.0 | 96.6 | 81.8 | 96.7 | 3.1

Table 3: Ablation study of fine-tuning methods for CLIP image encoder. We
additionally note the number of learnable parameters of CLIP and memory
consumption during training. Our method not only outperforms full fine-tuning,
but also requires a smaller computational footprint. F.T.: Fine-Tune

Methods | A-847 | PC-459 | A-150 | PC-59 | PAS-20 | $\textnormal{PAS-20}^{b}$
---|---|---|---|---|---|---
CLIP image enc. [44] | 10.6 | 15.6 | 30.1 | 60.2 | 95.9 | 80.5
Swin-B [35] | 10.8 | 20.4 | 31.5 | 62.0 | 96.6 | 81.8
CLIP image enc.$\dagger$ [44] | 7.7 | 14.4 | 25.7 | 55.8 | 85.1 | 78.4

Table 4: Ablation on feature backbone. CLIP with ViT-L is used for all
methods. $\dagger$: Stop-gradient operation is applied to cost computation.

##  4 Experiments

###  4.1 Experimental Setup

#### Datasets and evaluation metric.

We train our model on the COCO-Stuff dataset, which has 118k densely annotated
training images with 171 categories, following [33]. We evaluate our model on
ADE20K [62], PASCAL VOC [16], and PASCAL-Context [41] datasets. ADE20K has 20k
training and 2k validation images, with two sets of categories: A-150 with 150
frequent classes and A-847 with 847 classes. PASCAL-Context contains 5k
training and validation images, with 459 classes in the full version (PC-459)
and the most frequent 59 classes in the PC-59 version. PASCAL VOC has 20
object classes and a background class, with 1.5k training and validation
images. We report PAS-20 using 20 object classes. We also report the score for
PAS-$20^{b}$, which defines the “background” as classes present in PC-59 but
not in PAS-20, as in [17]. We adopt mean Intersection over Union (mIoU) as
evaluation metric for all experiments.

#### Implementation details.

We train the CLIP image encoder and the cost aggregation module with per-pixel
binary cross-entropy loss and freeze the CLIP text encoder for all settings.
Feature embeddings $E^{V}$ are obtained from ResNet-101 [21] pre-trained on
the ImageNet-1k [13] for model that uses CLIP with ViT-B [15] and Swin-B [35]
pre-trained on the ImageNet-21k for models that use CLIP with either ViT-L, H
or G [60]. We set $d_{F}=128$, $N_{B}=2$, $N_{U}=2$ for all of our models. We
implement our work using PyTorch [42] and Detectron2 [54]. AdamW [37]
optimizer is used with a learning rate of $2\cdot 10^{-4}$ for our model and
$2\cdot 10^{-6}$ for the CLIP image encoder, with weight decay set to
$10^{-4}$. The batch size is set to 4. We use 4 NVIDIA RTX 3090 GPUs for the
model variant that utilizes ViT-B, L, H [15] as the CLIP encoder, and use a
single NVIDIA A100 for ViT-G [60]. All of the models are trained for 80k
iterations, which takes about 8 hours for model using ViT-L. Further details
can be found in supplementary material. Our code and pre-trained weights will
be made publicly available.

###  4.2 Main Results

In Table 1, we provide quantitative comparisons to competitors. For a fair
comparison, we note the type of the vision-language model (VLM), the feature
backbone, and the training dataset. We also specify the additional dataset if
it is used. We first compare our method with those that employ ResNet-101 as
the feature backbone and ViT-B as vision-language models.

Overall, our method significantly outperforms all the other competitors,
including those [17, 33] that leverage additional dataset [7, 43] for further
performance improvements. For methods that employ Swin-B as the feature
backbone and ViT-L as vision-language models, our method surpasses the closest
competitor by 20% in A-847 and 65% in PC-459. Furthermore, if we choose to
scale our VLM model to a larger variant, performance is further boosted. For
CLIP variants introduced in [9], _i.e_. ViT-H and ViT-G, which are trained on
LAION-2B [47], our method enjoys further performance gains. These results
confirm the validity of our approach to leverage cost aggregation, which
offers an effective means of transferring knowledge from large-scale vision-
language models.

We also present qualitative results of PASCAL-Context with 459 categories in
Fig. 6, demonstrating the efficacy of our proposed approach in comparison to
the current state-of-the-art methods [14, 56, 33].

###  4.3 Analysis and Ablation Study

In this section, we provide ablation study and analysis to validate our
choices. For all the experiments, we employ ViT-L variant for the CLIP encoder
[44] and Swin-B [35] for the feature backbone if not mentioned.

#### Component analysis.

Table 2 shows the effectiveness of the main components within our architecture
through quantitative results. First, we introduce the baseline models in (I)
and (II), which simply feed the feature embeddings or the cost volume to the
proposed upsampling decoder. We then progressively add each proposed
components from (IV) to (VI) to validate our approach to factorize the
aggregation process into spatial and class aggregations, and the proposed
embedding guidance. Note that for the design of (III), we employ a Linear
Transformer [29] that aggregates the entire input without separately
processing spatial and class dimensions. For this analysis, we control the
number of parameters to be similar across all the variants from (III) to (VI).

As shown, we stress the gap between (I) and (II), which supports the findings
presented in Fig. 2. We also highlight that as we gradually incorporate the
proposed spatial and class aggregation techniques, our approach (V)
outperforms (III), demonstrating the effectiveness of our design. Finally,
(VI) shows that our embedding guidance further improves performance across
most of the benchmarks, with the exception of PAS-20, where the error levels
have become saturated as its class categories mostly overlap with the training
dataset.

(a) (b)

Figure 5: Run-time comparison. (a) Comparison of training time among different
fine-tuning choices. (b) Comparison of inference time to [14]. F.T.: Fine-
Tuning.

Methods | CLIP variants | A-847 | PC-459 | A-150 | PC-59 | PAS-20 | $\textnormal{PAS-20}^{b}$
---|---|---|---|---|---|---|---
ZegFormer [14] | ViT-B/16 | 5.6 | 10.4 | 18.0 | 45.5 | 89.5 | 65.5
ZSseg [56] | ViT-B/16 | 7.0 | 9.0 | 20.5 | 47.7 | 88.4 | 67.9
CAT-Seg (ours) | ViT-B/16 | 8.4 | 16.6 | 27.2 | 57.5 | 93.7 | 78.3
ZegFormer [14] | ViT-L/14 | 6.5 | 12.2 | 21.1 | 50.1 | 93.2 | 67.9
ZSseg [56] | ViT-L/14 | 6.8 | 10.6 | 22.3 | 53.8 | 94.7 | 72.2
CAT-Seg (ours) | ViT-L/14 | 10.8 | 20.4 | 31.5 | 62.0 | 96.6 | 81.8
ZegFormer [14] | ViT-H/14* | 5.8 | 9.0 | 17.7 | 43.5 | 86.3 | 65.4
ZSseg [56] | ViT-H/14* | 5.7 | 9.1 | 18.4 | 48.8 | 90.5 | 70.4
CAT-Seg (ours) | ViT-H/14* | 12.4 | 20.1 | 34.4 | 61.2 | 96.7 | 80.2
ZegFormer [14] | ViT-G/14* | 6.4 | 10.1 | 20.3 | 45.9 | 89.8 | 66.8
ZSseg [56] | ViT-G/14* | 6.7 | 10.5 | 20.9 | 51.7 | 93.7 | 73.0
CAT-Seg (ours) | ViT-G/14* | 13.3 | 21.4 | 36.2 | 61.5 | 97.1 | 81.4

Table 5: Scalability comparison. *: Model trained on LAION-2B [47] dataset
introduced in [9].

Methods | Training dataset | A-847 | PC-459 | A-150 | PC-59 | PAS-20 | $\textnormal{PAS-20}^{b}$
---|---|---|---|---|---|---|---
ZegFormer [14] | COCO-Stuff | 5.6 | 10.4 | 18.0 | 45.5 | 89.5 | 65.5
ZSseg [56] | COCO-Stuff | 7.0 | 9.0 | 20.5 | 47.7 | 88.4 | 67.9
CAT-Seg (ours) | COCO-Stuff | 8.4 | 16.6 | 27.2 | 57.5 | 93.7 | 78.3
ZegFormer [14] | A-150 | 6.8 | 7.1 | 33.1 | 34.7 | 77.2 | 53.6
ZSseg [56] | A-150 | 7.6 | 7.1 | 40.3 | 39.7 | 80.9 | 61.1
CAT-Seg (ours) | A-150 | 10.6 | 14.5 | 46.8 | 46.7 | 85.5 | 70.3
ZegFormer [14] | PC-59 | 3.8 | 8.2 | 13.1 | 48.7 | 86.5 | 66.8
ZSseg [56] | PC-59 | 3.0 | 7.6 | 11.9 | 54.7 | 87.7 | 71.7
CAT-Seg (ours) | PC-59 | 5.6 | 12.9 | 23.0 | 62.4 | 87.3 | 79.0

Table 6: Training on various datasets. CLIP with ViT-B is used for all
methods. Our model demonstrates remarkable generalization capabilities even on
relatively smaller datasets. The scores evaluated on the same dataset used for
training are colored in gray for clarity.

#### Comparison among different fine-tuning approaches.

In this ablation study, we examine both effectiveness and efficiency of the
proposed approach to fine-tune the CLIP image encoder. In Table 3, we report
the results of different approaches, which include the variant without fine-
tuning, adopting VPT [27], fine-tuning the entire network and our approach. We
additionally show the comparison of time taken for training in Fig. 5 (a). In
short, we find that our proposed approach not only yields the best
performance, but also requires less memory and training time than full fine-
tuning, which highlights the effectiveness and efficiency of our approach.

#### Feature backbone.

Table 4 presents a comparison of variants with different feature backbones for
providing feature embeddings $E^{V}$. The first row reports the results using
the CLIP image encoder [44] whereas the second row reports the results using
Swin-B [35] as the feature backbone. The results indicate that the use of the
CLIP image encoder generally reduces performance across all benchmarks,
although it still outperforms other approaches [32, 56, 14, 33, 17]. In
contrast, using a separate network, Swin-B [35], leads to the best
performance. We attribute this trend to the same observation shown in Fig. 2,
where the direct optimization of the CLIP image embeddings harms its open-
vocabulary capability. However, we find that optimizing the CLIP image
embedding through the cost volume can mitigate the performance degradation. To
verify this, we include additional results in the last row that employ the
CLIP image encoder as feature backbone, but prohibit gradient flow through the
cost volume, allowing the gradient to flow only through the feature embedding
to the encoder. The degradation in performance is found to be significant,
indicating that using the cost volume can serve as an effective means to
prevent such degradation.

(a) Input (b) ZegFormer [14]

(c) ZSSeg [56] (d) OVSeg [33]

(e) CAT-Seg (ours) (f) Ground-truth

Figure 6: Qualitative results on PASCAL-Context with 459 categories. In
comparison to the two-stage approach [33], where each region of an image is
independently classified using CLIP, our model excels at capturing the context
of an image, as the CLIP image encoder can process the image as a whole,
thereby allowing the CLIP to better capture the relationships and contextual
information between objects.

(a) OVSeg [33] (b) CAT-Seg (ours)

Figure 7: Visualization of part segmentation. Our method can identify
different parts of a person, whereas OVSeg [33] can only identify person as a
whole, visualizing the bias of region proposals in two-stage methods.

#### Scaling comparison.

In this analysis, we examine the scaling property of our method and compare
with other recent methods [14, 56] in Table 5. We use four variants of CLIP
model, ViT-B, L, H and ViT-G, where ViT-B model has the least number of
learnable parameters and ViT-G model has the most. As shown, we observe only
marginal performance gains or even severe drops for competitors when we
progressively increase the model capacity. In contrast, our proposed framework
benefits and shows apparent improvements on almost all the datasets, which
indicate the favorable scaling capacity of our work. Notably, on A-150 and
A-847, our work achieves mIoU improvements of +9.0 and +4.9 over the ViT-B
model, respectively.

#### Training with various datasets.

In this experiment, we further examine the generalization power of our method
in comparison to other methods [14, 56] by training our model on smaller-scale
datasets, which include A-150 and PC-59, that poses additional challenges to
achieve good performance. The results are shown in Table 6. As shown, we find
that although we observe some performance drops, which seems quite natural
when smaller dataset is used, our work significantly outperforms other
competitors. These results highlight the strong generalization power of our
framework, which is a favorable characteristic that suggests practicality of
our approach.

#### Inference speed comparison.

In Fig. 5 (b), we visualize the run-time comparison of different CLIP ViT
variants [44], and compare our results with a recent two-stage framework [14].
Note that as the run-time of our method resort on the number of classes at
inference, which results in different run-time for each dataset, we report the
mean value of the run-time. As illustrated, our approach generally runs faster
than the competitor. We highlight that as we scale the model size, our
approach increasingly benefits from fast inference time, which is favorable
for practicality.

###  4.4 Generalization to Part Segmentation

In Fig. 7, we show that our framework can generalize to part segmentation
task, even when fine-grained categories that were not observed during training
are given as inputs. We compare with state-of-the-art two-stage approach [33].
Interestingly, [33] fails to account for fine-grained, unseen categories, as
its mask proposals exhibit bias towards its training dataset. For example, as
“person” seen during training, it can only identify person as a whole, failing
to segment fine-grained categories such as “arm” and “leg”. In contrast, our
framework does not suffer from such bias, enabling robust recognition of
unseen objects of different granularity.

##  5 Conclusion

In this paper, we have introduced cost aggregation to open-vocabulary semantic
segmentation, which jointly aggregates both image and text modalities within
the matching cost. Our framework, namely CAT-Seg, not only effectively
transfers the knowledge of CLIP to unseen classes, but also leverages the
relations between image semantics and class labels through carefully designed
spatial and class aggregation module aided by the additional technique called
embedding guidance. With this approach, new state-of-the-art is set on every
benchmarks, which highlights the superior generalization ability of our model.

## References

  * [1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [2] Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez.  Zero-shot semantic segmentation.  Advances in Neural Information Processing Systems, 32, 2019.
  * [3] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari.  Coco-stuff: Thing and stuff classes in context.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1209–1218, 2018.
  * [4] Changjiang Cai, Matteo Poggi, Stefano Mattoccia, and Philippos Mordohai.  Matching-space stereo networks for cross-domain generalization.  In 2020 International Conference on 3D Vision (3DV), pages 364–373. IEEE, 2020.
  * [5] Jia-Ren Chang and Yong-Sheng Chen.  Pyramid stereo matching network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5410–5418, 2018.
  * [6] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2017.
  * [7] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  arXiv preprint arXiv:1504.00325, 2015.
  * [8] Bowen Cheng, Alex Schwing, and Alexander Kirillov.  Per-pixel classification is not all you need for semantic segmentation.  Advances in Neural Information Processing Systems, 34:17864–17875, 2021.
  * [9] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev.  Reproducible scaling laws for contrastive language-image learning.  arXiv preprint arXiv:2212.07143, 2022.
  * [10] Seokju Cho, Sunghwan Hong, Sangryul Jeon, Yunsung Lee, Kwanghoon Sohn, and Seungryong Kim.  Cats: Cost aggregation transformers for visual correspondence.  Advances in Neural Information Processing Systems, 34:9011–9023, 2021.
  * [11] Seokju Cho, Sunghwan Hong, and Seungryong Kim.  Cats++: Boosting cost aggregation with convolutions and transformers.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [12] Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan.  Coatnet: Marrying convolution and attention for all data sizes.  Advances in Neural Information Processing Systems, 34:3965–3977, 2021.
  * [13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [14] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai.  Decoupling zero-shot semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11583–11592, 2022.
  * [15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [16] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International journal of computer vision, 88:303–308, 2009.
  * [17] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin.  Scaling open-vocabulary image segmentation with image-level labels.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXVI, pages 540–557. Springer, 2022.
  * [18] Xiaoyang Guo, Kai Yang, Wukui Yang, Xiaogang Wang, and Hongsheng Li.  Group-wise correlation stereo network.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3273–3282, 2019.
  * [19] Junjun He, Zhongying Deng, Lei Zhou, Yali Wang, and Yu Qiao.  Adaptive pyramid context network for semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7519–7528, 2019.
  * [20] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [22] Sunghwan Hong, Seokju Cho, Jisu Nam, Stephen Lin, and Seungryong Kim.  Cost aggregation with 4d convolutional swin transformer for few-shot segmentation.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIX, pages 108–126. Springer, 2022.
  * [23] Sunghwan Hong, Jisu Nam, Seokju Cho, Susung Hong, Sangryul Jeon, Dongbo Min, and Seungryong Kim.  Neural matching fields: Implicit representation of matching fields for visual correspondence.  arXiv preprint arXiv:2210.02689, 2022.
  * [24] Asmaa Hosni, Christoph Rhemann, Michael Bleyer, Carsten Rother, and Margrit Gelautz.  Fast cost-volume filtering for visual correspondence and beyond.  PAMI, 2012.
  * [25] Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li.  Flowformer: A transformer architecture for optical flow.  arXiv preprint arXiv:2203.16194, 2022.
  * [26] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In International Conference on Machine Learning, pages 4904–4916. PMLR, 2021.
  * [27] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII, pages 709–727. Springer, 2022.
  * [28] Zhenchao Jin, Tao Gong, Dongdong Yu, Qi Chu, Jian Wang, Changhu Wang, and Jie Shao.  Mining contextual information beyond image for semantic segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7231–7241, 2021.
  * [29] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret.  Transformers are rnns: Fast autoregressive transformers with linear attention.  In International Conference on Machine Learning, pages 5156–5165. PMLR, 2020.
  * [30] Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry.  End-to-end learning of geometry and context for deep stereo regression.  In Proceedings of the IEEE international conference on computer vision, pages 66–75, 2017.
  * [31] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár.  Panoptic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9404–9413, 2019.
  * [32] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl.  Language-driven semantic segmentation.  arXiv preprint arXiv:2201.03546, 2022.
  * [33] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu.  Open-vocabulary semantic segmentation with mask-adapted clip.  arXiv preprint arXiv:2210.04150, 2022.
  * [34] Biyang Liu, Huimin Yu, and Guodong Qi.  Graftnet: Towards domain generalized stereo matching with a broad-spectrum and task-oriented feature.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13012–13021, 2022.
  * [35] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
  * [36] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [37] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [38] Thomas Mensink, Jasper Uijlings, Alina Kuznetsova, Michael Gygli, and Vittorio Ferrari.  Factors of influence for transfer learning across diverse appearance domains and task types.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):9298–9314, 2021.
  * [39] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781, 2013.
  * [40] George A Miller.  WordNet: An electronic lexical database.  MIT press, 1998.
  * [41] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille.  The role of context for object detection and semantic segmentation in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 891–898, 2014.
  * [42] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32, 2019.
  * [43] Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari.  Connecting vision and language with localized narratives.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, pages 647–664. Springer, 2020.
  * [44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [45] Ignacio Rocco, Relja Arandjelovic, and Josef Sivic.  Convolutional neural network architecture for geometric matching.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6148–6157, 2017.
  * [46] Daniel Scharstein and Richard Szeliski.  A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.  International journal of computer vision, 47:7–42, 2002.
  * [47] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  arXiv preprint arXiv:2210.08402, 2022.
  * [48] Xiao Song, Guorun Yang, Xinge Zhu, Hui Zhou, Zhe Wang, and Jianping Shi.  Adastereo: A simple and efficient approach for adaptive stereo matching.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10328–10337, 2021.
  * [49] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.  Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume.  In CVPR, 2018.
  * [50] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International conference on machine learning, pages 6105–6114. PMLR, 2019.
  * [51] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In International conference on machine learning, pages 10347–10357. PMLR, 2021.
  * [52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [53] Yuxin Wu and Kaiming He.  Group normalization.  In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018.
  * [54] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [55] Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata.  Semantic projection network for zero-and few-label semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8256–8265, 2019.
  * [56] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai.  A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIX, pages 736–753. Springer, 2022.
  * [57] Gengshan Yang, Joshua Manela, Michael Happold, and Deva Ramanan.  Hierarchical deep stereo matching on high-resolution images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5515–5524, 2019.
  * [58] Changqian Yu, Jingbo Wang, Changxin Gao, Gang Yu, Chunhua Shen, and Nong Sang.  Context prior for scene segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12416–12425, 2020.
  * [59] Yuhui Yuan, Xilin Chen, and Jingdong Wang.  Object-contextual representations for semantic segmentation.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16, pages 173–190. Springer, 2020.
  * [60] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.  Scaling vision transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12104–12113, 2022.
  * [61] Hang Zhao, Xavier Puig, Bolei Zhou, Sanja Fidler, and Antonio Torralba.  Open vocabulary scene parsing.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2002–2010, 2017.
  * [62] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Semantic understanding of scenes through the ade20k dataset.  International Journal of Computer Vision, 127:302–321, 2019.
  * [63] Chong Zhou, Chen Change Loy, and Bo Dai.  Extract free dense labels from clip.  In ECCV, pages 696–712. Springer, 2022.
  * [64] Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Luc Van Gool.  Rethinking semantic segmentation: A prototype view.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2582–2593, 2022.

Figure 8: More architectural details of CAT-Seg: (a) overall architecture. (b)
embedding guidance. Note that a generalized embedding guidance is illustrated
to include different attention designs, _i.e_. shifted window attention [35]
or linear attention [29]. (c) upsampling decoder layer. GN: Group
Normalization [53]. LN: Layer Normalization [1].

## Appendix

In the following, we first provide more implementation details in Section A.
We then provide additional experimental results and ablation study in Section
B. Finally, we present qualitative results for all the benchmarks and human
part segmentation in Section C and a discussion of limitations in Section D.

##  A More Details

###  A.1 Architectural Details

In the following, we provide more architectural details. Our overall
architecture is first illustrated in Fig. 8 (a).

Embedding guidance. In this paragraph, we provide more details of embedding
guidance, which is designed to facilitate the cost aggregation process by
exploiting its rich semantics for a guidance. We first extract visual and text
embeddings from feature backbone and frozen CLIP text encoder [44],
respectively. The embeddings then undergo linear projection and concatenated
to the cost volume before query and key projections in aggregation layer. The
design is illustrated in Fig. 8 (b). In each case, for employing Swin
Transformer [35] as a feature backbone, the stage 3 output is used, whereas
for ResNet-101 [21], the output from the last conv layer of conv4_x is used as
our guidance.

Upsampling decoder. The detailed architecture is illustrated in Fig. 8 (c).
For multi-level features $E^{V}_{Dec}$, we leverage the features of last
layers from conv2_x and conv3_x when ResNet-101 [21] is used, whereas the
output features of stages 1 and 2 are leveraged when Swin Transformer [35] is
used.

Figure 9: Illustration of the proposed fine-tuning approach. To fine-tune the
CLIP image encoder, we optimize the attention layers and position embeddings
exclusively. This approach not only improves efficiency but also enhances
performance. Here, $N$ refers to the number of layers in the CLIP image
encoder.

###  A.2 Other Implementation Details

Training details. In Fig. 9, we visualize the proposed approach to fine-tune
the CLIP image encoder, which was introduced in Section 3.5 in the main paper.
A cost volume resolution of $H=W=24$ is used for training. The position
embeddings of the CLIP image encoder is initialized with bicubic interpolation
[51], and we set training resolution as $384\times 384$. For ViT-B and ViT-L
variants, we initialize CLIP [44] with official weights of ViT-B/16 and
ViT-L/14@336px respectively. For ViT-H and ViT-G variants, we initialize with
OpenCLIP [9] weights trained with LAION-2B [47]. The CLIP text encoder remains
frozen across all experiments. The feature backbone networks are pre-trained
on ImageNet-1k [13] in $224\times 224$ resolution for ResNet-101 [21], or on
ImageNet-21k in $384\times 384$ resolution for Swin Transformer [35]. All
hyperparameters are kept constant across the evaluation datasets.

(a) Feature Aggregation (b) Cost Aggregation

Figure 10: Visualization of aggregation baselines. (a) concatenates the
features extracted from CLIP image and text encoders to feed into the
upsampling decoder, while (b) constructs a cost volume using the image and
text features from CLIP.

Text prompt templates. To obtain text embeddings from the text encoder, we
follow CLIP [44] and form sentences with the class names, such as "a photo of
a {class}". We ensemble 80 text prompts originally used in CLIP for ImageNet
classification, without additionally curating text prompts or synonyms.

Feature and cost aggregation baselines. In this paragraph, we provide more
details of the architecture of two models introduced in Fig. 2: One is feature
aggregation method and the other is cost aggregation method. As shown in Fig.
10 (a), the feature aggregation method directly leverages the features
extracted from CLIP image encoder by feeding the concatenated image and text
embeddings into the upsampling decoder. Fig. 10 (b) shows the cost aggregation
approach that constructs cost volume instead, and subsequent embedding layer
processes it to feed into upsampling decoder.

###  A.3 Patch Inference

The practicality of Vision Transformer (ViT) [15] for high-resolution image
processing has been limited due to its quadratic complexity with respect to
the sequence length. As our model leverages ViT to extract image embeddings,
CAT-Seg may struggle to output to the conventional image resolutions commonly
employed in semantic segmentation literature, such as $640\times 640$ [8, 17],
without sacrificing some accuracy made by losing some fine-details. Although
we can adopt the same approach proposed in [63] to upsample the positional
embedding [63], we ought to avoid introducing excessive computational burdens,
and thus adopt an effective inference strategy without requiring additional
training which is illustrated in Fig. 11.

To this end, we begin by partitioning the input image into overlapping patches
of size $\frac{H}{N_{P}}\times\frac{W}{N_{P}}$. Intuitively, given an image
size of $640\times 640$, we partition the image to sub-images of size
$384\times 384$, which matches the image resolution at training phase, and
each sub-images has overlapping regions $128\times 128$. Subsequently, we feed
these sub-images and the original image that is resized to $384\times 384$
into the model. Given the results for each patches and the image, we merge the
obtained prediction, while the overlapping regions are averaged to obtain the
final prediction. In practice, we employ $N_{P}=2$, while adjusting the
overlapping region to match the effective resolution of $640\times 640$.

Figure 11: Illustration of the patch inference. During inference, we divide
the input image into patches, thereby increasing the effective resolution.

##  B Additional Ablation Study

###  B.1 Comparison of Aggregation Baselines

| Methods | A-847 | PC-459 | A-150 | PC-59 | PAS-20 | $\textnormal{PAS-20}^{b}$
---|---|---|---|---|---|---|---
(I) | Feature agg. + Freeze | 3.1 | 8.7 | 16.6 | 46.8 | 92.3 | 69.7
(II) | Feature agg. + F.T. | 3.8 | 10.9 | 19.1 | 53.5 | 96.2 | 74.2
(III) | Cost agg. + Freeze | 2.7 | 5.4 | 11.0 | 27.7 | 65.1 | 44.9
(IV) | Cost agg. + F.T. | 9.0 | 17.2 | 26.9 | 57.0 | 96.9 | 76.8

Table 7: Quantitative comparison between feature and cost aggregation. Cost
aggregation acts as an effective alternative to direct fine-tuning of CLIP
image encoder. F.T.: Fine-Tuning.

In this ablation study, we provide a quantitative comparison of two
aggregation baselines, feature aggregation and cost aggregation, in Table 7.
We freeze the CLIP image encoder and only optimize the upsampling decoder, and
the results are summarized in (I) and (III). Subsequently, in (II) and (IV),
we fine-tune the CLIP image encoder on top of (I) and (III). Our results show
that feature aggregation can benefit from fine-tuning, but the gain is only
marginal. On the other hand, cost aggregation benefits significantly from
fine-tuning, highlighting the effectiveness of cost aggregation to transfer
knowledge in CLIP encoder.

###  B.2 Ablation Study of the Number of Layers $N_{B}$

# of layers | A-847 | PC-459 | A-150 | PC-59 | PAS-20 | $\textnormal{PAS-20}^{b}$
---|---|---|---|---|---|---
1 | 10.7 | 16.3 | 30.6 | 61.2 | 96.6 | 81.7
2 | 10.8 | 20.4 | 31.5 | 62.0 | 96.6 | 81.8
3 | 10.8 | 20.0 | 31.5 | 61.9 | 96.5 | 81.8
4 | 10.8 | 20.4 | 31.5 | 62.1 | 96.6 | 82.2

Table 8: Effects of varying $N_{B}$.

Table 8 summarizes the effects of varying $N_{B}$. From the results, we find
that choosing higher $N_{B}$ does not always lead to performance improvements.
Note that we chose $N_{B}=2$ to balance between performance and model
capacity.

###  B.3 Ablation Study of Inference Strategy

Methods | A-847 | PC-459 | A-150 | PC-59 | PAS-20 | $\textnormal{PAS-20}^{b}$
---|---|---|---|---|---|---
CAT-Seg w/ training reso. | 10.1 | 18.9 | 29.8 | 59.8 | 96.1 | 79.7
CAT-Seg $\dagger$ | OOM | 14.8 | 30.4 | 59.7 | 95.9 | 80.3
CAT-Seg (ours) | 10.8 | 20.4 | 31.5 | 62.0 | 96.6 | 81.8

Table 9: Ablation study of inference strategy. CLIP with ViT-L is used for
ablation. $\dagger$: Process $640\times 640$ by upsampling position embedding
of CLIP ViT. OOM: Out-of-memory.

Table 9 presents effects of different inference strategies for our model. The
first row shows the results using the training resolution at inference time.
The second row represents a variant that upsamples the positional embedding
within the CLIP image encoder, allowing the encoder to process higher
resolution images. Finally, the last row adopts the proposed patch inference
strategy. It is shown that increasing the positional embedding introduces
substantial memory overhead. Moreover, for some datasets, using training
resolution yields better performance. On the other hand, our proposed approach
can bring large performance gains, while also ensuring high efficiency.

###  B.4 Effects of Upsampling Decoder

Methods | A-847 | PC-459 | A-150 | PC-59 | PAS-20 | $\textnormal{PAS-20}^{b}$
---|---|---|---|---|---|---
CAT-Seg w/o upsampling decoder | 10.1 | 18.9 | 29.8 | 59.8 | 96.1 | 79.7
CAT-Seg (ours) | 10.8 | 20.4 | 31.5 | 62.0 | 96.6 | 81.8

Table 10: Ablation study of upsampling decoder. CLIP with ViT-L is used for
ablation.

We provide an quantitative results of adopting the proposed upsampling decoder
in Table 10. The results show consistent improvements across all the
benchmarks.

###  B.5 Comparison of Inference Time

Methods | VLM | Mean | A-847 | PC-459 | A-150 | PC-59 | PAS-20
---|---|---|---|---|---|---|---
ZegFormer [14] | ViT-B/16 | 0.85 | - | - | - | - | -
CAT-Seg (ours) | ViT-B/16 | 0.73 (-0.12) | 1.83 (+0.98) | 1.04 (+0.19) | 0.40 (-0.45) | 0.22 (-0.63) | 0.14 (-0.71)
ZegFormer [14] | ViT-L/14 | 2.70 | - | - | - | - | -
CAT-Seg (ours) | ViT-L/14 | 0.79 (-1.91) | 1.91 (-0.79) | 1.09 (-1.61) | 0.46 (-2.24) | 0.28 (-2.42) | 0.21 (-2.49)
ZegFormer [14] | ViT-H/14 | 5.01 | - | - | - | - | -
CAT-Seg (ours) | ViT-H/14 | 0.88 (-4.13) | 2.01 (-3.00) | 1.21 (-3.80) | 0.54 (-4.47) | 0.35 (-4.66) | 0.28 (-4.73)
ZegFormer [14] | ViT-G/14 | 13.40 | - | - | - | - | -
CAT-Seg (ours) | ViT-G/14 | 1.14 (-12.26) | 2.32 (-11.08) | 1.51 (-11.89) | 0.77 (-12.63) | 0.58 (-12.82) | 0.51 (-12.89)

Table 11: Comparison of inference time in seconds. Faster scores are
highlighted in green, while slower scores are displayed in red. For ZegFormer
[14], we report only the mean inference time, as the method exhibits minimal
variance across different datasets.

In Table 11, we show run-time comparison at inference phase to the recent two-
stage approach [14]. We report the results with different VLM, e.g., ViT-B, L,
H and G. Note that the inference speed of the proposed method differs
depending on the number of categories, resulting in different run-times across
the evaluation datasets. In comparison to [14], we report the mean run-time,
as its performance demonstrates minimal variation with respect to the number
of categories.

When utilizing ViT-B for vision-language models, our method exhibits a
relatively slower inference time for some scenarios, including A-847 and
PC-459. However, as we scale the VLM to larger variants, such as ViT-L, H, and
G, our approach enjoys significantly faster inference by almost over 20 times
faster, highlighting the efficiency of the proposed method.

##  C More Qualitative Results

We provide more qualitative results on A-847 [62] in Fig. 12, PC-459 [41] in
Fig. 13, A-150 [62] in Fig. 14, and PC-59 [41] in Fig. 15. We also further
compare the results in A-847 [62] with other methods [14, 56, 33] in Fig. 16.

Qualitative results on part segmentation. We further compare human part
segmentation results with [33] in Fig. 17.

##  D Limitations

To evaluate open-vocabulary semantic segmentation results, we follow [17, 33]
and compute the metrics using the other segmentation datasets. However, since
the ground-truth segmentation maps involve some ambiguities, the reliability
of the evaluation dataset is somewhat questionable. For example, the last row
of Fig. 6 (e) exemplifies how our predictions in the mirror, “sky” and “car”,
as well as “plant” in between the “fence”, are classified as wrong
segmentation as the ground truth classes are “mirror” and “fence”.
Constructing a more reliable dataset including ground-truths accounting for
above issue for accurate evaluation is an intriguing topic.

Figure 12: Qualitative results on ADE20K [62] with 847 categories.
Figure 13: Qualitative results on PASCAL Context [41] with 459 categories.
Figure 14: Qualitative results on ADE20K [62] with 150 categories.
Figure 15: Qualitative results on PASCAL Context [41] with 59 categories.
Figure 16: Comparison of qualitative results on ADE20K [62] with 847
categories. We compare CAT-Seg with ZegFormer [14], ZSseg [56], and OVSeg [33]
on A-847 dataset.
Figure 17: Qualitative results on human part segmentation. We compare CAT-Seg
with OVSeg [33].


* [7] Villinger, C. _et al._ FIB/SEM tomography with TEM-like resolution for 3D imaging of high-pressure frozen cells.  _Histochemistry and Cell Biology_ 138, 549–556, DOI: 10.1007/s00418-012-1020-6 (2012).
  * [8] Heinrich, L. _et al._ Whole-cell organelle segmentation in volume electron microscopy.  _Nature_ 599, 141–146, DOI: 10.1038/s41586-021-03977-3 (2021).
  * [9] McCafferty, C. L. _et al._ Integrating cellular electron microscopy with multimodal data to explore biology across space and time.  _Cell_ 187, 563–584, DOI: 10.1016/j.cell.2024.01.005 (2024).
  * [10] Stone, M. B., Shelby, S. A. & Veatch, S. L.  Super-Resolution Microscopy: Shedding Light on the Cellular Plasma Membrane.  _Chemical Reviews_ 117, 7457–7477, DOI: 10.1021/acs.chemrev.6b00716 (2017).
  * [11] Schermelleh, L. _et al._ Super-resolution microscopy demystified.  _Nature Cell Biology_ 21, 72–84, DOI: 10.1038/s41556-018-0251-8 (2019).
  * [12] Hake, J. _et al._ Modelling cardiac calcium sparks in a three-dimensional reconstruction of a calcium release unit: Calcium sparks in reconstructed release unit.  _The Journal of Physiology_ 590, 4403–4422, DOI: 10.1113/jphysiol.2012.227926 (2012).
  * [13] Bell, M. K., Holst, M. V., Lee, C. T. & Rangamani, P.  Dendritic spine morphology regulates calcium-dependent synaptic weight change.  _The Journal of General Physiology_ 154, e202112980, DOI: 10.1085/jgp.202112980 (2022).
  * [14] Garcia, G. C. _et al._ Mitochondrial morphology provides a mechanism for energy buffering at synapses.  _Scientific Reports_ 9, 18306, DOI: 10.1038/s41598-019-54159-1 (2019).
  * [15] Holash, R. J. & MacIntosh, B. R.  A stochastic simulation of skeletal muscle calcium transients in a structurally realistic sarcomere model using MCell.  _PLoS Computational Biology_ 15, e1006712, DOI: 10.1371/journal.pcbi.1006712 (2019).
  * [16] Daversin-Catty, C., Richardson, C. N., Ellingsrud, A. J. & Rognes, M. E.  Abstractions and Automated Algorithms for Mixed Domain Finite Element Methods.  _ACM Transactions on Mathematical Software_ 47, 31:1–31:36, DOI: 10.1145/3471138 (2021).
  * [17] Wang, X. & Danuser, G.  Remeshing flexible membranes under the control of free energy.  _PLOS Computational Biology_ 18, e1010766, DOI: 10.1371/journal.pcbi.1010766 (2022).
  * [18] Means, S. _et al._ Reaction Diffusion Modeling of Calcium Dynamics with Realistic ER Geometry.  _Biophysical Journal_ 91, 537–557, DOI: 10.1529/biophysj.105.075036 (2006).
  * [19] Lee, C. T. _et al._ 3D mesh processing using GAMer 2 to enable reaction-diffusion simulations in realistic cellular geometries.  _PLOS Computational Biology_ 16, e1007756, DOI: 10.1371/journal.pcbi.1007756 (2020).
  * [20] Lee, C. T. _et al._ An Open-Source Mesh Generation Platform for Biophysical Modeling Using Realistic Cellular Geometries.  _Biophysical Journal_ 118, 1003–1008, DOI: 10.1016/j.bpj.2019.11.3400 (2020).
  * [21] Venkatraman, K. _et al._ Cristae formation is a mechanical buckling event controlled by the inner mitochondrial membrane lipidome.  _The EMBO Journal_ 42, e114054, DOI: 10.15252/embj.2023114054 (2023).
  * [22] Mesa, M. H., Garcia, G. C., Hoerndli, F. J., McCabe, K. J. & Rangamani, P.  Spine apparatus modulates Ca2+ in spines through spatial localization of sources and sinks, DOI: 10.1101/2023.09.22.558941 (2023).
  * [23] Alnæs, M. _et al._ The FEniCS Project Version 1.5.  _$ <$p$>$Archive of Numerical Software_ Vol 3, $<$strong$>$Starting Point and Frequency: $<$/strong$>$Year: 2013$<$/p$>$, DOI: 10.11588/ANS.2015.100.20553 (2015).
  * [24] Laughlin, J. G. _et al._ SMART: Spatial Modeling Algorithms for Reactions and Transport.  _Journal of Open Source Software_ 8, 5580, DOI: 10.21105/joss.05580 (2023).
  * [25] Laughlin, J. _et al._ RangamaniLabUCSD/smart: V2.2.1.  Zenodo, DOI: 10.5281/zenodo.11268687 (2024).
  * [26] Schaff, J., Fink, C., Slepchenko, B., Carson, J. & Loew, L.  A general computational framework for modeling cellular structure and function.  _Biophysical Journal_ 73, 1135–1146, DOI: 10.1016/S0006-3495(97)78146-3 (1997).
  * [27] Cowan, A. E., Moraru, I. I., Schaff, J. C., Slepchenko, B. M. & Loew, L. M.  Spatial Modeling of Cell Signaling Networks.  In _Methods in Cell Biology_ , vol. 110, 195–221, DOI: 10.1016/B978-0-12-388403-9.00008-4 (Elsevier, 2012).
  * [28] Kerr, R. A. _et al._ Fast Monte Carlo Simulation Methods for Biological Reaction-Diffusion Systems in Solution and on Surfaces.  _SIAM Journal on Scientific Computing_ 30, 3126–3149, DOI: 10.1137/070692017 (2008).
  * [29] Hucka, M. _et al._ The systems biology markup language (SBML): A medium forrepresentation and exchange of biochemical network models.  _Bioinformatics_ 19, 524–531, DOI: 10.1093/bioinformatics/btg015 (2003).
  * [30] Wu, Y. _et al._ Contacts between the endoplasmic reticulum and other membranes in neurons.  _Proceedings of the National Academy of Sciences of the United States of America_ 114, E4859–E4867, DOI: 10.1073/pnas.1701078114 (2017).
  * [31] Balay, S., Gropp, W., McInnes, L. C. & Smith, B. F.  PETSc, the portable, extensible toolkit for scientific computation (1998).
  * [32] Sun, M., Spill, F. & Zaman, M. H.  A Computational Model of YAP/TAZ Mechanosensing.  _Biophysical Journal_ 110, 2540–2550, DOI: 10.1016/j.bpj.2016.04.040 (2016).
  * [33] Scott, K. E., Fraley, S. I. & Rangamani, P.  A spatial model of YAP/TAZ signaling reveals how stiffness, dimensionality, and shape contribute to emergent outcomes.  _Proceedings of the National Academy of Sciences_ 118, e2021571118, DOI: 10.1073/pnas.2021571118 (2021).
  * [34] Maduram, J. H., Goluch, E., Hu, H., Liu, C. & Mrksich, M.  Subcellular curvature at the perimeter of micropatterned cells influences lamellipodial distribution and cell polarity.  _Cell motility and the cytoskeleton_ 65, 841–852, DOI: 10.1002/cm.20305 (2008).
  * [35] Bell, M., Bartol, T., Sejnowski, T. & Rangamani, P.  Dendritic spine geometry and spine apparatus organization govern the spatiotemporal dynamics of calcium.  _Journal of General Physiology_ 151, 1017–1034, DOI: 10.1085/jgp.201812261 (2019).
  * [36] Leung, A., Ohadi, D., Pekkurnaz, G. & Rangamani, P.  Systems modeling predicts that mitochondria ER contact sites regulate the postsynaptic energy landscape.  _NPJ Systems Biology and Applications_ 7, 26, DOI: 10.1038/s41540-021-00185-7 (2021).
  * [37] Li, J., Richmond, B. & Hong, T.  Organization of Ca2+ Signaling Microdomains in Cardiac Myocytes.  In Parinandi, N. L. & Hund, T. J. (eds.) _Cardiovascular Signaling in Health and Disease_ (Springer, Cham (CH), 2022).
  * [38] Hoshijima, M. _et al._ CCDB:3603, MUS MUSCULUS, T-tubules, sarcoplasmic reticulum, myocyte, DOI: doi:10.7295/W9CCDB3603 (2004).
  * [39] Mendelsohn, R. _et al._ Morphological principles of neuronal mitochondria.  _The Journal of Comparative Neurology_ 530, 886–902, DOI: 10.1002/cne.25254 (2022).
  * [40] Garcia, G. C., Gupta, K., Bartol, T. M., Sejnowski, T. J. & Rangamani, P.  Mitochondrial morphology governs ATP production rate.  _Journal of General Physiology_ 155, DOI: 10.1085/jgp.202213263 (2023).
  * [41] Meyers, J., Craig, J. & Odde, D. J.  Potential for Control of Signaling Pathways via Cell Size and Shape.  _Current Biology_ 16, 1685–1693, DOI: 10.1016/j.cub.2006.07.056 (2006).
  * [42] Rangamani, P. _et al._ Decoding Information in Cell Shape.  _Cell_ 154, 1356–1369, DOI: 10.1016/j.cell.2013.08.026 (2013).
  * [43] Neves, S. R. _et al._ Cell Shape and Negative Links in Regulatory Motifs Together Control Spatial Information Flow in Signaling Networks.  _Cell_ 133, 666–680, DOI: 10.1016/j.cell.2008.04.025 (2008).
  * [44] Calizo, R. C. _et al._ Cell shape regulates subcellular organelle location to control early Ca2+ signal dynamics in vascular smooth muscle cells.  _Scientific Reports_ 10, 17866, DOI: 10.1038/s41598-020-74700-x (2020).
  * [45] Johnson, G. T. _et al._ Building the next generation of virtual cells to understand cellular biology.  _Biophysical Journal_ 122, 3560–3569, DOI: 10.1016/j.bpj.2023.04.006 (2023).
  * [46] Viana, M. P. _et al._ Integrated intracellular organization and its variations in human iPS cells.  _Nature_ 613, 345–354, DOI: 10.1038/s41586-022-05563-7 (2023).
  * [47] Xu, C. S. _et al._ An open-access volume electron microscopy atlas of whole cells and tissues.  _Nature_ 599, 147–151, DOI: 10.1038/s41586-021-03992-4 (2021).
  * [48] Edwards, J. _et al._ VolRoverN: Enhancing Surface and Volumetric Reconstruction for Realistic Dynamical Simulation of Cellular and Subcellular Function.  _Neuroinformatics_ 12, 277–289, DOI: 10.1007/s12021-013-9205-2 (2014).
  * [49] Hu, Y., Schneider, T., Wang, B., Zorin, D. & Panozzo, D.  Fast tetrahedral meshing in the wild.  _ACM Transactions on Graphics_ 39, 117:117:1–117:117:18, DOI: 10.1145/3386569.3392385 (2020).
  * [50] Mitusch, S. K., Funke, S. W. & Dokken, J. S.  Dolfin-adjoint 2018.1: Automated adjoints for FEniCS and Firedrake.  _Journal of Open Source Software_ 4, 1292, DOI: 10.21105/joss.01292 (2019).
  * [51] Novak, I. L. _et al._ Diffusion on a Curved Surface Coupled to Diffusion in the Volume: Application to Cell Biology.  _Journal of computational physics_ 226, 1271–1290, DOI: 10.1016/j.jcp.2007.05.025 (2007).
  * [52] Hepburn, I., Chen, W., Wils, S. & De Schutter, E.  STEPS: Efficient simulation of stochastic reaction–diffusion models in realistic morphologies.  _BMC Systems Biology_ 6, 36, DOI: 10.1186/1752-0509-6-36 (2012).
  * [53] Hepburn, I. _et al._ Vesicle and reaction-diffusion hybrid modeling with STEPS.  _Communications Biology_ 7, 1–15, DOI: 10.1038/s42003-024-06276-5 (2024).
  * [54] Roberts, E., Stone, J. E. & Luthey-Schulten, Z.  Lattice microbes: High-performance stochastic simulation method for the reaction-diffusion master equation.  _Journal of Computational Chemistry_ 34, 245–255, DOI: 10.1002/jcc.23130 (2013).
  * [55] Hoops, S. _et al._ COPASI—a COmplex PAthway SImulator.  _Bioinformatics_ 22, 3067–3074, DOI: 10.1093/bioinformatics/btl485 (2006).
  * [56] Clark, A. P., Chowkwale, M., Paap, A., Dang, S. & Saucerman, J. J.  Logic-based modeling of biological networks with Netflux.  _bioRxiv_ 2024.01.11.575227, DOI: 10.1101/2024.01.11.575227 (2024).
  * [57] Geuzaine, C. & Remacle, J.-F.  Gmsh: A 3-D finite element mesh generator with built-in pre- and post-processing facilities: THE GMSH PAPER.  _International Journal for Numerical Methods in Engineering_ 79, 1309–1331, DOI: 10.1002/nme.2579 (2009).
  * [58] Francis, E. A., Finsberg, H. N. & Dokken, J. S.  Biological test cases implemented in SMART.  Zenodo, DOI: 10.5281/zenodo.11268945 (2024).
  * [59] Alnæs, M. S., Logg, A., Ølgaard, K. B., Rognes, M. E. & Wells, G. N.  Unified form language: A domain-specific language for weak formulations of partial differential equations.  _ACM Transactions on Mathematical Software_ 40, 9:1–9:37, DOI: 10.1145/2566630 (2014).
  * [60] Grecco, H. E.  Pint: Physical quantities module.
  * [61] Dalcin, L. D., Paz, R. R., Kler, P. A. & Cosimo, A.  Parallel distributed computing using Python.  _Advances in Water Resources_ 34, 1124–1139, DOI: 10.1016/j.advwatres.2011.04.013 (2011).
  * [62] Meurer, A. _et al._ SymPy: Symbolic computing in Python.  _PeerJ Computer Science_ 3, e103, DOI: 10.7717/peerj-cs.103 (2017).