# Neural Clustering based Visual Representation Learning

Guikun Chen1, Xia Li2, Yi Yang1, Wenguan Wang1111Corresponding author: Wenguan
Wang.
1 ReLER, CCAI, Zhejiang University 2 ETH Zürich
https://github.com/guikunchen/FEC/

###### Abstract

We investigate a fundamental aspect of machine vision: the measurement of
features, by revisiting clustering, one of the most classic approaches in
machine learning and data analysis. Existing visual feature extractors,
including Conv-Nets, ViTs, and MLPs, represent an image as rectangular
regions. Though prevalent, such a grid-style paradigm is built upon
engineering practice and lacks explicit modeling of data distribution. In this
work, we propose feature extraction with clustering (FEC), a conceptually
elegant yet surprisingly ad-hoc interpretable neural clustering framework,
which views feature extraction as a process of selecting representatives from
data and thus automatically captures the underlying data distribution. Given
an ima-ge, FEC alternates between grouping pixels into individual clusters to
abstract representatives and updating the deep features of pixels with current
representatives. Such an ite-rative working mechanism is implemented in the
form of several neural layers and the final representatives can be used for
downstream tasks. The cluster assignments across layers, which can be viewed
and inspected by humans, make the forward process of FEC fully transparent and
empower it with promising ad-hoc interpretability. Extensive experiments on
various visual recognition models and tasks veri-fy the effectiveness,
generality, and interpretability of FEC. We expect this work will provoke a
rethink of the current de facto grid-style paradigm.

##  1 Introduction

The measurement of features, which explores how to extract abstract,
meaningful features from high-dimensional image data, is a topic of enduring
interest in machine vision throughout its history [1, 2, 3]. This pursuit,
initially dominated by manually engineered descriptors [4, 5, 6, 7, 8, 9], has
evolved under the influence of deep learning paradigms, transitioning from
convolutional landscapes [10, 11] to the frontiers of attention-driven
mechanisms [12, 13] and MLP-based approaches [14, 15]. Convolutional networks
(Conv-Nets, Fig. 1a) treat an image as rectangular regions and execute in a
sliding window manner. Attention-based methods (Fig. 1b) usually divide an
image into several non-overlap patches and use an additional [CLS] token to
represent the whole image. MLP-based backbones (Fig. 1c) also follow the grid-
style paradigm while extracting features without convolution or attention
operations.

Figure 1: How to represent an image in a low-dimensional space and what could
explain it? (abc) Existing visual backbones rely on the computational modeling
of rigid grids. (d) Derived from a neural clustering view, FEC reformulates
the procedures of feature extraction as clustering, thereby representing the
image with its representatives. Our approach possesses promising _ad-hoc_
interpretability and demonstrates the emergence of segmentation despite being
trained only on the classification task.

Upon observing the array of visual backbones shown in Fig. 1, the following
questions naturally arise: ❶ What is the relation between them? and more
critically, ❷ If these neural networks indeed implicitly capture some
intrinsic properties of image data, might there exist a more transparent and
interpretable method to measure the visual features?

The pursuit of the question ❶ uncovers a persistent adherence to a grid-
centric view in the realm of image data analysis [16, 17, 18]. Concretely, the
basic elements during the forward process of existing backbones are
rectangular image regions, _e.g_., the kernels (filters), sliding window, and
receptive field in convolution-based backbones, or the image patches in vision
Transformers (ViTs) and MLPs. Such a widespread paradigm, though instrumental
in the evolution of convolutional networks and their successors, seems to be
based more on engineering convention than on the emulation of natural image
structures. Most existing efforts are expected to generate more abstract
features as the network’s layers deepen, while nobody knows how they make it
[19]. Therefore, question ❷ becomes more fundamental: ❸ What are the inherent
limitations of this grid-style paradigm? and ❹ Can we evolve beyond the grid-
based uniformity assumption that fails to encapsulate the organic structure of
images?

Driven by question ❸, we uncover two critical limitations: First, the grid
model is at odds with the true nature of pixel organization, thereby failing
to grasp the comple-xity of data distribution [20]. Second, the black-box
nature of deep feature extractors impedes interpretability, veiling the
rationale behind feature selection and significance. This leads us to the
question ❹, probing the opacity of current methodologies and their divergence
from human perception and cognition [21, 22, 23], which possess a remarkable
ability to break down visual scenes into semantic-meaning components. The goal
is to build feature extractors that can better capture the data distribution
of pixels and mirror the cognitive processes of human vision so as to enhance
both interpretability and transparency. To bridge the identified gaps, a
fundamental paradigm shift is imperative: i) pivo-ting from the grid-view of
image representations towards a more fluid model that embraces the dynamic
nature of visual data; and ii) stepping away from the black-box models towards
an ambitious hybrid that integrates powerful representation learning and
interpretable feature encoding.

In this vein, we introduce FEC (§3), a _mechanistically interpretable_
backbone that roots in the principle of clustering. It begins with a window-
based pooling to generate pixel blocks that serve as initial elements.
Afterward, FEC iterates through two key processes: i) Clustering-based Feature
Pooling. Neural clustering is used to model representatives of the given
inputs (pixel blocks or previous clusters), leading to more abstract (growing)
clusters. Due to the clustering nature, a representative (cluster)
_explicitly_ represents a set of pixels in any position. This is where FEC
differs from grid-style paradigm. ii) Clustering-based Feature Encoding. Here
the representatives are first estimated and then used for redistributing
features to each pixel given the similarity between the pixel and its
representative. Within such a clustering based framework, the basic elements
during FEC’s forward process are gradually growing clusters.

FEC exhibits several compelling characteristics: First, enhanced simplicity,
and _transparency_. The streamlined design, coupled with the semantic meaning
of clustering during feature extraction, renders FEC both conceptually elegant
and straightforward to implement. The mechanism by which representatives are
modeled ensures that the forward process of FEC is fully transparent. Second,
automated discovery of _underlying data distribution_. The deterministic
clustering reveals the latent relationships between pixels of the image data,
capturing the varying semantic granularity that standard backbones might
overlook. As depicted in Fig. 1d, FEC can learn to distinguish non-grid
semantic regions autonomously without explicit supervision of cluster
assignments. Third, _ad-hoc interpretability_. If further inspecting the
_cluster assignments_ in each feature pooling and combining them together, FEC
can interpret its prediction based on the aggregated clusters during forward
process and allow users to intuitively view the semantic components. Such _ad-
hoc_ interpretability is valuable in safety-sensitive scenarios and provides a
feasible way for humans to understand the _forward_ process of feature
extraction.

By${}_{\\!}$ answering${}_{\\!}$ questions${}_{\\!}$ ❶-❹,${}_{\\!}$
we${}_{\\!}$ formalize${}_{\\!}$ visual${}_{\\!}$ feature extraction within a
neural clustering-based, fully transpa-rent framework, bridging the gap
between classic clustering algorithm and neural network interpretability. We
provide a literature review and related discussions in §4. FEC represents an
intuitive and versatile feature extractor, seamlessly compatible with
established visual recognition models and tasks, requiring no modifications.
Experimental results in §5.1 show FEC achieves 72.7% top-1 accuracy on
ImageNet [24] with only 5.5M parameters. In §5.2, with the modeled
representatives, FEC can interpret how it captures the data distribution. In
§5.3 and §5.4, the transferability and versatility of FEC are validated on
three fundamental recognition tasks. Finally, we draw conclusions in §6.

##  2 Existing Visual Feature Extractors as Fixed Grid-style Parsers

${}_{\\!}$Problem${}_{\\!}$ Statement.${}_{\\!}$ Here${}_{\\!}$ we${}_{\\!}$
study${}_{\\!}$ the${}_{\\!}$ standard${}_{\\!}$ classifica- tion${}_{\\!}$
setting.${}_{\\!}$ Let${}_{\\!}$ $\mathcal{X}\\!$ be${}_{\\!}$ the${}_{\\!}$
input${}_{\\!}$ space${}_{\\!}$ (_i.e_.,${}_{\\!}$ image${}_{\\!}$
space${}_{\\!}$ for${}_{\\!}$ vi- sual${}_{\\!}$ recognition),${}_{\\!}$
and${}_{\\!}$ $\mathcal{Y}\\!=\\!\\{\texttt{cat},\cdots,\texttt{dog}\\}$
denote the set${}_{\\!}$ of${}_{\\!}$ semantic${}_{\\!}$ categories,${}_{\\!}$
_e.g_.,${}_{\\!}$ $|\mathcal{Y}|\\!=\\!1000$ for${}_{\\!}$ ImageNet-1K[24].

${}_{\\!}$Standard${}_{\\!}$ Pipeline.${}_{\\!}$ The${}_{\\!}$
current${}_{\\!}$ common${}_{\\!}$ practice${}_{\\!}$ of${}_{\\!}$ classifi-
cation${}_{\\!}$ is${}_{\\!}$ to${}_{\\!}$ decompose${}_{\\!}$ the${}_{\\!}$
deep${}_{\\!}$ neural${}_{\\!}$ network${}_{\\!}$
$h:\mathcal{X}\\!\mapsto\\!\mathcal{Y}$ into${}_{\\!}$
$f\\!:\\!\mathcal{X}\\!\mapsto\\!\mathcal{F}$ and${}_{\\!}$
$g\\!:\\!\mathcal{F}\\!\mapsto\\!\mathcal{Y}$ that${}_{\\!}$ $h\\!=\\!g\circ
f$,${}_{\\!}$ where${}_{\\!}$ $f\\!$ and${}_{\\!}$ $g$ denote${}_{\\!}$
the${}_{\\!}$ feature${}_{\\!}$ extractor${}_{\\!}$ and${}_{\\!}$
the${}_{\\!}$ classifier,${}_{\\!}$ respectively.${}_{\\!}$ Given${}_{\\!}$
an${}_{\\!}$ input${}_{\\!}$ image${}_{\\!}$ $\bm{X}$, $f$ maps${}_{\\!}$
it${}_{\\!}$ into${}_{\\!}$ a${}_{\\!}$ d-dimensional${}_{\\!}$ re-
presentation${}_{\\!}$ space${}_{\\!}$ $\mathcal{F}\\!\in\\!\mathbb{R}^{C}$,
_i.e_., $\bm{f}\\!=\\!f(\bm{X})\in\mathbb{R}^{C}$; and $g$ fur- ther predicts
the class prediction $\hat{y}$ based on the intermediate feature${}_{\\!}$
$\bm{f}$,${}_{\\!}$ _i.e_.,${}_{\\!}$
$\hat{y}\\!=\\!g(\bm{f})\\!\in\\!\mathcal{Y}$.${}_{\\!}$ This${}_{\\!}$
work${}_{\\!}$ focuses${}_{\\!}$ on${}_{\\!}$ the${}_{\\!}$ $f\\!$ only.

ConvNets.${}_{\\!}$ Convolution-based${}_{\\!}$ feature${}_{\\!}$
extractor${}_{\\!}$ has${}_{\\!}$ domina- ted academia and industry for years,
whose detailed architectures are reviewed as follows. Formally, given an input
image $\bm{X}\\!\in\\!\mathbb{R}^{3\\!\times\\!H\\!\times\\!W}$, ConvNets
extract feature embeddings $\\{\bm{F}^{l}\\}_{l\\!=\\!1}^{4}$, where the
resolutions are $\frac{1}{4},\frac{1}{8},\frac{1}{16},\frac{1}{32}$ of the
original image,${}_{\\!}$ respectively.${}_{\\!}$ These${}_{\\!}$
four${}_{\\!}$ feature${}_{\\!}$ embeddings${}_{\\!}$ are${}_{\\!}$
generated${}_{\\!}$ by four separate stages, each containing grid-style
feature pooling and encoding. Taking the 2nd stage of ResNet18 [11] as an
example, given
$\bm{F}^{1}\\!\in\\!\mathbb{R}^{64\\!\times\\!56\\!\times\\!56}$ from
the${}_{\\!}$ 1st${}_{\\!}$ stage,${}_{\\!}$ a${}_{\\!}$ low-
dimensional${}_{\\!}$ feature${}_{\\!}$ map${}_{\\!}$ is${}_{\\!}$
generated${}_{\\!}$ as:${}_{\\!}$

$\hat{\bm{F}}^{2}=\texttt{grid\\_pool}(\bm{F}^{1})\in\mathbb{R}^{128\\!\times\\!28\\!\times\\!28},\vspace{-5pt}$
(1)

where grid_pool denotes a convolutional layer with a stride of $2$, which can
also be implemented with max poo- ling, average pooling, _etc_. After that,
feature encoding is performed to get the outputs of this stage:

$\bm{F}^{2}=\texttt{encode}(\hat{\bm{F}}^{2})\in\mathbb{R}^{128\\!\times\\!28\\!\times\\!28},\vspace{-5pt}$
(2)

where encode denotes several convolutional layers which keep the output
resolution consistent. This step is the key essential to distinguish different
backbones, which is implemented as self-attention in ViTs and token mixers in
MLPs.

ViTs [13] and MLPs [14] both commence their operations by generating visual
token embeddings for all non-overlap patches of an image:

$\bm{E}=\texttt{token\\_emb}(\bm{X}).\vspace{-5pt}$ (3)

After which, ViTs use the [CLS] token to represent the whole image while MLPs
take the average of all patch embeddings to do so. Since the sequence of image
patches is used throughout the forward process of feature extraction, we also
categorize them as the grid-style paradigm.

In general, existing backbones are built upon the computational modeling of
rigid grids, which uses regular regions to represent an image. However, this
paradigm underestimates the dynamic nature of visual scenes, assuming a
spatial uniformity that clashes with the _underlying data distribution of
pixels_. In addition, it overlooks the essence of human perception, which does
not bind itself to rigid grids but instead fluidly navigates through semantic
context [25].

After tackling question ❸, in the next section we will detail our clustering
based transparent visual feature extractor, which serves as a solid response
to question ❹.

$\times L^{1}$

$\times L^{i}$

Eq. 4Eq. 6

$\bm{X}$

$\bm{X}$

$\bm{R}_{m}$$\hat{\bm{F}}^{1}$$\bm{F}^{1}$$\bm{F}^{4}$$\bar{\bm{S}}^{l-1}$$\bar{\bm{S}}^{l}$Eq.
8Eq. 7

Figure 2: (a) Overall framework of FEC (§3). Each stage $i$ contains $L^{i}$
clustering-based encode layers. (b) Illustration of our clustering-based
feature pooling and encoding. (c) The basic elements during FEC’s forward
process are _growing clusters_ instead of _image patches_.

##  3 Feature Extraction with Clustering (FEC)

Algorithmic Overview. FEC is a neural clustering-based framework for visual
feature extraction, building upon the idea of selecting representatives
hierarchically. Concretely, given an input image, FEC initiates with a
standard convolution whose kernel-size and stride are set to be 4 and 4,
respectively. Subsequent feature extraction builds upon these resulted
$4\\!\times\\!4$ pixel patches. Afterward, FEC alternates between the
following steps for each given input feature:

  * •

Clustering-based Feature Encoding, _i.e_.,
$\mathbb{R}^{C\\!\times\\!W\\!\times\\!H}\\!\mapsto\\!\mathbb{R}^{C\\!\times\\!W\\!\times\\!H}$.
It divides pixels from feature maps into several non-overlap clusters, by
projecting the pixel features into a similarity space and using adaptive (the
stride and kernel-size are automatically selected to adapt to the desired
resolution) average pooling to initialize the cluster centers. As such,
cluster assignments can be obtained according to the similarity between pixels
and centers. Then, the pixel features are aggregated to construct cluster
representations. Subsequently, feature dispatching, which uses the aggregated
center to redistribute pixel features within the cluster, is employed to
encode pixel-level features, _i.e_., information communication. Hence elements
(pixels) inside the same cluster become more consistent in the feature space.

  * •

Clustering-based Feature Pooling, _i.e_.,
$\mathbb{R}^{C\\!\times\\!W\\!\times\\!H}\\!\mapsto\\!\mathbb{R}^{C^{\prime}\\!\times\\!W/2\\!\times\\!H/2}$.
Similar to the feature encoding process,${}_{\\!}$ this module uses clustering
to obtain the cluster assignments. The difference is that it directly returns
the cluster representations to form low-dimensional feature maps without
encoding features. These strategies not only preserve the compositional
structures of varying semantic${}_{\\!}$ levels${}_{\\!}$ but${}_{\\!}$
also${}_{\\!}$ seamlessly${}_{\\!}$ integrate${}_{\\!}$ the${}_{\\!}$
concept${}_{\\!}$ of${}_{\\!}$ clustering into the feed-forward feature
extraction.

To sum up, we formalize our target task — extracting deep features for visual
inputs — as representative selection. By doing so, the intermediate
representatives can be a natural substitute for grid_pool. Since these
representatives are computed from the context of each input, it can also be
used to communicate information by feature dispatching, which serves the same
purpose as the encode operation. Next, we will detail the operations of those
essential parts of FEC.

${}_{\\!}$Center${}_{\\!}$ Initialization.${}_{\\!}$ Given${}_{\\!}$
an${}_{\\!}$ input${}_{\\!}$ feature${}_{\\!}$ map${}_{\\!}$
$\bm{F}\\!\\!\in\\!\\!\mathbb{R}^{N\\!\times\\!C}$,${}_{\\!}$ where${}_{\\!}$
$N\\!\\!=\\!\\!W\\!\\!\times\\!\\!H$,${}_{\\!}$ we${}_{\\!}$ first${}_{\\!}$
project${}_{\\!}$ it${}_{\\!}$ into${}_{\\!}$ key${}_{\\!}$ and${}_{\\!}$
value${}_{\\!}$ spaces${}_{\\!}$ using${}_{\\!}$ $1\\!\times\\!1\\!$
convolutional${}_{\\!}$ layers,${}_{\\!}$ resulting${}_{\\!}$ in${}_{\\!}$
$\bm{K}\in\mathbb{R}^{N\\!\times\\!C^{\prime}\\!}$ and${}_{\\!}$
$\bm{V}\\!\in\\!\mathbb{R}^{N\\!\times\\!C^{\prime}}$, respectively. Here
$C^{\prime}$ is a hyperparameter to control the dimension. We then initialize
the cluster centers with their key and value features:

$\displaystyle=\texttt{ada\\_pool}_{O}(\bm{K})\in\mathbb{R}^{O\\!\times\\!C^{\prime}},$
(4) $\displaystyle[\bm{C}_{1}^{v};{\cdots};\bm{C}_{O}^{v}]$
$\displaystyle=\texttt{ada\\_pool}_{O}(\bm{V})\in\mathbb{R}^{O\\!\times\\!C^{\prime}},$

where${}_{\\!}$ ada_pool${}_{O}\\!$ refers${}_{\\!}$ generating${}_{\\!}$
$O$${}_{\\!}$ feature${}_{\\!}$ centers${}_{\\!}$ in${}_{\\!}$ the${}_{\\!}$
projected${}_{\\!}$ spaces${}_{\\!}$ using${}_{\\!}$ adaptive${}_{\\!}$
average${}_{\\!}$ pooling.${}_{\\!}$ As such,${}_{\\!}$ the${}_{\\!}$
centers${}_{\\!}$ are${}_{\\!}$ initialized${}_{\\!}$ adaptively${}_{\\!}$
for${}_{\\!}$ each${}_{\\!}$ input${}_{\\!}$ itself and gradients can be
passed through all indices.

${}_{\\!}$Representative${}_{\\!}$ Modeling.${}_{\\!}$ To${}_{\\!}$
assign${}_{\\!}$ each${}_{\\!}$ element${}_{\\!}$ into${}_{\\!}$ a${}_{\\!}$
cluster,${}_{\\!}$ the${}_{\\!}$ similarity${}_{\\!}$ matrix${}_{\\!}$
$\bm{M}$ is${}_{\\!}$ computed${}_{\\!}$ as:

$\bm{M}=\langle\bm{K},[\bm{C}_{1}^{k};{\cdots};\bm{C}_{O}^{k}]\rangle\in\mathbb{R}^{N\\!\times\\!O},\vspace{-4pt}$
(5)

where $\langle\cdot,\cdot\rangle$ denotes the cosine similarity. Each element
is assigned to a cluster exclusively according to $\arg\max(\bm{M})$,
resulting in an assignment matrix $\bm{A}$ which contains $N$ one-
hot${}_{\\!}$ vectors.${}_{\\!}$ With${}_{\\!}$ the${}_{\\!}$
cluster${}_{\\!}$ assignments,${}_{\\!}$ the${}_{\\!}$ deep${}_{\\!}$ features
of the $o$-th representative (cluster) are aggregated by:

$\\!\\!\bm{R}_{o}\\!=\\!\big{(}\bm{C}_{o}^{v}\\!+\Sigma_{n=1}^{N}\bm{A}_{no}\bm{V}_{n}\big{)}/\big{(}{1+\Sigma_{n=1}^{N}\bm{A}_{no}}\big{)}\\!\in\\!\mathbb{R}^{C^{\prime}}.\vspace{-4pt}$
(6)

So${}_{\\!}$ far,${}_{\\!}$ we’ve obtained${}_{\\!}$ the${}_{\\!}$ low-
dimensional${}_{\\!}$ features${}_{\\!}$ $\bm{R}\\!=\\!$
$[\bm{R}_{1};{\cdots};\bm{R}_{O}]$ (_i.e_., representatives), which can
seamlessly replace the grid_pool module in grid-style paradigm.

Feature Dispatching. With the insight that elements inside the same cluster
shall have similar properties, we propose to propagate the information within
each cluster to enhance this phenomenon. Concretely, we choose to achieve this
with modulated propagation with respect to the similarity with the
corresponding center [26, 27]. For element $n$ in cluster $o$, we update its
feature $\bm{F}_{n}\\!\in\\!\mathbb{R}^{C}$ by:

$\bm{F}_{n}^{\prime}=\bm{F}_{n}+\texttt{MLP}(\sigma(\alpha\bm{M}_{no}+\beta)\bm{R}_{o})\in\mathbb{R}^{C},\vspace{-4pt}$
(7)

where $\sigma$ denotes the sigmoid function. $\alpha$ and $\beta$ are
learnable parameters to scale and shift the similarity. The updated features
$[\bm{F}_{1}^{\prime};{\cdots};\bm{F}_{N}^{\prime}]$ are the outputs of the
encode operation in FEC (can be repeated many times). Since the center
features are adaptively sampled from a group of elements, this dispatching
process enables effective communication between elements within a cluster and
those formed in the cluster center, leading to the overall understanding of
the underlying data distribution and context of the image. From a higher
perspective, FEC can be seen as an _exclusive_ variant (non-overlap clusters)
of self-attention [12], _e.g_., center initialization _vs_ key and value
matrices, representative modeling _vs_ attention scores, and feature
dispatching _vs_ weighted aggregation. More implementation details are left in
the appendix.

Automated Discovery of Underlying Data Distribution. The cluster assignments
elucidate not only the relationship between elements and their representatives
but also the underlying data distribution within the feature maps. Pixels
assigned the $o$-th centroid at the $l$-th level
$\\{n\mid\bm{A}_{no}^{l}=1\\}$ coalesce into a cluster (segment)
$\bm{S}_{o}^{l}$, thereby $\\{\bm{S}_{o}^{l}\mid 1\leq o\leq O\\}$ decomposing
the entire feature map into $O$ discernible segments at the $l$-th layer. By
linking the clusters across sequential layers as follows:

$\bar{\bm{S}}_{h}^{l}=\texttt{Union}(\\{\bm{S}_{o}^{l-1}\mid\bm{A}_{oh}^{l}=1\\}),\vspace{-4pt}$
(8)

we create a hierarchical pyramid
$[\bar{\bm{S}}^{1},\bar{\bm{S}}^{2},\cdots,\bar{\bm{S}}^{L}]$ which coalesces
pixels into increasingly larger segments, and explicitly reveals the
underlying data distribution.

_Ad-hoc_ Interpretability. The described configuration facilitates a direct
forward process $l\\!\\!=\\!\\!1\\!\\!\to\\!\\!L$ that yields a linked spatial
decomposition $[\bar{\bm{S}}^{1},\bar{\bm{S}}^{2},\cdots,\bar{\bm{S}}^{L}]$,
intuitively conveying image parsing to the observer. In contrast, earlier
techniques (_e.g_., Grad-CAM [28]) demand a retrospective process to
accentuate the activated regions. These methods typically require complex
post-processing to elucidate the concealed parsing mechanism. However, FEC is
_mechanistically interpretable_ because its forward process based on gradually
growing clusters (segments) is _fully transparent_. See [29] for a more
detailed discussion.

Versatility. After modeling the feature extraction process with
representatives, one may wonder about the applicability of this new paradigm
to dense prediction tasks. For instance, in the detection task, the training
process of widely-used models like YOLO [30] and Faster-RCNN [31], relies on
the grid-based label assignments (anchors). To keep the necessary grid-based
information, we introduce the residual connection [11] to the feature of
representatives:

$\bm{R}_{i}=\texttt{ResConn}(\bm{F}_{i})+\bm{R}_{i}\in\mathbb{R}^{C^{\prime}}.\vspace{-4pt}$
(9)

This modification allows each $\bm{R}_{i}\\!\in\\!\mathbb{R}^{C^{\prime}}$ to
represent both the rectangular regions in grid-style paradigms and the
selected representatives in our clustering-based paradigm. As shown in Fig.
2a, we also adopt four stages during feature extraction as in existing
standard backbones, resulting in the same resolutions of output features. In a
nutshell, FEC marks a fundamental paradigm shift for visual feature extraction
while retaining full compatibility with previous works. The computational
pipeline is also detailed in §5.2.

Adaptation to Downstream Tasks. As aforementioned, with the introduction of
residual connections, FEC can be seamlessly incorporated into dense prediction
tasks like detection and segmentation without any architectural modifications.
In terms of classification, we use the standard classification head over the
final feature map $\bm{F}^{L}$, _i.e_., a single-layer MLP, which takes the
average of all representatives. More${}_{\\!}$ details${}_{\\!}$ are${}_{\\!}$
left${}_{\\!}$ in${}_{\\!}$ the${}_{\\!}$ appendix.${}_{\\!}$ We${}_{\\!}$
expect${}_{\\!}$ the${}_{\\!}$ recent${}_{\\!}$ set-prediction architectures
(_e.g_., DETR [32]) can better utilize the modeled representatives and leave
it as future works.

##  4 Related Work

In this section, we review representative work in clustering, visual${}_{\\!}$
feature${}_{\\!}$ extraction,${}_{\\!}$ and${}_{\\!}$ neural${}_{\\!}$
network${}_{\\!}$ interpretability.

Clustering stands as a foundational technique in machine learning that
involves grouping akin data points together based on their intrinsic
characteristics. Given numerous data, the goal of clustering is to model
meaningful clusters (given pre-defined numbers or not), which can be viewed as
the summary of the raw data. Subsequently, the similarities between each data
and the cluster representations can be quantified. Clustering has been widely
used in various fields, _e.g_., scene understanding [33, 34, 35, 36], point
clouds [37, 38, 39], segmentation [40, 41, 42, 43, 44], and AI4Science [45,
46, 47, 48].

In a departure from previous endeavors that harnessed clustering mechanisms as
add-on heads to facilitate specific tasks, our approach pioneers the idea of
learning universal visual representations from the clustering view. The
proposed clustering-based feature extraction shares a kinship with the classic
vision technique of clustering similar pixels [49]. Nonetheless, our
innovation lies in its capacity to capture underlying data distribution and
generate continuous representations for downstream tasks. Via reformulating
the entire process of feature extraction as selecting representatives in a
clustering fashion, FEC capitalizes on the robustness of end-to-end
representation learning while preserving the transparent nature inherent to
clustering. Inspired by biological systems that process inputs from different
modalities simultaneously, Perceivers [50, 51] model a set of _latent vectors_
correlated with inputs. FEC is motivated from the classic idea of clustering,
and its intermediate elements have _explicit_ meaning, _i.e_., segments (Eq.
8).

${}_{\\!}$Feature Extractors for Vision.${}_{\\!}$ Feature extraction is a
pivotal aspect of machine learning, serving as a crucial step in transforming
raw data into informative representations conducive to computational analysis.
While some pre-deep learning methods [52, 53] also aim to get rid of grids,
they are not _fully_ end-to-end and do not scale well. Since the 2010s,
convolutional networks [54] have reigned supreme in computer vision, with
significant milestones like VGG [10], ResNet [11], ConvNeXt [55] and so on
[56, 57, 58] consistently pushing the envelope by deepening and optimizing
convolutional layers, enabling the extraction of hierarchical and abstract
features.${}_{\\!}$ Innovations${}_{\\!}$ like${}_{\\!}$ depthwise${}_{\\!}$
convolution${}_{\\!}$ [59]${}_{\\!}$ and${}_{\\!}$ deformable convolution [60]
have further enhanced feature extraction within${}_{\\!}$ these${}_{\\!}$
networks.${}_{\\!}$ More recently,${}_{\\!}$ the breakthroughs in NLP ushered
in a revolution in feature extraction, introducing${}_{\\!}$ attention-
based${}_{\\!}$ Transformer${}_{\\!}$ architectures${}_{\\!}$ [12].${}_{\\!}$
Vision${}_{\\!}$ Transformer${}_{\\!}$ (ViT)${}_{\\!}$ is${}_{\\!}$
a${}_{\\!}$ time-honored${}_{\\!}$ work${}_{\\!}$ [13],${}_{\\!}$
which${}_{\\!}$ adopts${}_{\\!}$ self-attention${}_{\\!}$ mechanisms${}_{\\!}$
to${}_{\\!}$ image${}_{\\!}$ classification,${}_{\\!}$ achieving${}_{\\!}$
remarkable results.${}_{\\!}$ Note${}_{\\!}$ that${}_{\\!}$ ViTs’${}_{\\!}$
effectiveness${}_{\\!}$ often${}_{\\!}$ hinges${}_{\\!}$ on large-scale
training datasets [61, 62]. Recent advancements have endeavored to bridge the
gap between convolution and attention through hybrid models like CoAtNet [63]
and Mobile-Former [64], which seamlessly combine both design paradigms.
Besides, models employing multi-layer perceptrons (MLPs) [14, 15, 65, 66, 67,
68] for spatial interactions and techniques such as shifting [69, 70] or
pooling [71] for local context ${}_{\\!}$ have${}_{\\!}$ emerged${}_{\\!}$
to${}_{\\!}$ explore${}_{\\!}$ the${}_{\\!}$ power${}_{\\!}$ of${}_{\\!}$
MLPs.

Though impressive, existing feature extractors are built upon rectangular
receptive fields (as in CNN and variants, which represent an image as a grid
of regular regions), or queries (as in ViTs, which incorporate an additional
query token to represent an image), or rectangular image patches (as in MLPs,
which use mixer layers to perform communication between patches). As far as we
know, CoC [26] is the only backbone using a clustering algorithm. Compared to
CoC, we want to highlight that our essential paradigm (_i.e_., representing an
image as representatives _vs_ rectangular regions), goal (_i.e_.,
_transparent_ and _ad-hoc interpretable_ backbone _vs_ clustering based method
for information exchange), and core techniques (_i.e_., modeling _gradually
growing_ representatives _vs_ modeling context clusters at specific
resolution) are different. FEC takes the _first_ step towards _fully_
clustering based feature extraction, by reformulating its workflow as
selecting representatives.

Neural Network Interpretability. The opaque nature of deep neural networks
(DNNs) has posed challenges to their adoption in decision-critical
applications, sparking a surge of interest in enhancing their transparency and
interpretabi-lity. One key distinction in interpretable methods lies in
whether they produce posterior explanations for pre-trained DNNs${}_{\\!}$
or${}_{\\!}$ aim${}_{\\!}$ to${}_{\\!}$ develop${}_{\\!}$ inherently${}_{\\!}$
interpretable${}_{\\!}$ DNNs${}_{\\!}$ from${}_{\\!}$ the${}_{\\!}$
outset.${}_{\\!}$ Posterior${}_{\\!}$ explanations,${}_{\\!}$ though${}_{\\!}$
prevalent,${}_{\\!}$ face${}_{\\!}$ cri- ticism for their nature of
approximation and limited capaci- ty to truly elucidate the inner workings of
DNNs [72, 73, 74, 29, 75]. Representative works involve reverse-engineer
importance values [76, 77, 78, 28, 79, 80, 81, 82, 83] and${}_{\\!}$
sensitivities${}_{\\!}$ of inputs [84, 85, 86, 87]. Recent advances in self-
supervised learning, such as DINO [88, 89], have revealed${}_{\\!}$
emergent${}_{\\!}$ segmentation${}_{\\!}$ properties${}_{\\!}$ in${}_{\\!}$
ViTs.${}_{\\!}$ On${}_{\\!}$ the${}_{\\!}$ other${}_{\\!}$ hand,${}_{\\!}$
methods${}_{\\!}$ focused${}_{\\!}$ on${}_{\\!}$ ad-hoc${}_{\\!}$
explainability${}_{\\!}$ strive${}_{\\!}$ to${}_{\\!}$ incorporate${}_{\\!}$
more${}_{\\!}$ interpretable elements into black-box DNNs [90, 91, 92] or
impose specific properties on the model’s representations [93, 94,
95]${}_{\\!}$ to${}_{\\!}$ bolster${}_{\\!}$ interpretability.${}_{\\!}$
A${}_{\\!}$ notable${}_{\\!}$ instance${}_{\\!}$ is the deep nearest centroids
[96], which introduces a nonparametric classifier and uses case-based
reasoning, thereby presenting a novel and explainable paradigm for visual
recognition.

In${}_{\\!}$ a${}_{\\!}$ related${}_{\\!}$ vein,${}_{\\!}$ CRATE${}_{\\!}$
[97]${}_{\\!}$ adopts${}_{\\!}$ the${}_{\\!}$ white-box Transformer${}_{\\!}$
[98] to${}_{\\!}$ delve${}_{\\!}$ into${}_{\\!}$ visual${}_{\\!}$
attention${}_{\\!}$ and${}_{\\!}$ the${}_{\\!}$ emer- gence of segmentation on
the supervised classification task, showcasing${}_{\\!}$ considerable
results.${}_{\\!}$ Nonetheless,${}_{\\!}$ we argue that CRATE${}_{\\!}$
falls${}_{\\!}$ within${}_{\\!}$ the${}_{\\!}$ realm${}_{\\!}$ of${}_{\\!}$
posterior${}_{\\!}$ explanations,${}_{\\!}$ whereas our FEC possesses an
inherent capacity to elucidate the reasons behind its feature extraction
process. Such innate transparency sets FEC apart from counterparts that solely
offer _post-hoc_ explainability. This work represents a small yet solid stride
towards empowering the forward process of feature extraction with _ad-hoc_
interpretability through an integrated clustering-based backbone and offers a
fresh perspective on deep visual representations.

##  5 Experiment

FEC is proposed as the first framework to support feature extraction in a
fully clustering-based manner. We evaluate the proposed backbone for four
recognition tasks viz. image classification, object detection, semantic
segmentation, and instance segmentation on three prevalent benchmarks (_i.e_.,
ImageNet-1K [24], MS COCO [99], and ADE20k [100]).

FEC combines _ad-hoc_ interpretability with promising performance${}_{\\!}$
across${}_{\\!}$ diverse${}_{\\!}$ recognition${}_{\\!}$ tasks${}_{\\!}$
and${}_{\\!}$ datasets.${}_{\\!}$ Note that our objective is not to chase the
state-of-the-art${}_{\\!}$ performance${}_{\\!}$ like${}_{\\!}$ ConvNeXt
[55],${}_{\\!}$ but${}_{\\!}$ rather${}_{\\!}$ to${}_{\\!}$ assess${}_{\\!}$
FEC’s${}_{\\!}$ capabilities${}_{\\!}$ (_e.g_.,${}_{\\!}$
effectiveness,${}_{\\!}$ efficiency,${}_{\\!}$ transferability,${}_{\\!}$
_etc_.) through a comprehensive series of experiments.

###  5.1 Experiments on Image Classification

Dataset.${}_{\\!}$ ImageNet-1K [24] is a well-benchmarked image dataset.
Following conventional procedures, it is divided into 1.28M/50K/100K images
for train/val/test.

Training. We use timm [101] as our codebase and follow the standard training
protocols as detailed in [63, 71, 26]. We use an AdamW [102] optimizer using a
cosine decay learning rate scheduler and 5 epochs of warm-up. The momentum and
weight decay are set to 0.9 and 0.05, respectively. A batch size of 1024 and
an initial learning rate of 0.001 are used. More details are left in the
appendix.

Test. Following [26], we use one input image scale of 224$\times$224 with
center cropping without any data augmentation. By default, the models are
trained on 4 NVIDIA Tesla A100 GPUs with 80GB memory per card. Testing is
conducted on the same machine.

Metric. We report the parameters used, FLOPs, and Top-1 classification
accuracy on a single crop following previous works [11, 103]. Throughput
(image/s), or FPS, is measured using the same script as in [103, 104] on a
single V100 GPU using a batch size of 256.

Table 1: Quantitative results on ImageNet-1K [24] val for ima-ge
classification (§5.1). All models are trained and tested at 224$\times$224
resolution, except ViT-B [13] and ViT-L [13].

| Method |  #Param (M) |  FLOPs (G) |  Top-1 (%)$\uparrow$ |  FPS (img/s)$\uparrow$
---|---|---|---|---|---
Clu. |  CoC-Tiny${}_{\\!\\!}$ [26] | 5.3 | 1.1 | 71.8 | 1146.7
CoC-Small${}_{\\!\\!}$ [26] | 14.0 | 2.8 | 77.5 | 852.1
CoC-Medium [26] | 27.9 | 5.9 | 81.0 | 345.7
MLP |  ResMLP-12${}_{\\!}$ [15] | 15.0 | 3.0 | 76.6 | 1499.0
ResMLP-24${}_{\\!}$ [15] | 30.0 | 6.0 | 79.4 | 741.6
ResMLP-36${}_{\\!}$ [15] | 45.0 | 8.9 | 79.7 | 484.6
MLP-Mixer-B/16 [14] | 59.0 | 12.7 | 76.4 | 387.7
MLP-Mixer-L/16 [14] | 207.0 | 44.8 | 71.8 | 114.2
gMLP-Ti [105] | 6.0 | 1.4 | 72.3 | 1440.0
gMLP-S [105] | 20.0 | 4.5 | 79.6 | 650.5
Attention |  ViT-B/16 [13] | 86.0 | 55.5 | 77.9 | 86.4
ViT-L/16 [13] | 307 | 190.7 | 76.5 | 26.6
PVT-Tiny [106] | 13.2 | 1.9 | 75.1 | -
PVT-Small [106] | 24.5 | 3.8 | 79.8 | -
Swin-Tiny [103] | 29 | 4.5 | 81.3 | 631.1
Swin-Small [103] | 50 | 8.7 | 83.0 | 374.9
Conv. |  ResNet18 [11] | 12 | 1.8 | 69.8 | 4284.9
ResNet50 [11] | 26 | 4.1 | 79.8 | 1206.0
ConvMixer${}_{512/16\\!}$ [107] | 5.4 | - | 73.8 | -
ConvMixer1024/12[107] | 14.6 | - | 77.8 | -
ConvMixer${}_{768/32\\!}$ [107] | 21.1 | - | 80.2 | 139.7
Clu. | FEC-Small (Ours) | 5.5 | 1.4 |  72.7±0.06 | 1042.9
FEC-Base ${}_{\\!}$(Ours) | 14.4 | 3.4 |  78.1±0.00 | 754.1
FEC-Large (Ours) | 28.3 | 6.5 |  81.2±0.06 | 342.1

Performance${}_{\\!}$ Comparison.${}_{\\!}$ Table${}_{\\!}$ 1${}_{\\!}$
presents${}_{\\!}$ our${}_{\\!}$ classifi- cation${}_{\\!}$ results${}_{\\!}$
on${}_{\\!}$ ImageNet${}_{\\!}$ [24]${}_{\\!}$ val.${}_{\\!}$ As illustrated,
FEC achieves competitive performance and efficiency, outperforming ResNet18
[11] by 2.9% using less than half the number of parameters that ResNet18 has.
ResNet achieves high FPS due to optimizations for convolutional operators.

Figure 3: Inspection of the modeled representatives (§5.2) on ImageNet-1K [24]
val. Different colored masks indicate different clusters. As the number of
clusters decreases, each cluster tends to represent an entire object or a
portion of an object, suggesting that FEC effectively captures the underlying
data distribution of visual scenes.

Figure 4: ${}_{\\!}$Quantitative${}_{\\!}$ results${}_{\\!}$ on${}_{\\!}$
ADE20K [100]${}_{\\!}$ val${}_{\\!}$ for${}_{\\!}$ semantic segmentation
(§5.3). Semantic${}_{\\!}$ FPN [108] is adopted.

Backbone |  #Param (M) |  ADE20K mIoU(%)$\uparrow$
---|---|---
ResNet18 [11] | 15.5M | 32.9
ResNet50 [11] | 28.5M | 36.7
PVT-Tiny [106] | 17.0M | 35.7
PVT-Small [106] | 28.2M | 39.8
CoC-Small/4 [26] | 17.6M | 36.6
CoC-Small/25 [26] | 17.6M | 36.4
CoC-Small/49 [26] | 17.6M | 36.3
CoC-Medium/4 [26] | 31.5M | 40.2
CoC-Medium/25 [26] | 31.5M | 40.6
CoC-Medium/49 [26] | 31.5M | 40.8
FEC-Small | 9.1M |  35.3±0.15
FEC-Base | 18.0M |  37.7±0.15
FEC-Large | 31.9M |  40.5±0.10

Figure 5: Quantitative results on COCO [99] val2017 for object detection and
semantic segmentation (§5.4). We use Mask RCNN [109] to evaluate the
performance of the proposed backbone on two tasks.

Backbone |  #Param (M) |  AP${}^{\mathrm{box}}\uparrow$ |  COCO AP${}^{\mathrm{box}}_{50}\uparrow$ |  AP${}^{\mathrm{box}}_{75}\uparrow$ |  AP${}^{\mathrm{mask}}\uparrow$ |  COCO AP${}^{\mathrm{mask}}_{50}\uparrow$ |  AP${}^{\mathrm{mask}}_{75}\uparrow$
---|---|---|---|---|---|---|---
ResNet18 [11] | 31.2M | 34.0 | 54.0 | 36.7 | 31.2 | 51.0 | 32.7
ResNet50 [11] | 44.2M | 38.0 | 58.6 | 41.4 | 34.4 | 55.1 | 36.7
PVT-Tiny [106] | 32.9M | 36.7 | 59.2 | 39.3 | 35.1 | 56.7 | 37.3
PVT-Small [106] | 44.1M | 40.4 | 62.9 | 43.8 | 37.8 | 60.1 | 40.3
CoC-Small/4 [26] | 32.7M | 35.9 | 58.3 | 38.3 | 33.8 | 55.3 | 35.8
CoC-Small/25 [26] | 32.7M | 37.5 | 60.1 | 40.0 | 35.4 | 57.1 | 37.9
CoC-Small/49 [26] | 32.7M | 37.2 | 59.8 | 39.7 | 34.9 | 56.7 | 37.0
CoC-Medium/4 [26] | 46.7M | 38.6 | 61.1 | 41.5 | 36.1 | 58.2 | 38.0
CoC-Medium/25 [26] | 46.7M | 40.1 | 62.8 | 43.6 | 37.4 | 59.9 | 40.0
CoC-Medium/49 [26] | 46.7M | 40.6 | 63.3 | 43.9 | 37.6 | 60.1 | 39.9
FEC-Small | 24.3M |  35.6±0.06 |  57.5±0.10 |  38.2±0.15 |  33.6±0.10 |  54.7±0.10 |  35.7±0.15
FEC-Base | 33.1M |  37.9±0.06 |  60.1±0.06 |  40.8±0.15 |  35.5±0.06 |  57.2±0.06 |  37.8±0.10
FEC-Large | 47.1M |  39.9±0.10 |  62.5±0.06 |  43.2±0.12 |  37.3±0.06 |  59.5±0.15 |  39.5±0.12

###  5.2 Study of Ad-hoc Interpretability

We have empirically demonstrated FEC’s effectiveness and efficiency in image
classification. By redefining visual feature extraction as a clustering
process, where representatives are iteratively selected during the forward
process, FEC departs from the traditional grid-style paradigms. This shift
suggests that FEC offers exceptional _ad-hoc_ interpretability, a quality we
will now explore further.

We first detail the procedure to aggregate cluster assignments across layers.
Given an image (224$\times$224), a standard convolution-based pooling is used
to generate a low-dimensional feature map (56$\times$56) where each pixel
represents a 4$\times$4 region in the raw image. As for the first clustering-
based pooling layer, those pixel blocks (56$\times$56, each with 4$\times$4
pixels) will be assigned to one of a total of 28$\times$28 clusters
$\\{\bm{S}_{i}^{1},\\!1\\!\leq\\!i\\!\leq\\!28\\!\times\\!28\\}$. Afterward,
the subsequent pooling layers perform the same clustering procedure based on
previous clusters. In this way, the clusters at different levels can be
aggregated across layers according to Eq. 8, leading to 7$\times$7 segments
$\\{\bar{\bm{S}}_{i}^{4},\\!1\\!\leq\\!i\\!\leq\\!7\\!\times\\!7\\}$ in the
last step. While humans can view and examine the final cluster assignments, 49
clusters are simply too many to comprehend for a 224$\times$224 image.
Therefore, we use K-Means to further reduce the number of clusters.
Concretely, we use the deep features ($\bm{R}$ in Eq. 6) of those 49 clusters
as their representations. Default hyperparameters in the scikit-learn [110]
implementation are adopted. Medium filtering is adopted for better
visualization.

After the above introduction, we visualize cluster assignments in Fig. 3 to
clarify FEC’s principles. Two remarkable observations justify the _ad-hoc_
interpretability and effectiveness of FEC: i) Across successive pooling
layers, the clustering-based feature extraction progressively abstracts pixel
blocks, mirroring human cognitive processes and providing a natural
representation of visual data. ii) In final cluster assignments, we find
consistent semantic representations, with clusters often corresponding to
coherent objects or object parts. This aligns FEC with human perception and
enhances feature interpretability, linking clusters to identifiable image
elements.

Table 2: A set of ablative experiments (§5.1) on ImageNet [24] val, ADE20K
[100] val, and COCO [99] val2017. The adopted hyperparameters are marked in
red.

similarity measurement |  #Param. (M) |  Top-1 (%)$\uparrow$ |  Top-5 (%)$\uparrow$
---|---|---|---
Euclidean | 5.46 | 72.2 | 90.9
Dot Product | 5.46 | 72.7 | 91.1
Cosine | 5.46 | 72.7 | 91.2

(a) Similarity measurement (Eq. 5)

feature dispatching |  #Param. (M) |  Top-1 (%)$\uparrow$ |  Top-5 (%)$\uparrow$
---|---|---|---
w/o $\alpha$ and $\beta$ | 5.46 | 72.1 | 90.8
w/o $\alpha$ | 5.46 | 72.4 | 91.0
w/o $\beta$ | 5.46 | 72.2 | 90.9
Ours (Eq. 7) | 5.46 | 72.7 | 91.2

(b) Parameters used in feature dispatching (§3)

feature | #Param | ADE20K
---|---|---
dimension | (M) | mIoU(%)$\uparrow$
(48, 48, 96, 96) | 8.6M | 34.9
(72, 72, 144, 144) | 8.9M | 35.2
(96, 96, 192, 192) | 9.1M | 35.3
(120, 120, 240, 240) | 9.4M | 35.5

(c) Number of channels in encode (Eq. 4)

feature | #Param | COCO | COCO
---|---|---|---
dimension | (M) | APbox$\uparrow$ | AP${}^{\mathrm{box}}_{50}$$\uparrow$ | AP${}^{\mathrm{box}}_{75}$$\uparrow$ | AP${}^{\mathrm{box}}_{s}\uparrow$ | AP${}^{\mathrm{box}}_{m}\uparrow$ | AP${}^{\mathrm{box}}_{l}\uparrow$ | APmask$\uparrow$ | AP${}^{\mathrm{mask}}_{50}$$\uparrow$ | AP${}^{\mathrm{mask}}_{75}$$\uparrow$
(48, 48, 96, 96) | 23.8M | 34.9 | 57.2 | 36.9 | 20.3 | 36.9 | 45.8 | 33.4 | 54.5 | 35.0
(72, 72, 144, 144) | 24.0M | 35.2 | 57.2 | 37.7 | 20.1 | 37.4 | 46.7 | 33.5 | 54.5 | 35.6
(96, 96, 192, 192) | 24.3M | 35.6 | 57.5 | 38.2 | 20.9 | 37.7 | 46.8 | 33.6 | 54.7 | 35.7
(120, 120, 240, 240) | 24.5M | 35.5 | 57.5 | 37.8 | 20.7 | 37.6 | 47.4 | 33.8 | 54.7 | 35.9

(d) Number of channels in encode (Eq. 4)

###  5.3 Experiments on Semantic Segmentation

Dataset.${}_{\\!}$ ADE20K${}_{\\!}$ [100] is a prominent semantic segmentation
dataset${}_{\\!}$ renowned${}_{\\!}$ for${}_{\\!}$ its${}_{\\!}$
extensive${}_{\\!}$ collection${}_{\\!}$ of${}_{\\!}$ images. It${}_{\\!}$
encompasses${}_{\\!}$ 150${}_{\\!}$ object${}_{\\!}$ categories${}_{\\!}$
with${}_{\\!}$ a${}_{\\!}$ total${}_{\\!}$ of 25,000 images (20K/2K/3K for
train/val/test).

Training.${}_{\\!}$ We${}_{\\!}$ use${}_{\\!}$ mmsegmentation${}_{\\!}$
[111]${}_{\\!}$ as${}_{\\!}$ our${}_{\\!}$ codebase${}_{\\!}$ and follow the
standard training protocols as detailed in [106, 26]. We evaluate FEC on a
classic segmentation method, _i.e_., Semantic FPN [108], for its high
efficiency. We utilize an AdamW [102] optimizer for 80K iterations using the
polynomial decay learning rate scheduler with a power of 0.9. We set the batch
size as 16 and the initial learning rate as 0.0001. More details are left in
the appendix.

Test. We use one input image scale with a shorter side of 512 during inference
without applying data augmentation.

Metric. Mean intersection-over-union (mIoU) is reported.

Performance${}_{\\!}$ Comparison.${}_{\\!}$ From table 5 we can observe that
our approach yields remarkable performance on dense prediction tasks. For
example, our FEC-Base outperforms ResNet18, ResNet50, and PVT-Tiny by 4.8%,
1.0%, and 2.0% in terms of mIoU, respectively. In addition, FEC-Small can even
outperform ResNet18, _i.e_., 35.3% vs 32.9% in terms of mIoU. Such performance
gains remain consistent as in the classification task and are significant
considering the number of used parameters.

###  5.4 Experiments on Object Detection and Instance Segmentation

Dataset.${}_{\\!}$ The${}_{\\!}$ MS${}_{\\!}$ COCO${}_{\\!}$ 2017${}_{\\!}$
benchmark[99]${}_{\\!}$ features${}_{\\!}$ a di- verse${}_{\\!}$
collection${}_{\\!}$ of${}_{\\!}$ over${}_{\\!}$ 200K${}_{\\!}$ high-quality
images,${}_{\\!}$ annotated across 80 common object categories in daily
contexts.${}_{\\!}$ It${}_{\\!}$ is${}_{\\!}$ divided${}_{\\!}$ into${}_{\\!}$
118K/5K/41K images${}_{\\!}$ for${}_{\\!}$ train/val/test.

Training. We use mmdetection [112] as our codebase and follow the training
protocols as in [26]. We verify the effectiveness of FEC backbones on top of a
milestone model, _i.e_., Mask R-CNN [109]. AdamW [102] optimizer is used for
12 epochs (1$\times$ scheduler) and initialize the backbone with ImageNet [24]
pre-trained weights. A batch size of 16 and an initial learning rate of 0.0002
are used. More details are left in the appendix.

Test. We use one input image scale with a shorter side of 800 during inference
without applying data augmentation.

Metric. We report average precision (AP), AP50, AP75 for both object detection
and instance segmentation.

Performance${}_{\\!}$ Comparison.${}_{\\!}$ Table 5${}_{\\!}$
confirms${}_{\\!}$ again${}_{\\!}$ the${}_{\\!}$ tran-sferability and
versatility of FEC for the common instance-centric recognition tasks. On top
of a relatively conservative baseline, _i.e_., Mask-RCNN [109], our algorithm
outperforms both types of rivals. For instance, the performance of FEC-Tiny is
clear ahead compared to ResNet-18 [11] (_i.e_., 35.6% APbox vs 34.0% APbox and
33.6% APmask vs 31.2% APmask), and FEC-Base achieves promising gains of 1.2%
APbox and 0.4% APmask against PVT-Tiny [106].

###  5.5 Diagnostic Experiment

This${}_{\\!}$ section${}_{\\!}$ ablates${}_{\\!}$ FEC’s${}_{\\!}$
key${}_{\\!}$ components${}_{\\!}$ on${}_{\\!}$ ImageNet${}_{\\!}$
[24]${}_{\\!}$ val, ADE20K [100] val, and COCO [99] val2017. All experiments
use the FEC-Small model.

Similarity${}_{\\!}$ Measurement.${}_{\\!}$ We${}_{\\!}$ first${}_{\\!}$
examine${}_{\\!}$ the${}_{\\!}$ similarity measurement in the clustering
process (Eq. 5) by contras- ting it with several standard similarity
(distance) functions, _i.e_., Euclidean distance and dot product (unnormalized
cosine similarity). As shown in Table 2(a), dot product and cosine similarity
shows better performance than Euclidean distance, _e.g_., 72.7% vs 72.2% in
terms of Top-1 accuracy. Cosine similarity is adopted by default.

Feature${}_{\\!}$ Dispatching.${}_{\\!}$ We${}_{\\!}$ then${}_{\\!}$
investigate${}_{\\!}$ the${}_{\\!}$ effects${}_{\\!}$ of${}_{\\!}$
the${}_{\\!}$ learnable parameters for similarity scaling and shifting (Eq.
7). In table 2(b), the first three lines indicate the removal of the
corresponding hyperparameters. We find that introducing these two factors can
bring minor performance improvements, _e.g_., 72.7% vs 72.1% in terms of Top-1
accuracy and 91.2% vs 90.8% in terms of Top-5 accuracy.

Feature Dimension. Last, we ablate the feature dimension $C^{\prime}$ in the
encoding layer (Eq. 4). In the pooling layer, $C^{\prime}$ is used to increase
the number of channels for the next stage. In the encoding layer, $C^{\prime}$
is used to control the complexity of the projected key and value space (the
output channels can be further adjusted by the MLP during feature
dispatching). The results are summarized in Table 2(c) and Table 2(d). The
four values of each row correspond to the four encoding phases. In a nutshell,
the performance for downstream tasks improves as the feature dimension
increases, _e.g_., 35.5% vs 34.9% in terms of mIoU, 35.6% vs 34.9% in terms of
APbox, and 33.8% vs 33.4% in terms of APmask, at the expense of parameter
growth. For a fair comparison with previous work [26], the settings in the 3rd
row are adopted.

##  6 Conclusion and Discussion

In machine vision, extracting powerful distributed represen- tations for
visual data while preserving the interpretability and explicit modeling of
data distribution, presents a perennial challenge. While the community has
witnessed great strides in visual backbones, top-leading solutions remain
bound to the computational confines of processing rectangular image patches —
a stark contrast to the pixel organization observed in human perception. This
study represents a significant leap forward by reformulating feature
extraction as representative selection, resulting in a transparent and
interpretable feature extractor. Our goal is to pave the way for vision
systems that not only excel in performance but also possess an intrinsic
understanding of the underlying data distribution of visual scenes, thereby
enhancing both trust and clarity in their application.

## References

  * Snyder and Qi [2004] Wesley E Snyder and Hairong Qi.  _Machine vision_ , volume 1.  Cambridge University Press, 2004.
  * Bishop [2006] Christopher M Bishop.  Pattern recognition.  _Machine learning_ , 128(9), 2006.
  * LeCun et al. [2015] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  _nature_ , 521(7553):436–444, 2015.
  * Ng and Henikoff [2003] Pauline C Ng and Steven Henikoff.  Sift: Predicting amino acid changes that affect protein function.  _Nucleic acids research_ , 31(13):3812–3814, 2003.
  * Dalal and Triggs [2005] Navneet Dalal and Bill Triggs.  Histograms of oriented gradients for human detection.  In _CVPR_ , pages 886–893, 2005.
  * Bay et al. [2006] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool.  Surf: Speeded up robust features.  In _ECCV_ , pages 404–417, 2006.
  * Calonder et al. [2010] Michael Calonder, Vincent Lepetit, Christoph Strecha, and Pascal Fua.  Brief: Binary robust independent elementary features.  In _ECCV_ , pages 778–792, 2010.
  * Rublee et al. [2011] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski.  Orb: An efficient alternative to sift or surf.  In _ICCV_ , pages 2564–2571, 2011.
  * Leutenegger et al. [2011] Stefan Leutenegger, Margarita Chli, and Roland Y Siegwart.  Brisk: Binary robust invariant scalable keypoints.  In _ICCV_ , pages 2548–2555, 2011.
  * Simonyan and Zisserman [2015] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In _ICLR_ , 2015.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _NeurIPS_ , 2017.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _ICLR_ , 2020.
  * Tolstikhin et al. [2021] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al.  Mlp-mixer: An all-mlp architecture for vision.  In _NeurIPS_ , pages 24261–24272, 2021.
  * Touvron et al. [2022] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al.  Resmlp: Feedforward networks for image classification with data-efficient training.  _IEEE TPAMI_ , 45(4):5314–5321, 2022.
  * Gonzales and Wintz [1987] Rafael C Gonzales and Paul Wintz.  _Digital image processing_.  Addison-Wesley Longman Publishing Co., Inc., 1987.
  * Castleman [1996] Kenneth R Castleman.  _Digital image processing_.  Prentice Hall Press, 1996.
  * Beymer and Poggio [1996] David Beymer and Tomaso Poggio.  Image representations for visual learning.  _Science_ , 272(5270):1905–1909, 1996.
  * Guidotti et al. [2018] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi.  A survey of methods for explaining black box models.  _ACM computing surveys (CSUR)_ , 51(5):1–42, 2018.
  * Weinlich et al. [2016] Andreas Weinlich, Peter Amon, Andreas Hutter, and Andre Kaup.  Probability distribution estimation for autoregressive pixel-predictive image coding.  _IEEE TIP_ , 25(3):1382–1395, 2016.
  * Biederman [1987] Irving Biederman.  Recognition-by-components: a theory of human image understanding.  _Psychological review_ , 94(2):115, 1987.
  * Beutter and Stone [2000] Brent R Beutter and Leland S Stone.  Motion coherence affects human perception and pursuit similarly.  _Visual neuroscience_ , 17(1):139–153, 2000.
  * Bill et al. [2020] Johannes Bill, Hrag Pailian, Samuel J Gershman, and Jan Drugowitsch.  Hierarchical structure is employed by humans during visual motion perception.  _Proceedings of the National Academy of Sciences_ , 117(39):24581–24589, 2020.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li.  Imagenet large scale visual recognition challenge.  _IJCV_ , 115(3):211–252, 2015.
  * Rommers et al. [2013] Joost Rommers, Ton Dijkstra, and Marcel Bastiaansen.  Context-dependent semantic processing in the human brain: Evidence from idiom comprehension.  _Journal of Cognitive Neuroscience_ , 25(5):762–776, 2013.
  * Ma et al. [2023] Xu Ma, Yuqian Zhou, Huan Wang, Can Qin, Bin Sun, Chang Liu, and Yun Fu.  Image as set of points.  In _ICLR_ , 2023.
  * Liang et al. [2023a] James Liang, Yiming Cui, Qifan Wang, Tong Geng, Wenguan Wang, and Dongfang Liu.  Clusterfomer: Clustering as a universal visual learner.  In _NeurIPS_ , 2023a.
  * Selvaraju et al. [2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _ICCV_ , 2017.
  * Rudin et al. [2022] Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong.  Interpretable machine learning: Fundamental principles and 10 grand challenges.  _Statistics Surveys_ , 16:1–85, 2022.
  * Redmon et al. [2016] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In _CVPR_ , pages 779–788, 2016.
  * Ren et al. [2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In _NeurIPS_ , 2015.
  * Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _ECCV_ , 2020.
  * Li et al. [2022a] Lin Li, Long Chen, Hanrong Shi, Hanwang Zhang, Yi Yang, Wei Liu, and Jun Xiao.  Nicest: Noisy label correction and training for robust scene graph generation.  _arXiv preprint arXiv:2207.13316_ , 2022a.
  * Li et al. [2022b] Lin Li, Long Chen, Yifeng Huang, Zhimeng Zhang, Songyang Zhang, and Jun Xiao.  The devil is in the labels: Noisy label correction for robust scene graph generation.  In _CVPR_ , pages 18869–18878, 2022b.
  * Chen et al. [2023] Guikun Chen, Lin Li, Yawei Luo, and Jun Xiao.  Addressing predicate overlap in scene graph generation with semantic granularity controller.  In _ICME_ , pages 78–83, 2023.
  * Li et al. [2023a] Lin Li, Guikun Chen, Jun Xiao, Yi Yang, Chunping Wang, and Long Chen.  Compositional feature augmentation for unbiased scene graph generation.  In _ICCV_ , pages 21685–21695, 2023a.
  * Yin et al. [2022] Junbo Yin, Dingfu Zhou, Liangjun Zhang, Jin Fang, Cheng-Zhong Xu, Jianbing Shen, and Wenguan Wang.  Proposalcontrast: Unsupervised pre-training for lidar-based 3d object detection.  In _ECCV_ , pages 17–33, 2022.
  * Feng et al. [2023] Tuo Feng, Wenguan Wang, Xiaohan Wang, Yi Yang, and Qinghua Zheng.  Clustering based point cloud representation learning for 3d analysis.  In _ICCV_ , pages 8283–8294, 2023.
  * Feng et al. [2024] Tuo Feng, Ruijie Quan, Xiaohan Wang, Wenguan Wang, and Yi Yang.  Interpretable3d: An ad-hoc interpretable classifier for 3d point clouds.  In _AAAI_ , 2024.
  * Wang et al. [2018] Wenguan Wang, Jianbing Shen, Fatih Porikli, and Ruigang Yang.  Semi-supervised video object segmentation with super-trajectories.  _IEEE TPAMI_ , 41(4):985–998, 2018.
  * Liang et al. [2022] Chen Liang, Wenguan Wang, Jiaxu Miao, and Yi Yang.  Gmmseg: Gaussian mixture based generative semantic segmentation models.  In _NeurIPS_ , pages 31360–31375, 2022.
  * Zhou et al. [2022] Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Luc Van Gool.  Rethinking semantic segmentation: A prototype view.  In _CVPR_ , 2022.
  * Liang et al. [2023b] James Chenhao Liang, Tianfei Zhou, Dongfang Liu, and Wenguan Wang.  Clustseg: Clustering for universal segmentation.  In _ICML_ , pages 20787–20809, 2023b.
  * Li et al. [2023b] Liulei Li, Wenguan Wang, Tianfei Zhou, Jianwu Li, and Yi Yang.  Unified mask embedding and correspondence learning for self-supervised video segmentation.  In _CVPR_ , pages 18706–18716, 2023b.
  * Grabski et al. [2023] Isabella N Grabski, Kelly Street, and Rafael A Irizarry.  Significance analysis for clustering with single-cell rna-sequencing data.  _Nature Methods_ , 20(8):1196–1202, 2023.
  * Barrio-Hernandez et al. [2023] Inigo Barrio-Hernandez, Jingi Yeo, Jürgen Jänes, Milot Mirdita, Cameron LM Gilchrist, Tanita Wein, Mihaly Varadi, Sameer Velankar, Pedro Beltrao, and Martin Steinegger.  Clustering predicted structures at the scale of the known protein universe.  _Nature_ , 622(7983):637–645, 2023.
  * Wayment-Steele et al. [2024] Hannah K Wayment-Steele, Adedolapo Ojoawo, Renee Otten, Julia M Apitz, Warintra Pitsawong, Marc Hömberger, Sergey Ovchinnikov, Lucy Colwell, and Dorothee Kern.  Predicting multiple conformations via sequence clustering and alphafold2.  _Nature_ , 625(7996):832–839, 2024.
  * Quan et al. [2024] Ruijie Quan, Wenguan Wang, Fan Ma, Hehe Fan, and Yi Yang.  Clustering for protein representation learning.  In _CVPR_ , 2024.
  * Ren and Malik [2003] Ren and Malik.  Learning a classification model for segmentation.  In _ICCV_ , pages 10–17, 2003.
  * Jaegle et al. [2021] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira.  Perceiver: General perception with iterative attention.  In _ICML_ , pages 4651–4664, 2021.
  * Carreira et al. [2022] Joao Carreira, Skanda Koppula, Daniel Zoran, Adria Recasens, Catalin Ionescu, Olivier Henaff, Evan Shelhamer, Relja Arandjelovic, Matt Botvinick, Oriol Vinyals, et al.  Hip: Hierarchical perceiver.  _arXiv preprint arXiv:2202.10890_ , 2022.
  * Gu et al. [2009] Chunhui Gu, Joseph J Lim, Pablo Arbeláez, and Jitendra Malik.  Recognition using regions.  In _CVPR_ , pages 1030–1037, 2009.
  * Carreira et al. [2012] Joao Carreira, Rui Caseiro, Jorge Batista, and Cristian Sminchisescu.  Semantic segmentation with second-order pooling.  In _ECCV_ , pages 430–443, 2012.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In _NeurIPS_ , 2012.
  * Liu et al. [2022] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  In _CVPR_ , pages 11976–11986, 2022.
  * Szegedy et al. [2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In _CVPR_ , pages 1–9, 2015.
  * Huang et al. [2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In _CVPR_ , pages 4700–4708, 2017.
  * Tan and Le [2019] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In _ICML_ , pages 6105–6114, 2019.
  * Howard et al. [2017] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  In _CVPR_ , 2017.
  * Dai et al. [2017] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In _CVPR_ , 2017.
  * Khan et al. [2022] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah.  Transformers in vision: A survey.  _ACM computing surveys (CSUR)_ , 54(10s):1–41, 2022.
  * Gani et al. [2022] Hanan Gani, Muzammal Naseer, and Mohammad Yaqub.  How to train vision transformer on small-scale datasets?  In _BMVC_ , 2022.
  * Dai et al. [2021] Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan.  Coatnet: Marrying convolution and attention for all data sizes.  In _NeurIPS_ , pages 3965–3977, 2021.
  * Chen et al. [2022a] Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu.  Mobile-former: Bridging mobilenet and transformer.  In _CVPR_ , pages 5270–5279, 2022a.
  * Guo et al. [2022] Jianyuan Guo, Yehui Tang, Kai Han, Xinghao Chen, Han Wu, Chao Xu, Chang Xu, and Yunhe Wang.  Hire-mlp: Vision mlp via hierarchical rearrangement.  In _CVPR_ , pages 826–836, 2022.
  * Hou et al. [2022] Qibin Hou, Zihang Jiang, Li Yuan, Ming-Ming Cheng, Shuicheng Yan, and Jiashi Feng.  Vision permutator: A permutable mlp-like architecture for visual recognition.  _IEEE TPAMI_ , 45(1):1328–1334, 2022.
  * Tang et al. [2022] Yehui Tang, Kai Han, Jianyuan Guo, Chang Xu, Yanxi Li, Chao Xu, and Yunhe Wang.  An image patch is a wave: Phase-aware vision mlp.  In _CVPR_ , pages 10935–10944, 2022.
  * Chen et al. [2022b] Shoufa Chen, Enze Xie, Chongjian Ge, Runjian Chen, Ding Liang, and Ping Luo.  Cyclemlp: A mlp-like architecture for dense prediction.  In _ICLR_ , 2022b.
  * Huang et al. [2021] Zilong Huang, Youcheng Ben, Guozhong Luo, Pei Cheng, Gang Yu, and Bin Fu.  Shuffle transformer: Rethinking spatial shuffle for vision transformer.  _arXiv preprint arXiv:2106.03650_ , 2021.
  * Lian et al. [2021] Dongze Lian, Zehao Yu, Xing Sun, and Shenghua Gao.  As-mlp: An axial shifted mlp architecture for vision.  In _ICLR_ , 2021.
  * Yu et al. [2022] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan.  Metaformer is actually what you need for vision.  In _CVPR_ , pages 10819–10829, 2022.
  * Laugel et al. [2019] Thibault Laugel, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and Marcin Detyniecki.  The dangers of post-hoc interpretability: Unjustified counterfactual explanations.  In _IJCAI_ , 2019.
  * Rudin [2019] Cynthia Rudin.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ , 1(5):206–215, 2019.
  * Arrieta et al. [2020] Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al.  Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai.  _Information Fusion_ , 58:82–115, 2020.
  * Chen et al. [2019a] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su.  This looks like that: deep learning for interpretable image recognition.  In _NeurIPS_ , 2019a.
  * Erhan et al. [2009] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Visualizing higher-layer features of a deep network.  _University of Montreal_ , 1341(3):1, 2009.
  * Simonyan et al. [2013] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  _arXiv preprint arXiv:1312.6034_ , 2013.
  * Zeiler and Fergus [2014] Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In _ECCV_ , 2014.
  * Mahendran and Vedaldi [2015] Aravindh Mahendran and Andrea Vedaldi.  Understanding deep image representations by inverting them.  In _CVPR_ , 2015.
  * Yosinski et al. [2015] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson.  Understanding neural networks through deep visualization.  _arXiv preprint arXiv:1506.06579_ , 2015.
  * Bach et al. [2015] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  _PloS one_ , 10(7):e0130140, 2015.
  * Zhou et al. [2016] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In _CVPR_ , 2016.
  * Shrikumar et al. [2017] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje.  Learning important features through propagating activation differences.  In _ICML_ , 2017.
  * Ribeiro et al. [2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  ”why should i trust you?” explaining the predictions of any classifier.  In _KDD_ , 2016.
  * Zintgraf et al. [2017] Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling.  Visualizing deep neural network decisions: Prediction difference analysis.  In _ICLR_ , 2017.
  * Koh and Liang [2017] Pang Wei Koh and Percy Liang.  Understanding black-box predictions via influence functions.  In _ICML_ , 2017.
  * Lundberg and Lee [2017] Scott M Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  In _NeurIPS_ , 2017.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _ICCV_ , pages 9650–9660, 2021.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  _arXiv preprint arXiv:2304.07193_ , 2023.
  * Alvarez Melis and Jaakkola [2018] David Alvarez Melis and Tommi Jaakkola.  Towards robust interpretability with self-explaining neural networks.  In _NeurIPS_ , 2018.
  * Kim et al. [2018] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al.  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).  In _ICML_ , 2018.
  * Wang [2019] Tong Wang.  Gaining free or low-cost interpretability with interpretable partial substitute.  In _ICML_ , 2019.
  * Subramanian et al. [2018] Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg-Kirkpatrick, and Eduard Hovy.  Spine: Sparse interpretable neural embeddings.  In _AAAI_ , 2018.
  * Chen et al. [2016] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel.  Infogan: Interpretable representation learning by information maximizing generative adversarial nets.  In _NeurIPS_ , 2016.
  * You et al. [2017] Seungil You, David Ding, Kevin Canini, Jan Pfeifer, and Maya Gupta.  Deep lattice networks and partial monotonic functions.  In _NeurIPS_ , 2017.
  * Wang et al. [2023] Wenguan Wang, Cheng Han, Tianfei Zhou, and Dongfang Liu.  Visual recognition with deep nearest centroids.  In _ICLR_ , 2023.
  * Yu et al. [2023a] Yaodong Yu, Tianzhe Chu, Shengbang Tong, Ziyang Wu, Druv Pai, Sam Buchanan, and Yi Ma.  Emergence of segmentation with minimalistic white-box transformers.  _arXiv preprint arXiv:2308.16271_ , 2023a.
  * Yu et al. [2023b] Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin D Haeffele, and Yi Ma.  White-box transformers via sparse rate reduction.  _arXiv preprint arXiv:2306.01129_ , 2023b.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _ECCV_ , 2014.
  * Zhou et al. [2017] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In _CVPR_ , 2017.
  * Wightman [2019] Ross Wightman.  Pytorch image models.  https://github.com/rwightman/pytorch-image-models, 2019.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Liu et al. [2021a] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _ICCV_ , 2021a.
  * Touvron et al. [2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In _ICML_ , pages 10347–10357, 2021.
  * Liu et al. [2021b] Hanxiao Liu, Zihang Dai, David So, and Quoc V Le.  Pay attention to mlps.  In _NeurIPS_ , pages 9204–9215, 2021b.
  * Wang et al. [2021a] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  In _ICCV_ , pages 568–578, 2021a.
  * Trockman and Kolter [2022] Asher Trockman and J Zico Kolter.  Patches are all you need?  _arXiv preprint arXiv:2201.09792_ , 2022.
  * Kirillov et al. [2019] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár.  Panoptic feature pyramid networks.  In _CVPR_ , pages 6399–6408, 2019.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _ICCV_ , pages 2961–2969, 2017.
  * Pedregosa et al. [2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  _JMLR_ , 12:2825–2830, 2011.
  * Contributors [2020] MMSegmentation Contributors.  MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark.  https://github.com/open-mmlab/mmsegmentation, 2020.
  * Chen et al. [2019b] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al.  Mmdetection: Open mmlab detection toolbox and benchmark.  _arXiv preprint arXiv:1906.07155_ , 2019b.
  * Zhong et al. [2020] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.  Random erasing data augmentation.  In _AAAI_ , pages 13001–13008, 2020.
  * Zhang et al. [2017] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  _arXiv preprint arXiv:1710.09412_ , 2017.
  * Yun et al. [2019] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In _ICCV_ , pages 6023–6032, 2019.
  * Szegedy et al. [2016] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In _CVPR_ , pages 2818–2826, 2016.
  * Polyak and Juditsky [1992] Boris T Polyak and Anatoli B Juditsky.  Acceleration of stochastic approximation by averaging.  _SIAM journal on control and optimization_ , 30(4):838–855, 1992.
  * Mitchell [1999] Tom M Mitchell.  Machine learning and data mining.  _Communications of the ACM_ , 42(11):30–36, 1999.
  * Xu and Wunsch [2005] Rui Xu and Donald Wunsch.  Survey of clustering algorithms ieee transactions on neural networks, vol. 16 (3), 2005.
  * Van Gansbeke et al. [2020] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool.  Scan: Learning to classify images without labels.  In _ECCV_ , 2020.
  * Duncan and Ayache [2000] James S Duncan and Nicholas Ayache.  Medical image analysis: Progress over two decades and the challenges ahead.  _TPAMI_ , 22(1):85–106, 2000.
  * Yang et al. [2020] Xiao Yang, Fangyun Wei, Hongyang Zhang, and Jun Zhu.  Design and interpretation of universal adversarial patches in face detection.  In _ECCV_ , pages 174–191, 2020.
  * Dong et al. [2019] Yinpeng Dong, Hang Su, Baoyuan Wu, Zhifeng Li, Wei Liu, Tong Zhang, and Jun Zhu.  Efficient decision-based black-box adversarial attacks on face recognition.  In _CVPR_ , pages 7714–7722, 2019.
  * Wang et al. [2019] Yaqin Wang, Dongfang Liu, Hyewon Jeon, Zhiwei Chu, and Eric T Matson.  End-to-end learning approach for autonomous driving: A convolutional neural network model.  In _ICAART_ , 2019.
  * Liu et al. [2021c] Dongfang Liu, Yiming Cui, Xiaolei Guo, Wei Ding, Baijian Yang, and Yingjie Chen.  Visual localization for autonomous driving: Mapping the accurate location in the city maze.  In _ICPR_ , 2021c.
  * Antoniak [1974] Charles E Antoniak.  Mixtures of dirichlet processes with applications to bayesian nonparametric problems.  _The annals of statistics_ , pages 1152–1174, 1974.
  * Ferguson [1973] Thomas S Ferguson.  A bayesian analysis of some nonparametric problems.  _The annals of statistics_ , pages 209–230, 1973.
  * Chang and Fisher III [2013] Jason Chang and John W Fisher III.  Parallel sampling of dp mixture models using sub-cluster splits.  In _NeurIPS_ , 2013.
  * Chen [2015] Gang Chen.  Deep learning with nonparametric clustering.  _arXiv preprint arXiv:1501.03084_ , 2015.
  * Dinari et al. [2019] Or Dinari, Angel Yu, Oren Freifeld, and John Fisher.  Distributed mcmc inference in dirichlet process mixture models using julia.  In _2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)_ , pages 518–525, 2019.
  * Wang et al. [2021b] Zeya Wang, Yang Ni, Baoyu Jing, Deqing Wang, Hao Zhang, and Eric Xing.  Dnb: A joint learning framework for deep bayesian nonparametric clustering.  _TNNLS_ , 33(12):7610–7620, 2021b.
  * Blei and Jordan [2006] David M Blei and Michael I Jordan.  Variational inference for dirichlet process mixtures.  _Bayesian analysis_ , 2006.
  * Hoffman et al. [2013] Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley.  Stochastic variational inference.  _JMLR_ , 2013.
  * Hughes and Sudderth [2013] Michael C Hughes and Erik Sudderth.  Memoized online variational inference for dirichlet process mixture models.  In _NeurIPS_ , 2013.
  * Huynh et al. [2016] Viet Huynh, Dinh Phung, and Svetha Venkatesh.  Streaming variational inference for dirichlet process mixtures.  In _ACML_ , pages 237–252, 2016.
  * Kurihara et al. [2006] Kenichi Kurihara, Max Welling, and Nikos Vlassis.  Accelerated variational dirichlet process mixtures.  In _NeurIPS_ , 2006.
  * Ester et al. [1996] Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al.  A density-based algorithm for discovering clusters in large spatial databases with noise.  In _KDD_ , pages 226–231, 1996.
  * Shah and Koltun [2018] Sohil Atul Shah and Vladlen Koltun.  Deep continuous clustering.  _arXiv preprint arXiv:1803.01449_ , 2018.
  * Tapaswi et al. [2019] Makarand Tapaswi, Marc T Law, and Sanja Fidler.  Video face clustering with unknown number of clusters.  In _ICCV_ , pages 5027–5036, 2019.
  * Pakman et al. [2020] Ari Pakman, Yueqi Wang, Catalin Mitelut, JinHyung Lee, and Liam Paninski.  Neural clustering processes.  In _ICML_ , pages 7455–7465, 2020.
  * Ronen et al. [2022] Meitar Ronen, Shahaf E Finder, and Oren Freifeld.  Deepdpm: Deep clustering with an unknown number of clusters.  In _CVPR_ , 2022.

Supplementary Material

For a better understanding of the main paper, we provide additional details in
this supplementary material, which is organized as follows:

  * •

§A provides the pseudo code of FEC.

  * •

§B introduces more experimental details.

  * •

§C offers more results and discussions about the modeled representatives.

  * •

§D discusses our limitations, societal impact, and directions of future work.

##  A Pseudo Code

To facilitate a comprehensive understanding of FEC, we provide pseudo code for
our feature encoding and feature pooling in Algorithm S1.

##  B More Experimental Detail

Image Classification. In this task, several widely-used data augmentations are
adopted to better train the model, including random horizontal flipping,
random pixel erase [113], MixUp [114], CutMix [115], and label smoothing
[116]. We employ an AdamW [102] optimizer using a cosine decay learning rate
scheduler and 5 epochs of warm-up. The momentum and weight decay are set to
0.9 and 0.05, respectively. A batch size of 1024 and an initial learning rate
of 0.001 are used. We also use exponential moving average [117] to enhance the
training. Throughput (image/s), or FPS, is measured using the same script
[103, 104] on a single V100 GPU using a batch size of 256. The reported values
are averaged by 100 iterations after 20 warm iterations. We use the same
codebase and tricks (_e.g_., multi-head computing) as in [26]. In addition, we
use almost the same hyperparameters and architectures as in [26] for fair
comparison.

Downstream Tasks. During training, backbones are initialized with weights pre-
trained on ImageNet [24], while the other parts are initialized randomly.

##  C Modeled Representative

In the “Study of _Ad-hoc_ Interpretability” section of the main paper, it is
highlighted that FEC’s final cluster assignments display consistent semantic
representations. These representations frequently correlate with distinct
objects or their components and demonstrate a close alignment with human
perception. Here we visualize more results of cluster assignments in Fig. S1
to clarify the FEC’s principles. Similar conclusions can be drawn from Fig.
S1, which confirms again the _ad-hoc_ interpretability and effectiveness of
FEC.

Figure S1: Inspection of the modeled representatives (§C) on ImageNet-1K [24]
val.  Algorithm S1 Pseudo code of FEC in a PyTorch-like style.

⬇

# feat_i: input feature (N x C), where N = W x H

# C: number of channels

# N: resolution of input feature

# O: number of cluster centers. In pooling, O = N/4. In encoding, O is a
hyperparameter (O < N).

# M: similarity matrix (Eq.5)

# A: cluster assigment matrix

# R: representatives

# sig: sigmoid function

# alpha and beta: learnable parameters

def model_representatives(feat_i)

# center initialization (Eq.4)

feat_k = conv_k(feat_i) # (N x C’)

feat_v = conv_v(feat_i) # (N x C’)

feat_c_k = ada_pool(feat_k) # (O x C’)

feat_c_v = ada_pool(feat_v) # (O x C’)

# compute similarities and cluster assigments (Eq.5)

M = cosine_sim(feat_k, feat_c_k) # (N x O)

A = torch.argmax(M, dim=1) # (N x O)

# aggragate the feature of representatives (Eq.6)

R = aggragate_feature(feat_v, feat_c_v, A) # (O x C’)

return R, M

def pooling(feat_i)

R, _ = model_representatives(feat_i)

res_conn = ResConn(feat_i) # residual connection (Eq.9)

return R + res_conn

def encoding(feat_i)

R, M = model_representatives(feat_i)

# feature dispatching (Eq.7)

refined_M = sig(alpha * M + beta).permute(1,0) # (O x N)

feat_d = ( R.unsqueeze(dim=1) * refined_M.unsqueeze(dim=-1) ).sum(dim=0) # (N
x C’)

feat_d = MLP(feat_d) # (N x C)

out = feat_i + feat_d # residual connection

return out

##  D Discussion

Limitation Analysis. One limitation of our approach is the adoption of a
straightforward clustering mechanism, primarily aimed at ensuring
computational efficiency. While this design choice contributes to faster
processing times, it may inadvertently lead to sub-optimal performance in
certain scenarios. Additionally, akin to many parametric clustering algorithms
[118, 119, 120], our method requires the manual definition of the number of
clusters to keep the same resolution with previous works [11, 103, 15, 71].
This aspect introduces a degree of subjectivity and potential bias, as the
predetermined cluster count may not align perfectly with the intrinsic
structure of specific images, particularly in dealing with datasets where the
optimal number of clusters is not known a priori or varies significantly.

Societal Impact. This work provides a clustering perspective for transparent,
_ad-hoc_ interpretable feature extraction, and accordingly introduces a novel
visual backbone which reformulates the entire process of feature extraction as
representative selection. On positive side, the approach advances network
interpretability and is valuable in safety-sensitive applications, _e.g_.,
medical image analysis [121], face recognition [122, 123], and autonomous
driving [124, 125]. For potential negative social impact, the erroneous
recognition may cause inaccurate decision or planning of systems based on the
results. In addition, the potential bias inherent in the training data may be
exploited for malicious purposes.

Future Work. This work also comes with new challenges, certainly worth further
exploration:

  * •

Incorporating Advanced Clustering Algorithms. In future developments, we aim
to augment the FEC framework by incorporating advanced clustering algorithms.
Our current model prioritizes computational efficiency with a straightforward
clustering mechanism, but we recognize opportunities for enhancing performance
and accuracy. Upcoming versions will investigate sophisticated algorithms
adept at managing complex data structures and distributions, potentially
increasing the granularity and precision of feature extraction for more
refined and accurate visual representations. An intriguing avenue is
transitioning from parametric clustering, which presupposes a fixed number of
clusters, to nonparametric clustering, where the number of clusters is
undetermined. There are numerous techniques for nonparametric clustering,
including Bayesian nonparametric (BNP) mixture models (exemplified by the
Dirichlet Process Mixture (DPM) model [126, 127]), DPM sampler [128, 129, 130,
131], variational DPM inference [132, 133, 134, 135, 136], density-based
approach [137], nearest-neighbor graph [138], supervised approach [139, 140],
dynamic network architecture [141]. We have explored a very recent work,
_i.e_., DeepDPM [141]. However, after running their code, we find that DeepDPM
is notably complex and require substantial computational time. Moving forward,
our focus is on identifying better trade-offs between complexity,
computational efficiency, and performance.

  * •

Combination with Set-prediction Architectures. The recent emergence of set-
prediction architectures, such as DETR [32], presents a significant
opportunity to utilize the representatives modeled by FEC more effectively.
Unlike traditional methods that rely on hand-crafted components like non-
maximum suppression for post-processing and pre-defined anchors for label
assignments, these approaches simplify the pipeline by allowing for end-to-end
training and inference. This reduces the need for many of the specialized
components typically used in object detection systems and provides an ideal
framework for utilizing the representatives extracted by FEC. For example, the
modeled representatives can be applied as a metric for distance measurement,
aiding in the stabilization of bipartite matching. This integration
effectively infuses the concept of “instances” (or representatives) into the
feature extraction process, which stands as the primary motivation behind this
work.