# Probing clustering in neural network representations

Thao Nguyen
University of Washington
&Simon Kornblith11footnotemark: 1
Anthropic
Work done while at Google

###### Abstract

Neural network representations contain structure beyond what was present in
the training labels. For instance, representations of images that are visually
or semantically similar tend to lie closer to each other than to dissimilar
images, regardless of their labels. Clustering these representations can thus
provide insights into dataset properties as well as the network internals. In
this work, we study how the many design choices involved in neural network
training affect the clusters formed in the hidden representations. To do so,
we establish an evaluation setup based on the BREEDS hierarchy [42], for the
task of subclass clustering after training models with only superclass
information. We isolate the training dataset and architecture as important
factors affecting clusterability. Datasets with labeled classes consisting of
unrelated subclasses yield much better clusterability than those following a
natural hierarchy. When using pretrained models to cluster representations on
downstream datasets, models pretrained on subclass labels provide better
clusterability than models pretrained on superclass labels, but only when
there is a high degree of domain overlap between the pretraining and
downstream data. Architecturally, we find that normalization strategies affect
which layers yield the best clustering performance, and, surprisingly, Vision
Transformers attain lower subclass clusterability than ResNets.

##  1 Introduction

Today, it costs about 3000x less to train a computer vision model to AlexNet-
level accuracy than it did a decade ago. To make use of the power afforded by
new microprocessor technologies and more efficient architectures and training
techniques, researchers have constructed ever-larger image datasets. These
datasets are so large that it is impossible to understand what they contain
without the assistance of machine learning models. If one spent a second
looking at each image, it would take more than a lifetime to look at the
entire JFT-4B dataset, on which today’s top-performing ImageNet models are
pre-trained.

An easy way to distill a large dataset to a representative set of images is by
clustering representations. By embedding images in the representation space of
a pre-trained model and then examining only a few images from each cluster,
one can obtain an understanding of what the dataset contains without
laboriously inspecting individual images. For example, Oakden-Rayner et al.
[36] show that performing k-means clustering in the embedding space of a
network trained on the CXR-14 chest X-ray dataset reveals a large cluster of
cases with chest drains in the pneuomothorax class. Since a chest drain is
inserted to treat pneuomothorax, a model that simply detects the presence or
absence of a chest drain to identify pneuomothorax has no diagnostic value.
Given this observation, the authors highlight the importance of detecting
hidden stratification: when the labels used to train and evaluate machine
learning models do not sufficiently describe meaningful variations within the
population.

Despite the utility of clustering for understanding dataset composition, the
study of factors that affect clustering of representations in pre-trained
models has received relatively little attention in the deep learning
literature. Instead, significant research effort has been devoted to the
design of novel deep algorithms to cluster images in an unsupervised fashion,
where networks are trained solely for the purpose of clustering. By contrast,
we analyze how various aspects of the standard training procedure can affect
the cluster quality in representation spaces of the resulting model. We
believe that our setup closely resembles how clustering methods are likely to
be applied to large datasets in practice, since training new models on these
datasets requires significant computational resources.

Beyond offering insights on the data, clustering can also provide a novel lens
through which to view neural network representations. Previous work has
proposed to understand properties of hidden representations by measuring their
similarity [40, 24, 34, 52, 10], compatibility [27, 4, 8], or through the
performance of simple classifiers (“linear probes”) trained on them [2]. We
demonstrate that clustering is a useful addition to this toolbox. Compared to
the aforementioned approaches, clustering is less sensitive to distortions of
global representational structure, and because clusters can be visualized
directly, differences in representational properties revealed through
clustering are potentially more interpretable than a scalar measure of
similarity or accuracy.

In our experiments, we leverage the BREEDS hierarchy [42] to create a
realistic setup where models are trained on superclass labels, but each
superclass contains subclasses for which labels are not observed at training
time. We cluster representations within each superclass and measure the
accuracy with which we can recover the original subclasses. We also measure
clusterability of representations of downstream data that differs in
distribution from the initial training set. Using this setup, we uncover
several previously unrecognized phenomena. Our findings are as follows:

  * •

When dataset classes are generated by merging unrelated subclasses,
representations of images belonging to these subclasses tend to form separate
clusters. When subclasses follow a natural hierarchy, separating them in the
representation space can be significantly harder.

  * •

When using a pre-trained model to cluster image embeddings from downstream
datasets that have little overlap with the training set, pre-training on
subclass labels yields only marginally better clusterability of downstream
classes than pre-training on superclass labels.

  * •

Clusterability depends on architecture and normalization strategy.
Representations of Vision Transformers (ViTs) are less clusterable than
representations of similarly peforming ResNets.

  * •

Objectives and layers that offer the highest linear probe accuracy and
transferability to downstream tasks do not necessarily offer the best
clusterability.

  * •

Given the same training set and loss function, different training runs yield
similar clustering performance when compared to ground-truth subclasses.
However, the actual cluster assignments are highly inconsistent across runs.

All in all, these findings help us put the approach of clustering neural
network representations, especially in the presence of hidden stratification,
on a more solid empirical footing.

##  2 Related work

Clustering of embeddings has been commonly used in various machine learning
methods and applications. Oakden-Rayner et al. [36], Sohoni et al. [46]
investigate the use of clustering to identify “hidden stratification,” i.e.,
subclasses on which models achieve different performance; Sohoni et al. [46]
further show that retraining on the clusters found using distributionally
robust optimization can improve worst-case generalization error. Birodkar et
al. [5], Sorscher et al. [47] use clustering to reduce the number of dataset
examples on which models need to be trained to obtain high performance.
Comparatively little work has investigated properties of the resulting
clusters. Yang et al. [55] experiment with combining pairs of CIFAR-10 classes
into superclasses. Although extensive training leads to a “neural collapse”
phenomenon where representations collapse to class centroids [37], they show
that the representations within a given class nonetheless remain structured
and it is possible to recover the subclasses via clustering. Our work seeks to
study this clusterability property in a systematic way, by intervening on not
just the dataset structure, but also the training objective as well as the
model architecture.

As discussed in the introduction, our work is related to, but distinct from,
work that has proposed specialized deep representation learning methods for
clustering. Approaches such as DEC [53], SCAN [49], SPICE [35], and TCL [28]
assign images to clusters after one or more stages of model training using a
specialized clustering objective. Subclass distillation [33] learns subclasses
of dataset classes which are then used to improve distillation. Other methods
aim to cluster a novel set of images using knowledge gleaned from another
image set [17, 14, 15, 57], a task that has been termed novel category
discovery (NCD). Some self-supervised learning methods such as DeepCluster [6]
and SwAV [7] also perform clustering, but with the aim of learning good
representations for linear classification, semi-supervised learning, and
transfer learning, rather than producing cluster assignments for training
images. In our work, we experiment with using learned representations to
perform clustering on downstream datasets, whose classes overlap with ImageNet
categories to different extents.

Clustering is only one of many methods that can be used to simplify or
otherwise gain insight into data. Other methods seek to find influential
training points for a specific model prediction [23, 19], training datapoints
that are difficult for the model to learn [21, 1, 43], subsets of training
datapoints that are sufficient to achieve high performance [30, 39, 22, 47],
or types of data on which the model is consistently poor [56, 45, 12, 51, 9].
We demonstrate via comparison with other common representation understanding
techniques (e.g. linear probe), that subclass clustering offers a distinctive
lens through which to study what neural networks can learn from incomplete
labels.

##  3 Experiment setup

#### Datasets

To obtain datasets with natural subclass-superclass hierarchy, we leverage
BREEDS [42], a subset of ImageNet that follows a modified WordNet hierarchy to
better suit object recognition tasks. Compared to the standard WordNet
hierarchy, BREEDS makes two major changes: (i) BREEDS places nodes under a
common ancestor only if they share some visual information, whereas the
WordNet hierarchy is sometimes defined in abstract ways (e.g., “umbrella” and
“roof” are both subclasses of “covering”); and (ii) nodes of similar
specificity (e.g., “dog” and “cat”) share the same distance from the root.
These changes allow us to study how label hierarchy is reflected in neural
network representations in a controlled fashion.

Our experiments involve three benchmarks suggested in the original paper:
entity-13, living-17 and nonliving-26. Dataset names are based on the root of
the subtree in WordNet where nodes are taken from, and the resulting number of
superclasses. We modify entity-13 so that each superclass contains only 4
subclasses (i.e. original ImageNet classes), resembling the structure of the
other two datasets. Although substantially smaller than the pre-training
datasets of top-performing vision models, all BREEDS datasets still contain
$>$60,000 images, and they are the largest datasets of which we are aware with
a carefully constructed hierarchy. Refer to Table A.1 in the Appendix for more
details.

#### Models

Most of our experiments involve ResNet-50 [16] and VGG-16 [44], two
architectures that have been extensively studied in literature and differ in
many ways. Section 7.2 further explores one such difference — the presence of
normalization layers. In the architecture ablation study, we also report
clustering performance of ViTs [11].

#### Clustering method & evaluation

For each BREEDS dataset, we train models using corresponding images from the
train split and only the superclasses as the target labels. For evaluation, we
extract embeddings of the corresponding validation split for each dataset from
a certain layer, and perform clustering within each superclass: the number of
clusters is set to be the number of subclasses per superclass, multiplied by
some overclustering factor between 1 and 10, to account for the presence of
more fine-grained structure beyond the original ImageNet labels.

Previous work indicates that agglomerative clustering is among the best
clustering methods for representations obtained from image classifiers [31].
We provide a comparison between k-means and agglomerative clustering with
various linkage criteria in Table B.1 in the Appendix. This initial
exploration leads us to adopt the agglomerative clustering based on Ward
linkage [50] in our experiments. This clustering method is scalable to large
number of samples and clusters, and unlike k-means, it is deterministic. We
find that compared to using raw embeddings, normalizing the embeddings to have
unit norms also helps boost clustering performance while preserving the
overall trend, and thus adopt this approach in all experiments below.

To assess cluster quality, we use both purity and adjusted mutual information
(AMI). We find that the two metrics yield very similar trends across all of
our experiments (see Appendix C). To compute purity, we label all samples in
each cluster with the class that is most frequent in the cluster, and measure
the accuracy of this assignment with respect to the true labels. AMI measures
mutual information between the predicted clusters and the true labels,
adjusted for chance. Both metrics allow permutation of cluster labels, thus
accounting for the possibility of each subclass being split up into multiple
clusters (e.g. when overclustering factor $>$ 1). They also both range from 0
to 1, with a higher value indicating better alignment between the clusterings
and ground-truth labels. Implementations of clustering algorithms and metrics
are done through scikit-learn.

In the subsequent sections, we investigate how the clustering performance with
respect to the true subclass structure may be impacted by various components
of the learning process — training data, loss function and model architecture
— in settings where the target data comes from the same or different
distributions as the training set.

##  4 Clusterability trends within a network

We track the clusterability of subclasses in different layers throughout each
trained network. Clustering performance, measured with AMI and purity scores,
shows an overall increasing trend with depth. This is not surprising as deeper
layers have been shown to learn more complex features so as to predict the
labeled classes well. A similar phenomenon has been observed with existing
tools such as linear probes [2]. For each of the layers from which we obtain
embeddings and perform clustering, we also train a linear probe to map from
the layer representation to the subclass labels. Our clustering metrics and
linear probe accuracy both generally increase with depth, but while linear
probe accuracy is nearly monotonic, subclass clustering is not (Figure 1).

Appendix D contains additional results, as well as visualizations of image
clusters found at different stages of a ResNet-50. We also fix a layer and
analyze clusterability over the course of training for models trained on
different BREEDS datasets (Appendix Figure D.10). We find that in most cases,
AMI across layers fluctuates substantially throughout training. The only
exception is training with entity-13-shuffled (refer to Section 5.1 for its
construction), where AMI stays mostly unchanged for the majority of the
network after epoch 100 (i.e. 25% of the total training time), and only the
clusterability of the fully-connected layers continues to rise over time.

Figure 1: Clustering performance is not fully aligned with linear probe
accuracy. We compare AMI and purity of subclass clustering to linear probe
performance for different layers in a ResNet-50 trained on entity-13-shuffled.
Although all metrics generally increase with depth, clustering evaluations are
not perfectly correlated with linear probe accuracy, and reveal greater
variations across layers that belong to the same residual block.

##  5 The role of training data

###  5.1 Shuffling the dataset hierarchy

Figure 2: Clustering performance improves substantially when labeled classes
contain heterogeneous subclasses. We randomize the subclass-superclass mapping
in entity-13 and find that this increases AMI across layers for both VGG-16
(left) and ResNet-50 (right). The improvements in clustering quality also tend
to be the largest near the end of either network.

When trained on superclasses drawn from a hierarchical dataset, a network
could pick up only features that are shared across the subclasses of each
superclass, or it could instead learn separate features that distinguish each
subclass, and then combine these subclasses to predict the superclass. Our
clustering evaluation suggests that the strategy that networks use depends on
the properties of the training data.

For standard training on BREEDS (green lines in Figure 2), AMI values remain
low but non-zero throughout the network. We posit that this is due to the
natural hierarchy present in the training data: although images under each
superclass belong to different subclasses, they are still more visually
similar to one another than images from other superclasses. As an
intervention, we randomly map each subclass to one superclass in entity-13,
while still keeping the number of subclasses per superclass constant. We call
this variant entity-13-shuffled. As shown in Figure 2, making the images under
each superclass much more heterogeneous greatly improves subclass
clusterability (orange lines). Appendix E contains additional experiments with
the shuffled version of living-17 dataset. This result suggests that when
labeled classes contain heterogeneous subclasses without a lot of visual
features in common, the “simplest” strategy for the model to predict a labeled
class may be to learn a different set of features for each subclass. This in
turn leads to high subclass clusterability.

###  5.2 Impact of training distribution on clustering performance for
external datasets

Figure 3: When using pre-trained embedddings to recover classes in external
datasets, training with subclass labels yields only marginally better
clusterability than training with superclass labels. We evaluate the ability
of ResNet-50 models, trained on ImageNet or different BREEDS datasets, to
recover the original classes in four downstream datasets. We cluster the image
embeddings of all examples in each dataset and measure how well the cluster
assignments align with ground-truth labels. Using fine-grained (i.e. subclass)
labels during pre-training offers some improvement over using superclass
labels, but the improvement tends to be relatively small, suggesting that
subclass information provides little benefit when detecting class structure in
out-of-distribution data. We also compare with the self-supervised model BYOL
[13], which, despite not having access to any ground-truth labels during
training, offers good downstream clustering performance, especially on coarse-
grained data.

Having examined how hierarchical structure in the training data may affect
clustering performance on the same data distribution, we next explore the
implication of varying image distribution and label granularity on the ability
to recover class structure in external datasets. We use four downstream
datasets with different degrees of overlap with ImageNet and granular
structures:

  * •

Tf_flowers [48]: an image dataset of 5 classes of flowers, with ‘daisy’ also
being an ImageNet class.

  * •

UC Merced [54]: a 21-class land use remote sensing image dataset. Only the
‘buildings’ class may potentially overlap with ImageNet training data.

  * •

Oxford-IIIT Pets [38]: a 37-category image dataset of dog and cat breeds. Out
of the four in the list, this dataset has the most overlap with ImageNet,
sharing 25 of its classes.

  * •

Cars196 [26]: a dataset of 196 classes of cars based on make, model and year.
This is the most fine-grained dataset in our evaluation; ImageNet contains
cars, but only use general type labels (e.g. ‘sports car’).

For each test set, we extract the penultimate layer embeddings from a
ResNet-50 trained on BREEDS data, and cluster the embeddings into $N$
clusters, with $N$ being the original number of classes in the test set
multiplied by some overclustering factor between 1 and 10. In Figure 3, we
find that except for Cars196, training on ImageNet or just subsets of ImageNet
allows us to recover class structure in external datasets to a large extent,
even if these datasets have little in common with ImageNet.

Although using subclass labels during pre-training generally improves cluster
purity in downstream datasets compared to using superclass labels, this
improvement tends to be relatively small (Figure 3). Consequently, the
majority of the signals for clustering out-of-distribution data comes from
superclass supervision. The only exception to this observation is Oxford-IIIT
Pets, which also involves fine-grained classification and whose classes
directly overlap with the subclasses in entity-13 and living-17 (see Table
A.1). For this task, we observe significant increases in cluster purity when
using representations of models trained with subclass supervision.

##  6 The role of loss function

###  6.1 Supervised versus Self-supervised training

In Figure 3, we also compare clustering performance out-of-distribution
between a ResNet-50 trained on the full ImageNet in a supervised manner,
versus a ResNet-50 trained in a self-supervised manner without any ground-
truth labels. For the latter, we use Bootstrap Your Own Latent (BYOL) and
evaluate the model checkpoint open-sourced by Grill et al. [13].

We find that BYOL offers higher cluster purity compared to using label
supervision with less data, even for downstream datasets whose classes
directly overlap with the subclasses in the BREEDS training set, e.g. Oxford-
IIIT pet and living-17 share dog and cat classes. However, given the same
training set, supervised training produces better clustering performance than
self-supervised training in fine-grained classification tasks (i.e. Oxford-
IIIT pet and Cars196, see Figure 3).

###  6.2 Squared error versus Softmax cross-entropy

We next zoom in on the supervised setting and explore variations in loss
functions for models trained in a supervised manner. First, we experiment with
using squared loss instead of the standard softmax cross-entropy objective.
Figure 4 shows the AMI across layers of a ResNet-50 trained with either
objective. We find that squared error produces similar clustering performance
as softmax, except for at the last few layers of the network where squared
loss sees a drop in AMI. This observation is in line with previous results
showing that squared loss produces greater class separation and more
overfitting to the training classes compared to softmax [25].

When we use the representations obtained from these two loss functions to
recover the class structures in external datasets (see the list in Section
5.2), we find that squared error also leads to significantly lower
clusterability out-of-distribution, similar to what Kornblith et al. [25]
observe with transfer performance. We show the clustering results on UC Merced
and Oxford-IIIT Pet datasets in Figure 5 and defer results on the other two
external datasets to Figure F.1 in the Appendix. We repeat the same experiment
with other BREEDS datasets; see Appendix F for more details.

Figure 4: Training with squared error yields the same clusterability as
softmax cross-entropy, except at the later stages of the network. We compare
AMI across layers for training a ResNet-50 with squared error versus softmax
cross-entropy loss on living-17 dataset. The two objective functions produce
similar AMI profiles for most of the network, except at the later layers where
squared error sees a drop in AMI, possibly due to overfitting to the training
classes [25].

Figure 5: Compared to softmax cross-entropy, embeddings trained with squared
loss yield lower clustering performance on external datasets. Training a
ResNet-50 on living-17 dataset with squared loss also performs worse than
training with softmax cross-entropy at recovering the class structure in out-
of-distribution data. The full evaluations on all four external datasets can
be found in Appendix Figure F.1.

###  6.3 Impact of loss functions on clustering performance out-of-
distribution

To extend the findings from the previous section and investigate a wider range
of training objectives, we evaluate open-sourced models from [25] trained on
the full ImageNet dataset with nine different types of losses and
regularization methods. While Kornblith et al. [25] focus on the
transferability of the resulting hidden representations, our work measures the
quality of the representations by the ability to recover class structures in
external datasets (see the dataset list in Section 5.2). Clustering results
averaged across three seeds are reported in Figure 6.

We again find that the clustering performance from using models trained with
squared error is significantly worse than from models trained with other
objective functions. However, although Kornblith et al. [25] find that
unregularized softmax cross-entropy provides the most transferable features —
measured by the accuracies of linear transfer and $k$-nearest neighbors on a
variety of tasks (including Oxford-IIIT Pets) — our clustering evaluation
shows that softmax can yield inferior clusters compared to other loss
functions, especially on the Oxford-IIIT Pets classification task. More
specifically, when training a fixed model with the goal of using it to uncover
structure in downstream related datasets, regularizers such as dropout and
label smoothing or alternative losses such as sigmoid cross-entropy can be
very useful, even though they result in worse linear transfer.

Figure 6: Training objectives affect the clustering performance on external
data; although softmax cross-entropy has been shown to yield better linear
transfer than other losses, it actually produces worse clusterability. We take
the models open-sourced by Kornblith et al. [25] that have been trained on the
full ImageNet with nine different loss functions, and evaluate how well the
embeddings of these models can recover classes in external datasets. Across
four evaluation sets, squared loss yields the worst clusterings, followed by
cosine softmax loss. While Kornblith et al. [25] find that softmax offers the
highest accuracies for linear transfer and $k$-nearest neighbors on external
tasks, our evaluation metric shows that in the case of subclass clustering,
other loss functions and regularization methods such as sigmoid and dropout
may be more ideal.

##  7 The role of model architecture

###  7.1 VGG versus ResNet

Having examined the influence of training set and loss function on the
clustering of the hidden representations, we next look at another widely
studied component of training: model architecture. We track the clusterability
of subclasses in a range of layers making up each architecture. In ResNet-50,
we examine only the layers following a residual connection. In VGG-16, we look
at all convolutional layers as well as two fully connected layers preceding
the output layer.

On entity-13 and entity-13-shuffled, we observe that the first fully connected
layer (out of three) in VGG-16 is where subclass clustering is most aligned
with ground truth. In contrast, clusterability tends to increase throughout
and peak at the end for ResNet-50 architecture (Figure 7). Refer to Appendix G
for corresponding results on nonliving-26 dataset.

###  7.2 Normalization layers

We seek to better understand the differences in clustering behavior between
VGG-16 and ResNet-50 by looking at the principal differences in the building
blocks of the two architectures. Besides the absence of residual connections,
VGG-16 also has no batch normalization layer [20]. Regularization benefits of
batch norm have been studied extensively in prior work [41, 32, 29].

We proceed to add batch normalization layers after every convolutional and
dense layer in the original VGG-16 architecture. We also experiment with layer
normalization [3], another common normalization technique. Clustering results
for entity-13 and entity-13-shuffled datasets are shown in Figure 7 (orange
lines). Superclass classification accuracies can be found in Appendix Table
G.1.

We find that normalization layers boost clustering performance only for the
intermediate layers of VGG-16, with a larger boost when superclasses contain
heterogeneous subclasses. In addition, batch normalization appears to yield
larger improvements than layer normalization. While prior work has suggested
that batch normalization can reduce class selectivity of individual features
and encourage generalization [32], we find that averaged across three random
seeds, the maximum AMI attainable across layers of VGG-16 does not change with
batch normalization, only shifts to an earlier layer.

Figure 7: Both the model architecture and the type of normalization affect
which layer provides the best clustering. We compare how clustering
performance changes with depth, for different model architectures trained on
entity-13 (left) and entity-13-shuffled (right), averaged across three
training runs. We find that both the architecture family and the normalization
method affect where subclass clusterability peaks. Furthermore, layer
normalization and batch normalization yield different clustering behaviors,
but the maximum AMI attainable from a VGG-16 does not increase with
normalization. Notably, ViT-B can yield worse clusterability compared to
ResNet-50 and VGG-16.

###  7.3 Vision Transformers

We also experiment with training ViT-B models [11] on entity-13 and
entity-13-shuffled, measuring clusterability at the residual connections that
follow self-attention and MLP layers in each encoder block. The clustering
performance across layers could be found in Figure 7 (green line). We find
that the maximum AMI attained by ViT-B is actually lower than that of
ResNet-50, and is worse than VGG-16 in the case of entity-13-shuffled.

##  8 Comparing clusterings across networks

(a)  (b)

(c)  (d)

Figure 8: Different training runs yield similar layer-wise clustering
performance, but cluster assignments are highly inconsistent across runs. The
top row shows the AMI across layers for three training runs of VGG-16 on
entity-13 (left) and entity-13-shuffled (right). These random seeds produce
highly similar AMI profiles. However, when measuring how aligned the actual
clusters are across runs with adjusted Rand index, we find that the ARI scores
are relatively low for most layers (bottom row) — the images assigned to each
cluster vary substantially across runs. An exception is high cluster
consistency found in the last few layers of entity-13-shuffled model, where we
also observe a corresponding rise in AMI. The same trends apply to ResNet-50;
see Appendix Figure H.1.

The previous sections examine how factors such as dataset structure, model
architecture and training objective may affect the ability to recover
subclasses from the hidden representations of a single model. We next turn to
measuring the consistency of the clusters uncovered across multiple training
runs. In the top row of Figure 8, we show the AMI across layers for three
random seeds of a VGG-16, trained with the entity-13 (left) and
entity-13-shuffled (right) datasets. We observe that different training runs
of the same model yield highly similar AMI profiles. Does this mean that these
runs produce similar clusters?

To measure similarity of clusterings, we use adjusted Rand index (ARI) [18].
ARI counts the proportion of similarly classified pairs of data points,
ignoring permutations, and corrects for chance. An ARI value of 0 indicates
that cluster labels for two clusterings are independent from each other, and a
value of 1 means the clusterings are identical. With this metric, we find that
cluster consistency across different training runs is surprisingly low. For
VGG-16 trained on standard BREEDS datasets, the similarity of cluster
assignments tends to peak at the first convolutional layer of the network. In
contrast, the similarity is relatively high both at the start and near the end
of a ResNet-50. Refer to Appendix H for experiments on other datasets and
ResNet-50, as well as visualizations of the clusters found across different
training runs for the same superclass.

Figure 8 also demonstrates that randomizing subclass-superclass mapping helps
boost cluster similarity across training runs (bottom right plot), especially
for the later layers in the networks. We find that the cluster similarity
score is the highest at the same layers that also see high alignment with
ground-truth labels (AMI), e.g. fc6 in VGG-16. Figure H.1 (Appendix) shows the
corresponding results for ResNet-50. We also provide a comparison of
clustering alignment between every pair of layers found in two different
random seeds; see Appendix H for the full heatmaps.

##  9 Conclusion

Inspired by previous work that leverages clustering in representation spaces,
we perform a systematic study of how standard training components — training
dataset, model architecture, and loss function — affect the subclass
clusterability at different layers of the resulting network. In the process,
we uncover several distinct phenomena, demonstrating the potential of subclass
clustering as an useful addition to the toolbox of existing techniques (e.g.
linear probes, representation similarity measures) that seek to understand
neural network internals.

In particular, we find that given only superclass information as labels during
training, the corresponding subclasses are better separated in the internal
representations when they are very different from each other than when they
follow a natural hierarchy. Providing supervision at a finer level of this
natural hierarchy by training on subclass labels improves the ability to
recover class structure in external datasets, but only when these datasets
have high domain overlap with the pre-training data. A second major theme of
our findings is that linear transferability and clusterability sometimes fail
to align, both when varying the depth at which representations are taken and
when varying the loss function with which networks are trained.

Our work presents several opportunities for further investigation. One
question is whether other clustering techniques can significantly outperform
agglomerative clustering. Different methods are likely to yield correlated
results, but novel ways of applying clustering could provide other kinds of
insights into the hidden representations. Besides, several of our observations
are in need of further explanation. In particular, our finding that different
normalization techniques lead to different patterns of clustering across
layers is somewhat perplexing, and we are unaware of any existing work that
has characterized a related phenomenon. Last but not least, future work could
explore the possibility of learning a linear transformation on the hidden
representations to improve subclass clusterings.

## Acknowledgements

We thank Pieter-Jan Kindermans, Mitchell Wortsman and Dhruba Ghosh for
providing feedback on the manuscript.

## References

  * Agarwal et al. [2022] C. Agarwal, D. D’souza, and S. Hooker.  Estimating example difficulty using variance of gradients.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10368–10378, 2022.
  * Alain and Bengio [2017] G. Alain and Y. Bengio.  Understanding intermediate layers using linear classifier probes, 2017\.  URL https://openreview.net/forum?id=ryF7rTqgl.
  * Ba et al. [2016] J. L. Ba, J. R. Kiros, and G. E. Hinton.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Bansal et al. [2021] Y. Bansal, P. Nakkiran, and B. Barak.  Revisiting model stitching to compare neural representations.  _Advances in Neural Information Processing Systems_ , 34:225–236, 2021.
  * Birodkar et al. [2019] V. Birodkar, H. Mobahi, and S. Bengio.  Semantic redundancies in image-classification datasets: The 10% you don’t need.  _arXiv preprint arXiv:1901.11409_ , 2019.
  * Caron et al. [2018] M. Caron, P. Bojanowski, A. Joulin, and M. Douze.  Deep clustering for unsupervised learning of visual features.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 132–149, 2018.
  * Caron et al. [2020] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  _Advances in Neural Information Processing Systems_ , 33:9912–9924, 2020.
  * Csiszárik et al. [2021] A. Csiszárik, P. Kőrösi-Szabó, Á. Matszangosz, G. Papp, and D. Varga.  Similarity and matching of neural network representations.  _Advances in Neural Information Processing Systems_ , 34:5656–5668, 2021.
  * d’Eon et al. [2022] G. d’Eon, J. d’Eon, J. R. Wright, and K. Leyton-Brown.  The spotlight: A general method for discovering systematic errors in deep learning models.  In _2022 ACM Conference on Fairness, Accountability, and Transparency_ , pages 1962–1981, 2022.
  * Ding et al. [2021] F. Ding, J.-S. Denain, and J. Steinhardt.  Grounding representation similarity through statistical testing.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=_kwj6V53ZqB.
  * Dosovitskiy et al. [2020] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Eyuboglu et al. [2022] S. Eyuboglu, M. Varma, K. K. Saab, J.-B. Delbrouck, C. Lee-Messer, J. Dunnmon, J. Zou, and C. Re.  Domino: Discovering systematic errors with cross-modal embeddings.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=FPCMqjI0jXN.
  * Grill et al. [2020] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in neural information processing systems_ , 33:21271–21284, 2020.
  * Han et al. [2019] K. Han, A. Vedaldi, and A. Zisserman.  Learning to discover novel visual categories via deep transfer clustering.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8401–8409, 2019.
  * Han et al. [2020] K. Han, S.-A. Rebuffi, S. Ehrhardt, A. Vedaldi, and A. Zisserman.  Automatically discovering and learning new visual categories with ranking statistics.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=BJl2_nVFPB.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hsu et al. [2018] Y.-C. Hsu, Z. Lv, and Z. Kira.  Learning to cluster in order to transfer across domains and tasks.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=ByRWCqvT-.
  * Hubert and Arabie [1985] L. Hubert and P. Arabie.  Comparing partitions.  _Journal of classification_ , 2(1):193–218, 1985\.
  * Ilyas et al. [2022] A. Ilyas, S. M. Park, L. Engstrom, G. Leclerc, and A. Madry.  Datamodels: Understanding predictions with data and data with predictions.  In _International Conference on Machine Learning_ , pages 9525–9587. PMLR, 2022.
  * Ioffe and Szegedy [2015] S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International conference on machine learning_ , pages 448–456. PMLR, 2015.
  * Jiang et al. [2021] Z. Jiang, C. Zhang, K. Talwar, and M. C. Mozer.  Characterizing structural regularities of labeled data in overparameterized models.  In M. Meila and T. Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 5034–5044. PMLR, 18–24 Jul 2021.  URL https://proceedings.mlr.press/v139/jiang21k.html.
  * Killamsetty et al. [2021] K. Killamsetty, S. Durga, G. Ramakrishnan, A. De, and R. Iyer.  Grad-match: Gradient matching based data subset selection for efficient deep model training.  In _International Conference on Machine Learning_ , pages 5464–5474. PMLR, 2021.
  * Koh and Liang [2017] P. W. Koh and P. Liang.  Understanding black-box predictions via influence functions.  In _International conference on machine learning_ , pages 1885–1894. PMLR, 2017.
  * Kornblith et al. [2019] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton.  Similarity of neural network representations revisited.  In _International Conference on Machine Learning_ , pages 3519–3529. PMLR, 2019.
  * Kornblith et al. [2021] S. Kornblith, T. Chen, H. Lee, and M. Norouzi.  Why do better loss functions lead to less transferable features?  _Advances in Neural Information Processing Systems_ , 34:28648–28662, 2021.
  * Krause et al. [2013] J. Krause, M. Stark, J. Deng, and L. Fei-Fei.  3d object representations for fine-grained categorization.  In _4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13)_ , Sydney, Australia, 2013.
  * Lenc and Vedaldi [2015] K. Lenc and A. Vedaldi.  Understanding image representations by measuring their equivariance and equivalence.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 991–999, 2015.
  * Li et al. [2022] Y. Li, M. Yang, D. Peng, T. Li, J. Huang, and X. Peng.  Twin contrastive learning for online clustering.  _International Journal of Computer Vision_ , 130(9):2205–2221, 2022.
  * Luo et al. [2018] P. Luo, X. Wang, W. Shao, and Z. Peng.  Towards understanding regularization in batch normalization.  _arXiv preprint arXiv:1809.00846_ , 2018.
  * Mirzasoleiman et al. [2020] B. Mirzasoleiman, J. Bilmes, and J. Leskovec.  Coresets for data-efficient training of machine learning models.  In _International Conference on Machine Learning_ , pages 6950–6960. PMLR, 2020.
  * Monath et al. [2021] N. Monath, K. A. Dubey, G. Guruganesh, M. Zaheer, A. Ahmed, A. McCallum, G. Mergen, M. Najork, M. Terzihan, B. Tjanaka, et al.  Scalable hierarchical agglomerative clustering.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, pages 1245–1255, 2021.
  * Morcos et al. [2018] A. S. Morcos, D. G. Barrett, N. C. Rabinowitz, and M. Botvinick.  On the importance of single directions for generalization.  _arXiv preprint arXiv:1803.06959_ , 2018.
  * Müller et al. [2020] R. Müller, S. Kornblith, and G. Hinton.  Subclass distillation.  _arXiv preprint arXiv:2002.03936_ , 2020.
  * Nguyen et al. [2021] T. Nguyen, M. Raghu, and S. Kornblith.  Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=KJNcAkY8tY4.
  * Niu et al. [2022] C. Niu, H. Shan, and G. Wang.  Spice: Semantic pseudo-labeling for image clustering.  _IEEE Transactions on Image Processing_ , 31:7264–7278, 2022.
  * Oakden-Rayner et al. [2020] L. Oakden-Rayner, J. Dunnmon, G. Carneiro, and C. Ré.  Hidden stratification causes clinically meaningful failures in machine learning for medical imaging.  In _Proceedings of the ACM conference on health, inference, and learning_ , pages 151–159, 2020.
  * Papyan et al. [2020] V. Papyan, X. Han, and D. L. Donoho.  Prevalence of neural collapse during the terminal phase of deep learning training.  _Proceedings of the National Academy of Sciences_ , 117(40):24652–24663, 2020.
  * Parkhi et al. [2012] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar.  Cats and dogs.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2012\.
  * Paul et al. [2021] M. Paul, S. Ganguli, and G. K. Dziugaite.  Deep learning on a data diet: Finding important examples early in training.  _Advances in Neural Information Processing Systems_ , 34:20596–20607, 2021.
  * Raghu et al. [2017] M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl-Dickstein.  Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability.  _Advances in neural information processing systems_ , 30, 2017.
  * Santurkar et al. [2018] S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry.  How does batch normalization help optimization?  _Advances in neural information processing systems_ , 31, 2018.
  * Santurkar et al. [2021] S. Santurkar, D. Tsipras, and A. Madry.  {BREEDS}: Benchmarks for subpopulation shift.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=mQPBmvyAuk.
  * Siddiqui et al. [2022] S. A. Siddiqui, N. Rajkumar, T. Maharaj, D. Krueger, and S. Hooker.  Metadata archaeology: Unearthing data subsets by leveraging training dynamics.  _arXiv preprint arXiv:2209.10015_ , 2022.
  * Simonyan and Zisserman [2014] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Singla et al. [2021] S. Singla, B. Nushi, S. Shah, E. Kamar, and E. Horvitz.  Understanding failures of deep networks via robust feature extraction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12853–12862, 2021.
  * Sohoni et al. [2020] N. Sohoni, J. Dunnmon, G. Angus, A. Gu, and C. Ré.  No subclass left behind: Fine-grained robustness in coarse-grained classification problems.  _Advances in Neural Information Processing Systems_ , 33:19339–19352, 2020.
  * Sorscher et al. [2022] B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and A. S. Morcos.  Beyond neural scaling laws: beating power law scaling via data pruning.  In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=UmvSlP-PyV.
  * Team [2019] T. T. Team.  Flowers, jan 2019.  URL http://download.tensorflow.org/example_images/flower_photos.tgz.
  * Van Gansbeke et al. [2020] W. Van Gansbeke, S. Vandenhende, S. Georgoulis, M. Proesmans, and L. Van Gool.  Scan: Learning to classify images without labels.  In _European conference on computer vision_ , pages 268–285. Springer, 2020.
  * Ward Jr [1963] J. H. Ward Jr.  Hierarchical grouping to optimize an objective function.  _Journal of the American statistical association_ , 58(301):236–244, 1963.
  * Wiles et al. [2022] O. Wiles, I. Albuquerque, and S. Gowal.  Discovering bugs in vision models using off-the-shelf image generation and captioning.  _arXiv preprint arXiv:2208.08831_ , 2022.
  * Williams et al. [2021] A. H. Williams, E. Kunz, S. Kornblith, and S. Linderman.  Generalized shape metrics on neural representations.  _Advances in Neural Information Processing Systems_ , 34:4738–4750, 2021.
  * Xie et al. [2016] J. Xie, R. Girshick, and A. Farhadi.  Unsupervised deep embedding for clustering analysis.  In _International conference on machine learning_ , pages 478–487. PMLR, 2016.
  * Yang and Newsam [2010] Y. Yang and S. Newsam.  Bag-of-visual-words and spatial extensions for land-use classification.  In _ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (ACM GIS)_ , 2010.
  * Yang et al. [2022] Y. Yang, J. Steinhardt, and W. Hu.  Are neurons actually collapsed? on the fine-grained structure in neural representations.  In _I Can’t Believe It’s Not Better Workshop: Understanding Deep Learning Through Empirical Falsification_ , 2022.
  * Yeh et al. [2020] C.-K. Yeh, B. Kim, S. Arik, C.-L. Li, T. Pfister, and P. Ravikumar.  On completeness-aware concept-based explanations in deep neural networks.  _Advances in Neural Information Processing Systems_ , 33:20554–20565, 2020.
  * Zhong et al. [2021] Z. Zhong, E. Fini, S. Roy, Z. Luo, E. Ricci, and N. Sebe.  Neighborhood contrastive learning for novel class discovery.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10867–10875, 2021.

##  Appendix A Training details

#### Hyperparameters

Except for variations in loss function, model architecture, and dataset, all
of VGG and ResNet models from our experiments are trained with batch size 256
for 400 epochs, using SGD with momentum 0.9 and weight decay 0.0001. ViT-B
uses AdamW optimizer and weight decay 0.1. Model weights are updated with
exponential moving average decay rate of 0.99. When network architecture
involves batch normalization layers, we also set batch norm decay rate to be
0.99. For better training stability, we use learning rate 0.01 for VGGs, 0.001
for ViT-Bs, and 0.1 for ResNets. In experiments involving squared loss, we
reduce the learning rate to 0.001. We also employ a cosine learning rate
schedule, with 5 warmup epochs.

Dataset |  Root node |  Number of superclasses |  Number of subclasses per superclass |  Superclass names
---|---|---|---|---
Living-17 |  ‘living thing’ |  17 |  4 |  ‘salamander’, ‘turtle’, ‘lizard’, ‘snake, serpent, ophidian’, ‘spider’, ‘grouse’, ‘parrot’, ‘crab’, ‘dog, domestic dog, Canis familiaris’, ‘wolf’, ‘fox’, ‘domestic cat, house cat, Felis domesticus, Felis catus’, ‘bear’, ‘beetle’, ‘butterfly’, ‘ape’, ‘monkey’
Nonliving-26 |  ‘non-living thing’ |  26 |  4 |  ‘bag’, ‘ball’, ‘boat’, ‘body armor, body armour, suit of armor, suit of armour, coat of mail, cataphract’, ‘bottle’, ‘bus, autobus, coach, charabanc, double-decker, jitney, motorbus, motorcoach, omnibus, passenger vehicle’, ‘car, auto, automobile, machine, motorcar’, ‘chair’, ‘coat’, ‘digital computer’, ‘dwelling, home, domicile, abode, habitation, dwelling house’, ‘fence, fencing’, ‘hat, chapeau, lid’, ‘keyboard instrument’, ‘mercantile establishment, retail store, sales outlet, outlet’, ‘outbuilding’, ‘percussion instrument, percussive instrument’, ‘pot’, ‘roof’, ‘ship’, ‘skirt’, ‘stringed instrument’, ‘timepiece, timekeeper, horologe’, ‘truck, motortruck’, ‘wind instrument, wind’, ‘squash’
Entity-13 |  ‘entity’ |  13 |  4 |  ‘garment’, ‘bird’, ‘reptile, reptilian’, ‘arthropod’, ‘mammal, mammalian’, ‘accessory, accoutrement, accouterment’, ‘craft’, ‘equipment’, ‘furniture, piece of furniture, article of furniture’, ‘instrument’, ‘man-made structure, construction’, ‘wheeled vehicle’, ‘produce, green goods, green groceries, garden truck’
Table A.1: Details about the BREEDS datasets used in our experiments. We
modify entity-13 dataset compared to what was proposed in the original paper
[42], by reducing the number of subclasses per superclass to 4, to match the
structure of the other two benchmarks. Note that, since most ImageNet classes
contain 1300 training images, the size of each training set is approximately
$1300\times\texttt{num\\_superclasses}\times\texttt{num\\_subclasses\\_per\\_superclass}$.

##  Appendix B K-means versus Hierarchical clustering

Table B.1 shows results of our preliminary investigation with several choices
of clustering algorithm, applied to the embeddings from the penultimate layer
of BREEDS-trained models. Based on purity values obtained, we decide to go
with agglomerative clustering using ‘ward’ linkage criterion for our
subsequent experiments.

Method |  Purity (entity-13) |  Purity (living-17) |  Purity (nonliving-26)
---|---|---|---
K-means |  0.368 |  0.360 |  0.371
Agglomerative clustering, linkage = ‘ward’ |  0.368 |  0.364 |  0.377
Agglomerative clustering, linkage = ‘average’ |  0.324 |  0.324 |  0.303
Agglomerative clustering, linkage = ‘complete’ |  0.337 |  0.340 |  0.344
Agglomerative clustering, linkage = ‘single’ |  0.268 |  0.269 |  0.266
Table B.1: Comparison of cluster purity produced by different clustering
methods. We use raw image embeddings from the penultimate layer of a VGG-16
trained on entity-13, living-17 and nonliving-26 datasets.

##  Appendix C Adjusted mutual information versus Purity

For all BREEDS datasets and models that we experiment with, AMI and purity
exhibit highly similar trends across layers — see Figure C.1 for example.

Figure C.1: We observe similar patterns of clustering performance across
layers when using either AMI or purity to measure cluster quality. Here we
show results averaged across 3 random seeds for ResNet-50s trained on
entity-13 (left), living-17 (middle) and nonliving-26 (right).

##  Appendix D Evolution of clusterability for a single network

Below we show sample images from each of the 4 clusters uncovered under the
superclasses ‘boat’ and ‘ball’ of nonliving-26, using embeddings from
different layers of a ResNet50 trained on the same dataset. Each row
represents one cluster, and each image visualization is labeled with the
ground-truth subclass name:

Figure D.1: Sample images from 4 clusters under ‘boat’ uncovered in
stage1_block1 of a ResNet-50.

Figure D.2: Sample images from 4 clusters under ‘boat’ uncovered in
stage2_block1 of a ResNet-50.

Figure D.3: Sample images from 4 clusters under ‘boat’ uncovered in
stage3_block1 of a ResNet-50.

Figure D.4: Sample images from 4 clusters under ‘boat’ uncovered in
stage4_block1 of a ResNet-50.

Figure D.5: Sample images from 4 clusters under ‘ball’ uncovered in
stage1_block1 of a ResNet-50.

Figure D.6: Sample images from 4 clusters under ‘ball’ uncovered in
stage2_block1 of a ResNet-50.

Figure D.7: Sample images from 4 clusters under ‘ball’ uncovered in
stage3_block1 of a ResNet-50.

Figure D.8: Sample images from 4 clusters under ‘ball’ uncovered in
stage4_block1 of a ResNet-50. Figure D.9: We compare AMI and purity of
subclass clustering to linear probe performance for different layers of a
ResNet-50 trained on entity-13 dataset. Similar to what was observed in Figure
1 with entity-13-shuffled data, clustering metrics exhibit more variations
across layers, and are not entirely aligned with linear probe accuracy.

(a)  (b)

(c)  (d)

Figure D.10: We track AMI changes over time, across layers of VGG-16 models
trained on different BREEDS datasets. We find that AMI values for most layers
vary continuously throughout the course of training. The only exception is
training with entity-13-shuffled dataset, where the AMI stays mostly unchanged
for the majority of the network after epoch 100 (i.e., 25% of total training
duration), and only the clustering performance of the last few fully-connected
layers continues to increase over time.

##  Appendix E The role of training data

Figure E.1: We repeat experiment involving randomizing subclass-superclass
mapping described in Section 5.1 on living-17 dataset. Similar to what was
observed in Figure 2, this intervention increases AMI across layers in both
VGG-16 (left) and ResNet-50 (right). The improvements in clustering quality
tend to be the largest near the end of either architecture.

##  Appendix F The role of loss function

Figure F.1 shows the full results of performing clustering using the
representations of a ResNet-50 trained with squared loss, on four external
datasets, for the experiment described in Section 6.2. Recall that this model
was initially trained on living-17 dataset. We repeat the analysis for a
ResNet-50 trained on entity-13-shuffled, and observe similar effects when
using squared loss compared to using softmax cross-entropy. Refer to Figures
F.2 and F.3 for more details.

Figure F.1: On living-17 dataset, training a ResNet-50 with squared loss
performs worse than training with softmax cross-entropy at recovering the
class structures in out-of-distribution data. This trend holds across all four
external datasets that we look at.

Figure F.2: Comparing AMI across layers for training a ResNet-50 with squared
error vs softmax cross-entropy loss on entity-13-shuffled dataset. The two
objective functions show similar AMI profiles for most of the network, except
at the later layers where squared error sees a steep drop in AMI, possibly due
to overfitting to the training classes [25].  Figure F.3: On
entity-13-shuffled dataset, training a ResNet-50 with squared loss performs
worse than training with softmax cross-entropy at recovering the class
structure in out-of-distribution data. This trend holds across all four
external datasets that we look at.

##  Appendix G The role of model architecture

Figure G.1: Similar to the observation in Figure 7.1, we find that the first fully connected layer (out of three) in VGG-16 is where maximum clustering performance is achieved. In contrast, clusterability tends to increase throughout and peak at the end of the network for ResNet-50 architecture.  Method |  Living-17 |  Nonliving-26 |  Entity-13 |  Entity-13-shuffled
---|---|---|---|---
ResNet-50 |  93.2 |  85.9 |  92.2 |  88.3
VGG-16 |  83.1 |  53.4 |  86.3 |  76.5
VGG-16 + Batch Norm |  N/A |  N/A |  86.6 |  78.1
VGG-16 + Layer Norm |  N/A |  N/A |  77.2 |  73.6
ViT-B/32 (average pooling) |  N/A |  N/A |  81.3 |  72.6
Table G.1: Superclass classification accuracy of different model architectures
trained with the BREEDS datasets used in our experiments. Reported results are
averaged across 3 seeds.

##  Appendix H Comparing clusterings across networks

(a)  (b)

(c)  (d)

Figure H.1: Here we repeat the experiment described in Figure 8 for ResNet-50
architecture. Top row shows the AMI across layers for three different training
runs of ResNet-50 on entity-13 (left) and entity-13-shuffled (right). Similar
to Figure 8, we observe that the random seeds produce similar AMI profiles.
However, when measuring how aligned the actual clusters are across runs with
adjusted Rand index, we find that the ARI values are relatively low for most
layers (bottom row) — the images assigned to each cluster vary substantially
across runs. An exception is high cluster consistency found in the last few
layers of a model trained on entity-13-shuffled, where we also observe a
corresponding rise in AMI.

Figure H.2: We repeat the analysis described in Section 8 for three different
runs of a VGG-16 trained on living-17 dataset. Again, we find that the random
seeds exhibit very similar AMI profiles, but the cluster assignments between
them are highly inconsistent, with the peak consistency occurring in the early
convolutional layers.

Figure H.3: We repeat the analysis described in Section 8 for three different
runs of a ResNet-50 trained on nonliving-26 dataset. Again, we find that the
random seeds exhibit very similar AMI profiles, but the cluster assignments
between them are highly inconsistent, with the consistency being comparatively
high both at the start and at the end of the ResNet-50 network.

Figure H.4: We compare clustering consistency, measured with adjusted Rand
index, on entity-13 for every pair of layers found across two training runs of
the same model. We observe that in VGG-16 (left pane), corresponding layers
between two random seeds show relatively higher consistency compared to other
pairs of layers, though the ARI values are still low overall. However, this is
not the case for ResNet-50 (right pane), where pairs of layers from the same
‘stage’ can exhibit higher cluster alignment compared to corresponding layers.

Below we show sample images from each of the 4 clusters uncovered under the
superclass ‘ball’ of nonliving-26, using embeddings from layer stage4_block3
of ResNet-50s trained with different random initializations on the same
dataset. Each row represents one cluster, and each image visualization is
labeled with the ground-truth subclass name:

Figure H.5: Sample images from 4 clusters under ‘ball’ uncovered in
stage4_block3 of ResNet-50 random seed 1.

Figure H.6: Sample images from 4 clusters under ‘ball’ uncovered in
stage4_block3 of ResNet-50 random seed 2.

Figure H.7: Sample images from 4 clusters under ‘ball’ uncovered in
stage4_block3 of ResNet-50 random seed 3.


* Herrmann  _et al._ (2013) M. Herrmann, K. Matthies, H. Schwetlick, and J. Zimmer, SIAM J. Math. Anal. 45, 2625 (2013).
  * Efendiev and Truskinovsky (2010) Y. R. Efendiev and L. Truskinovsky, Cont. Mech. Thermodynam. 22, 679 (2010).
  * Benedito and Giordano (2020) M. Benedito and S. Giordano, Phys. Lett. A 384, 126124 (2020).
  * Blake and Cherkaev (2020) C. Blake and A. Cherkaev, Nonlin. Wave Dyn. Mater. Struct. 122, 77 (2020).
  * Truskinovsky and Vainchtein (2006) L. Truskinovsky and A. Vainchtein, Cont. Mech. Thermodyn. 18, 1 (2006).
  * Christov  _et al._ (2007) C. I. Christov, G. A. Maugin, and A. V. Porubov, C. R. Mécanique 335, 521 (2007).
  * Kunin (2012) I. A. Kunin,  _Elastic media with microstructure I: one-dimensional models_ , Vol. 26 (Springer Science & Business Media, 2012).
  * Charlotte and Truskinovsky (2008) M. Charlotte and L. Truskinovsky, Cont. Mech. Thermodynam. 20, 133 (2008).
  * Charlotte and Truskinovsky (2012) M. Charlotte and L. Truskinovsky, J. Mech. Phys. Solids 60, 1508 (2012).
  * Kamchatnov (2019) A. M. Kamchatnov, Phys. Rev. E 99, 012203 (2019).
  * Chong and Kevrekidis (2018) C. Chong and P. G. Kevrekidis, in  _Coherent Structures in Granular Crystals_ (Springer, 2018) pp. 13–22.
  * Benzoni-Gavage  _et al._ (2021) S. Benzoni-Gavage, C. Mietka, and L. M. Rodrigues, Nonlinearity 34, 578 (2021).
  * Blanc  _et al._ (2002) X. Blanc, C. Le Bris, and P.-L. Lions, Arch. Rat. Mech. Anal. 164, 341 (2002).
  * LeFloch (2002) P. G. LeFloch,  _Hyperbolic Systems of Conservation Laws: The theory of classical and nonclassical shock waves_ (Springer Science & Business Media, 2002).
  * Truskinovskii (1982) L. M. Truskinovskii, in  _Dokl. Akad. Nauk_ , Vol. 265 (Russian Academy of Sciences, 1982) pp. 306–310.
  * Abeyaratne and Knowles (2006) R. Abeyaratne and J. K. Knowles,  _Evolution of phase transitions: a continuum theory_ (Cambridge University Press, 2006).
  * Eshelby (1970) J. D. Eshelby, Inelastic behavior of solids (eds. MF Kanninen et al) McGraw-Hill, New York , 77 (1970).
  * Knowles (1979) J. K. Knowles, J. Elast. 9, 131 (1979).
  * Heidug and Lehner (1985) W. Heidug and F. K. Lehner, Pure Appl. Geophys. 123, 91 (1985).
  * Truskinovsky (2002) L. Truskinovsky, in  _Dynamics of Crystal Surfaces and Interfaces_ (Springer, 2002) pp. 185–197.
  * Rayleigh (1877) J. W. S. B. Rayleigh,  _The theory of sound_ , Vol. 1 (Macmillan, 1877).
  * Benjamin  _et al._ (1972) T. B. Benjamin, J. L. Bona, and J. J. Mahony, Phil. Trans. Royal Soc. London A 272, 47 (1972).
  * Ostrovskii and Sutin (1977) L. A. Ostrovskii and A. M. Sutin, J. Appl. Math. Mech. 41, 543 (1977).
  * Collins (1981) M. A. Collins, Chem. Phys. Lett. 77, 342 (1981).
  * Rosenau (1986) P. Rosenau, Phys. Lett. A 118, 222 (1986).
  * Kevrekidis  _et al._ (2002) P. G. Kevrekidis, I. G. Kevrekidis, A. R. Bishop, and E. S. Titi, Phys. Rev. E 65, 046613 (2002).
  * Feng  _et al._ (2004) B.-F. Feng, Y. Doi, and T. Kawahara, J. Phys. Soc. Japan 73, 2100 (2004).
  * Gelfand and Fomin (1963) I. M. Gelfand and S. V. Fomin,  _Calculus of Variations_ (Prentice Hall, Englewood Cliffs, NJ, 1963).
  * Theil and Levitas (2000) F. Theil and V. I. Levitas, Mathematics and Mechanics of Solids 5, 337 (2000).
  * Gorbushin  _et al._ (2020) N. Gorbushin, G. Mishuris, and L. Truskinovsky, Physical Review Letters 125, 195502 (2020).
  * Brillouin (1953) L. Brillouin,  _Wave propagation in periodic structures: electric filters and crystal lattices_ , Vol. 2 (Dover publications, 1953).
  * Marder and Gross (1995) M. Marder and S. Gross, J. Mech. Phys. Solids 43, 1 (1995).
  * Wang and Dai (2018) X. Wang and W. Dai, Comp. Appl. Math. 37, 6560 (2018).
  * Gurevich and Pitaevskii (1973) A. V. Gurevich and L. P. Pitaevskii, Zh. Eksp. Teor. Fiz. 65, 590 (1973).
  * Congy  _et al._ (2019) T. Congy, G. A. El, M. A. Hoefer, and M. Shearer, Stud. Appl. Math. 142, 241 (2019).
  * Ablowitz and Baldwin (2013) M. J. Ablowitz and D. E. Baldwin, Phys. Rev. E 87, 022906 (2013).
  * Holian (1995) B. L. Holian, Shock waves 5, 149 (1995).
  * Zhakhovskii  _et al._ (1999) V. V. Zhakhovskii, S. V. Zybin, K. Nishihara, and S. I. Anisimov, Phys. Rev. Lett. 83, 1175 (1999).
  * Stoltz (2005) G. Stoltz, Nonlinearity 18, 1967 (2005).
  * Noble (1958) B. Noble,  _Methods based on the Wiener-Hopf technique for the solution of partial differential equations_ , Vol. 332 (Taylor & Francis US, 1958).
  * Slepyan (1982) L. I. Slepyan, Mech. Solids 17, 101 (1982).
  * Sprenger and Hoefer (2020) P. Sprenger and M. A. Hoefer, Nonlinearity 33, 3268 (2020).
  * Gavrilyuk  _et al._ (2020) S. Gavrilyuk, B. Nkonga, K.-M. Shyue, and L. Truskinovsky, Nonlinearity 33, 5477 (2020).
  * Truskinovsky and Vainchtein (2010) L. Truskinovsky and A. Vainchtein, Cont. Mech. Thermodyn. 22, 485 (2010).
  * Ngan and Truskinovsky (2002) S.-C. Ngan and L. Truskinovsky, J. Mech. Phys. Solids 50, 1193 (2002).