DINO |  | $63$ | $43$ | $56$ | $59$ | $\underline{74}$ | $\underline{65}$ | $\underline{59}$ | $34$ | $45$ | $46$ | ${63}$ | $\mathbf{\underline{42}}$ | $19$ | $13$ | $\mathbf{\underline{90}}$ | $\mathbf{\underline{43}}$ | $47$ | ${20}$ | $12$ | $10$ | $\underline{37}$ | $\mathbf{\underline{58}}$ | $\underline{65}$ | $60$ | $61$ | $3$
MAE (CLS) |  | $28$ | $16$ | $30$ | $27$ | $49$ | $48$ | $25$ | $11$ | $27$ | $22$ | $33$ | $24$ | $10$ | $6$ | $51$ | $21$ | $19$ | $9$ | $9$ | $7$ | $32$ | $34$ | $53$ | $62$ | $53$ | $2$
MAE (avg) |  | $28$ | $15$ | $30$ | $30$ | $44$ | $43$ | $23$ | $12$ | $24$ | $23$ | $34$ | $24$ | $8$ | $6$ | $59$ | $17$ | $17$ | $8$ | $12$ | $9$ | $30$ | $42$ | $53$ | $52$ | $56$ | $2$
MoCo-v3 | ✓ | $73$ | ${53}$ | $\mathbf{\underline{70}}$ | $\mathbf{\underline{69}}$ | $67$ | $61$ | $55$ | ${44}$ | ${48}$ | $59$ | $61$ | $\mathbf{\underline{42}}$ | $\mathbf{\underline{21}}$ | ${29}$ | $71$ | $29$ | ${49}$ | ${19}$ | $12$ | $\underline{12}$ | $26$ | $48$ | $55$ | $61$ | $59$ | $3$
DINO | ✓ | $\mathbf{\underline{75}}$ | ${53}$ | $63$ | $62$ | $68$ | $62$ | $56$ | $43$ | ${49}$ | $56$ | ${63}$ | $41$ | $\mathbf{\underline{21}}$ | ${29}$ | $67$ | $26$ | $47$ | $19$ | $11$ | $11$ | $26$ | $45$ | $54$ | $59$ | $57$ | $3$
MAE (avg) | ✓ | $\mathbf{\underline{74}}$ | $\mathbf{\underline{54}}$ | $64$ | $63$ | $68$ | $63$ | $57$ | $\mathbf{\underline{45}}$ | $\mathbf{\underline{50}}$ | $62$ | $\mathbf{\underline{65}}$ | $\mathbf{\underline{42}}$ | $20$ | $\mathbf{\underline{31}}$ | $76$ | $31$ | $\mathbf{\underline{51}}$ | $\mathbf{\underline{20}}$ | $\underline{13}$ | $12$ | $29$ | $49$ | $58$ | $63$ | $61$ | $\underline{5}$

Table 13: AMI score (%) using Affinity Propagation.

|  | In-domain | Domain-shift | Near-OOD | Fine-grained | Far-OOD
---|---|---|---|---|---|---
Encoder | FT | IN1k | INv2 | C10 | C100 | IN9 | 9-FG | 9-MR | IN-R | IN-S | IN-O | LSU | P365 | Air | Cars | F102 | Bio | Birds | iNat | CelA | UTKF | BHis | DTD | ESAT | MNST | Fash | SVHN
Raw image |  | $2$ | $1$ | $11$ | $10$ | $6$ | $6$ | $1$ | $3$ | $10$ | $6$ | $7$ | $3$ | $3$ | $2$ | $16$ | $8$ | $2$ | $0$ | $4$ | $3$ | $16$ | $5$ | $26$ | $43$ | $43$ | $4$
RN50 — Rand. |  | $2$ | $1$ | $7$ | $7$ | $5$ | $7$ | $2$ | $2$ | $10$ | $5$ | $5$ | $3$ | $2$ | $1$ | $14$ | $11$ | $2$ | $0$ | $3$ | $2$ | $18$ | $5$ | $24$ | $40$ | $40$ | $1$
X-Ent. |  | $\underline{67}$ | $\underline{44}$ | $54$ | $\underline{52}$ | $70$ | $71$ | $60$ | $34$ | $42$ | $56$ | $57$ | $\underline{42}$ | $18$ | $\underline{27}$ | $64$ | $31$ | $\underline{48}$ | $\underline{20}$ | $13$ | $12$ | $28$ | $49$ | $63$ | $63$ | $65$ | ${7}$
MoCo-v3 |  | $50$ | $30$ | $54$ | $51$ | $68$ | $60$ | $46$ | $27$ | $38$ | $39$ | $53$ | $35$ | $\mathbf{\underline{22}}$ | $12$ | ${\underline{80}}$ | $33$ | $33$ | $11$ | $\underline{14}$ | ${\underline{13}}$ | $32$ | $\underline{52}$ | $64$ | $\mathbf{\underline{76}}$ | $61$ | $\mathbf{\underline{15}}$
DINO |  | $48$ | $28$ | $43$ | $42$ | $68$ | $63$ | $41$ | $19$ | $29$ | $36$ | $53$ | $37$ | $19$ | $14$ | ${\underline{80}}$ | $\underline{40}$ | $24$ | $10$ | $\underline{14}$ | $\mathbf{\underline{13}}$ | $\mathbf{\underline{45}}$ | $52$ | $70$ | $62$ | $59$ | $4$
VICReg |  | $49$ | $29$ | $45$ | $45$ | $70$ | $64$ | $42$ | $21$ | $32$ | $37$ | $54$ | $35$ | $16$ | $13$ | ${\underline{80}}$ | $37$ | $25$ | $10$ | $\underline{14}$ | $\mathbf{\underline{13}}$ | $38$ | $\underline{52}$ | ${\underline{70}}$ | $66$ | $55$ | $6$
MoCo-v3 | ✓ | $65$ | $\underline{44}$ | $\underline{57}$ | $\underline{52}$ | $79$ | $70$ | ${\underline{66}}$ | $\underline{35}$ | $\underline{44}$ | $61$ | $\underline{58}$ | $\underline{42}$ | $19$ | $\underline{27}$ | $61$ | $26$ | $\underline{48}$ | $19$ | $12$ | $12$ | $23$ | $50$ | $60$ | $51$ | $\underline{65}$ | ${8}$
DINO | ✓ | $66$ | $\underline{44}$ | $55$ | $\underline{52}$ | ${\underline{79}}$ | ${\underline{73}}$ | $64$ | $34$ | $43$ | ${\underline{61}}$ | $58$ | $41$ | $18$ | $26$ | $61$ | $26$ | $\underline{47}$ | $18$ | $13$ | $12$ | $24$ | $48$ | $60$ | $61$ | $60$ | $6$
VICReg | ✓ | $65$ | $\underline{44}$ | $54$ | $51$ | ${\underline{79}}$ | $71$ | $64$ | $33$ | $41$ | $61$ | $57$ | $41$ | $18$ | $\underline{27}$ | $63$ | $27$ | $47$ | $18$ | $13$ | ${\underline{13}}$ | $24$ | $51$ | $60$ | $58$ | $64$ | $7$
ViT-B — Rand. |  | $3$ | $1$ | $9$ | $9$ | $6$ | $5$ | $1$ | $2$ | $10$ | $7$ | $9$ | $4$ | $3$ | $2$ | $18$ | $17$ | $3$ | $0$ | $3$ | $3$ | $18$ | $8$ | $31$ | $20$ | $20$ | $0$
X-Ent. |  | $71$ | $50$ | ${72}$ | ${65}$ | $66$ | $64$ | $53$ | $39$ | $44$ | $\mathbf{\underline{64}}$ | $62$ | $42$ | $20$ | $30$ | $68$ | $31$ | $45$ | $19$ | $13$ | $11$ | $26$ | $45$ | $59$ | $71$ | $\mathbf{\underline{68}}$ | $4$
MoCo-v3 |  | $57$ | $36$ | $68$ | $63$ | $75$ | $68$ | $46$ | $26$ | $37$ | $36$ | $63$ | $42$ | $15$ | $12$ | $78$ | ${41}$ | $34$ | $14$ | ${15}$ | $\mathbf{\underline{13}}$ | $41$ | ${55}$ | $69$ | ${\underline{72}}$ | $20$ | $4$
DINO |  | $62$ | $41$ | $63$ | $62$ | $\mathbf{\underline{84}}$ | $\mathbf{\underline{75}}$ | $\mathbf{\underline{66}}$ | $33$ | $43$ | $44$ | $62$ | $\mathbf{\underline{44}}$ | $\mathbf{\underline{22}}$ | $15$ | $\mathbf{\underline{87}}$ | $\mathbf{\underline{44}}$ | $45$ | $19$ | $\mathbf{\underline{16}}$ | $12$ | ${\underline{44}}$ | $\mathbf{\underline{60}}$ | $\mathbf{\underline{74}}$ | $67$ | $65$ | $2$
MAE (CLS) |  | $18$ | $10$ | $26$ | $23$ | $40$ | $38$ | $19$ | $9$ | $24$ | $18$ | $28$ | $15$ | $10$ | $4$ | $45$ | $17$ | $9$ | $2$ | $6$ | $5$ | $33$ | $30$ | $48$ | $47$ | $43$ | $\underline{6}$
MAE (avg) |  | $23$ | $12$ | $30$ | $25$ | $47$ | $45$ | $17$ | $9$ | $23$ | $18$ | $35$ | $23$ | $8$ | $6$ | $52$ | $18$ | $13$ | $3$ | $8$ | $9$ | $36$ | $35$ | $58$ | $64$ | $11$ | $1$
MoCo-v3 | ✓ | $73$ | ${52}$ | $\mathbf{\underline{74}}$ | $\mathbf{\underline{68}}$ | $68$ | $61$ | $55$ | ${44}$ | ${48}$ | $59$ | $61$ | ${44}$ | $21$ | $32$ | $70$ | $32$ | ${49}$ | ${22}$ | $14$ | $12$ | $26$ | $50$ | $57$ | $64$ | ${67}$ | $3$
DINO | ✓ | $\mathbf{\underline{74}}$ | ${52}$ | $68$ | $62$ | $70$ | $61$ | $56$ | $43$ | ${48}$ | $56$ | $\mathbf{\underline{63}}$ | $43$ | ${\underline{21}}$ | ${33}$ | $67$ | $29$ | $48$ | $21$ | $14$ | $11$ | $26$ | $45$ | $57$ | $63$ | $65$ | $3$
MAE (avg) | ✓ | $\mathbf{\underline{73}}$ | $\mathbf{\underline{53}}$ | $68$ | $62$ | $70$ | $63$ | $57$ | $\mathbf{\underline{45}}$ | $\mathbf{\underline{50}}$ | ${62}$ | $\mathbf{\underline{63}}$ | ${44}$ | $19$ | $\mathbf{\underline{34}}$ | $76$ | $33$ | $\mathbf{\underline{51}}$ | $\mathbf{\underline{23}}$ | ${15}$ | $12$ | $28$ | $49$ | $63$ | $68$ | $64$ | $4$

Table 14: AMI score (%) using HDBSCAN. In this analysis, we evaluate the
HDBSCAN output by counting all the samples labelled as “noise” as being
combined together into their own cluster. Such analysis is contrary to the
intended usage of HDBSCAN and will negatively impact the measured performance
of HDBSCAN, but is the fairest comparison available. For an evaluation of
HDBSCAN excluding rejected samples, see Table 15.

|  | In-domain | Domain-shift | Near-OOD | Fine-grained | Far-OOD
---|---|---|---|---|---|---
Encoder | FT | IN1k | INv2 | C10 | C100 | IN9 | 9-FG | 9-MR | IN-R | IN-S | IN-O | LSU | P365 | Air | Cars | F102 | Bio | Birds | iNat | CelA | UTKF | BHis | DTD | ESAT | MNST | Fash | SVHN
Raw image |  | $1$ | $1$ | $7$ | $6$ | $6$ | $5$ | $1$ | $1$ | $6$ | $7$ | $5$ | $1$ | $3$ | $1$ | $9$ | $1$ | $1$ | $0$ | $1$ | $1$ | $12$ | $4$ | $21$ | $65$ | $38$ | $3$
RN50 — Rand. |  | $0$ | $0$ | $4$ | $4$ | $3$ | $5$ | $1$ | $1$ | $5$ | $2$ | $3$ | $1$ | $2$ | $0$ | $8$ | $2$ | $1$ | $0$ | $0$ | $0$ | $11$ | $3$ | $17$ | $36$ | $27$ | $1$
X-Ent. |  | $\underline{64}$ | $38$ | $37$ | $\underline{43}$ | $\underline{50}$ | $48$ | ${\underline{47}}$ | $\underline{32}$ | $\underline{30}$ | $52$ | $39$ | $\underline{25}$ | $10$ | $13$ | $56$ | $6$ | $28$ | $\underline{8}$ | $3$ | $\mathbf{\underline{3}}$ | $20$ | $42$ | $58$ | $70$ | $49$ | ${6}$
MoCo-v3 |  | $34$ | $18$ | $37$ | $38$ | $46$ | $42$ | $35$ | $22$ | $25$ | $30$ | $39$ | $20$ | $14$ | $7$ | $76$ | $8$ | $26$ | $5$ | ${4}$ | $\mathbf{\underline{2}}$ | $23$ | $47$ | $64$ | $\mathbf{\underline{77}}$ | $45$ | $\mathbf{\underline{11}}$
DINO |  | $29$ | $15$ | $28$ | $28$ | $49$ | $47$ | $33$ | $13$ | $19$ | $26$ | $34$ | $20$ | $13$ | $7$ | ${\underline{78}}$ | ${\underline{9}}$ | $16$ | $4$ | $\mathbf{\underline{5}}$ | $\mathbf{\underline{2}}$ | $\mathbf{\underline{37}}$ | $43$ | $\mathbf{\underline{72}}$ | $68$ | $44$ | $4$
VICReg |  | $32$ | $18$ | $27$ | $32$ | $42$ | $45$ | $32$ | $16$ | $22$ | $29$ | $37$ | $19$ | $12$ | $8$ | $77$ | $\underline{9}$ | $13$ | $4$ | $\mathbf{\underline{4}}$ | $\mathbf{\underline{3}}$ | $28$ | $\underline{48}$ | $\mathbf{\underline{72}}$ | $72$ | ${51}$ | ${5}$
MoCo-v3 | ✓ | $61$ | $\underline{39}$ | $\underline{43}$ | $42$ | $\underline{50}$ | ${\underline{49}}$ | $39$ | $31$ | $\underline{30}$ | $62$ | $\underline{51}$ | $24$ | $15$ | $\underline{14}$ | $51$ | $5$ | $27$ | $7$ | $2$ | $\mathbf{\underline{3}}$ | $16$ | $45$ | $52$ | $59$ | $44$ | ${5}$
DINO | ✓ | $61$ | $38$ | $35$ | $39$ | $49$ | $46$ | $38$ | $28$ | $\underline{30}$ | ${\underline{63}}$ | $36$ | $23$ | ${\underline{16}}$ | $12$ | $54$ | $5$ | $29$ | $6$ | $3$ | $\mathbf{\underline{3}}$ | $17$ | $45$ | $53$ | $66$ | $47$ | $4$
VICReg | ✓ | $60$ | $37$ | $38$ | $40$ | $\underline{50}$ | $45$ | $44$ | $28$ | $29$ | $61$ | $43$ | $22$ | $10$ | $13$ | $58$ | $5$ | $\underline{31}$ | $6$ | $3$ | $2$ | $15$ | $45$ | $57$ | $54$ | $\mathbf{\underline{52}}$ | $5$
ViT-B — Rand. |  | $1$ | $0$ | $7$ | $5$ | $5$ | $4$ | $1$ | $1$ | $6$ | $5$ | $7$ | $1$ | $2$ | $1$ | $11$ | $4$ | $1$ | $0$ | $1$ | $1$ | $15$ | $6$ | $26$ | $15$ | $18$ | $0$
X-Ent. |  | $72$ | $51$ | $70$ | ${54}$ | $51$ | ${49}$ | $43$ | $37$ | $34$ | $\mathbf{\underline{64}}$ | $53$ | ${30}$ | $14$ | $13$ | $64$ | $7$ | $28$ | $9$ | $2$ | $\mathbf{\underline{2}}$ | $19$ | $41$ | $46$ | $71$ | $46$ | $\underline{3}$
MoCo-v3 |  | $49$ | $27$ | $62$ | $50$ | $49$ | $48$ | $35$ | $23$ | $27$ | $28$ | $46$ | $26$ | $11$ | $6$ | $75$ | ${10}$ | $22$ | $7$ | $\mathbf{\underline{4}}$ | $\mathbf{\underline{3}}$ | $32$ | ${52}$ | $56$ | ${\underline{76}}$ | $48$ | $\underline{3}$
DINO |  | $56$ | $34$ | $56$ | $51$ | $\mathbf{\underline{52}}$ | $\mathbf{\underline{50}}$ | $\mathbf{\underline{49}}$ | $28$ | $31$ | $36$ | ${59}$ | $26$ | $15$ | $8$ | $\mathbf{\underline{84}}$ | $\mathbf{\underline{11}}$ | $34$ | $9$ | $\mathbf{\underline{4}}$ | $\mathbf{\underline{3}}$ | ${\underline{36}}$ | $\mathbf{\underline{53}}$ | $\underline{67}$ | $74$ | $45$ | $2$
MAE (CLS) |  | $2$ | $1$ | $4$ | $5$ | $9$ | $7$ | $1$ | $1$ | $9$ | $4$ | $6$ | $2$ | $3$ | $2$ | $24$ | $2$ | $5$ | $3$ | $4$ | $1$ | $11$ | $11$ | $32$ | $40$ | $31$ | $0$
MAE (avg) |  | $11$ | $6$ | $18$ | $16$ | $33$ | $38$ | $16$ | $6$ | $16$ | $12$ | $26$ | $10$ | $6$ | $4$ | $44$ | $5$ | $5$ | $1$ | $3$ | $\mathbf{\underline{2}}$ | $29$ | $29$ | $45$ | $68$ | $46$ | $1$
MoCo-v3 | ✓ | $75$ | $55$ | $\mathbf{\underline{76}}$ | $\mathbf{\underline{61}}$ | $\mathbf{\underline{52}}$ | ${49}$ | $47$ | $\mathbf{\underline{43}}$ | $\mathbf{\underline{39}}$ | $56$ | $\mathbf{\underline{63}}$ | $\mathbf{\underline{31}}$ | $15$ | $\mathbf{\underline{22}}$ | $66$ | $6$ | ${36}$ | ${10}$ | $3$ | $\mathbf{\underline{2}}$ | $17$ | $44$ | $44$ | $71$ | $45$ | $2$
DINO | ✓ | $\mathbf{\underline{77}}$ | ${55}$ | $64$ | $52$ | $51$ | ${49}$ | $45$ | $41$ | $39$ | $53$ | $48$ | ${30}$ | $\mathbf{\underline{18}}$ | $17$ | $64$ | $5$ | $32$ | ${10}$ | $3$ | $2$ | $18$ | $42$ | $46$ | $61$ | ${\underline{51}}$ | $2$
MAE (avg) | ✓ | ${76}$ | $\mathbf{\underline{56}}$ | ${75}$ | $53$ | $\mathbf{\underline{52}}$ | ${49}$ | ${48}$ | $\mathbf{\underline{43}}$ | $\mathbf{\underline{39}}$ | $61$ | $48$ | $30$ | ${16}$ | ${21}$ | $71$ | $7$ | $\mathbf{\underline{38}}$ | $\mathbf{\underline{11}}$ | $4$ | $\mathbf{\underline{3}}$ | $21$ | $43$ | $47$ | $75$ | $49$ | $\underline{3}$

Table 15: AMI score (%) using HDBSCAN, excluding samples rejected by the
clusterer as background noise. Caution: These scores are highly inflated
because HDBSCAN will reject the samples which are hardest to cluster, and is
only being evaluated here on the samples it was confident in clustering—we
found HDBSCAN frequently rejected half the samples in a dataset. For the rate
at which samples were accepted by HDBSCAN, see Table 16.

|  | In-domain | Domain-shift | Near-OOD | Fine-grained | Far-OOD
---|---|---|---|---|---|---
Encoder | FT | IN1k | INv2 | C10 | C100 | IN9 | 9-FG | 9-MR | IN-R | IN-S | IN-O | LSU | P365 | Air | Cars | F102 | Bio | Birds | iNat | CelA | UTKF | BHis | DTD | ESAT | MNST | Fash | SVHN
Raw image |  | $3$ | $2$ | $12$ | $15$ | $10$ | $8$ | $1$ | $3$ | $16$ | $10$ | $10$ | $3$ | $5$ | $2$ | $20$ | $3$ | $3$ | $1$ | $2$ | $1$ | $18$ | $6$ | $29$ | $75$ | $49$ | $6$
RN50 — Rand. |  | $1$ | $1$ | $6$ | $6$ | $4$ | $6$ | $2$ | $1$ | $10$ | $2$ | $4$ | $1$ | $2$ | $1$ | $13$ | $3$ | $2$ | $0$ | $1$ | $1$ | $14$ | $4$ | $21$ | $50$ | $34$ | $1$
X-Ent. |  | $\underline{82}$ | $61$ | $55$ | $62$ | $55$ | $54$ | ${\underline{55}}$ | $\underline{53}$ | $57$ | $73$ | $57$ | $\underline{47}$ | $19$ | $29$ | $76$ | $12$ | $54$ | $\underline{25}$ | $5$ | $\mathbf{\underline{4}}$ | $33$ | $59$ | $67$ | $75$ | $60$ | ${10}$
MoCo-v3 |  | $67$ | $41$ | $56$ | $\underline{64}$ | ${57}$ | $55$ | $49$ | $45$ | $52$ | $53$ | $54$ | $43$ | $\underline{26}$ | $16$ | $89$ | $15$ | $39$ | $16$ | $\underline{7}$ | $\mathbf{\underline{4}}$ | $38$ | $60$ | $74$ | ${\underline{82}}$ | $58$ | $\mathbf{\underline{21}}$
DINO |  | $62$ | $38$ | $46$ | $54$ | $\mathbf{\underline{62}}$ | $\mathbf{\underline{60}}$ | $47$ | $27$ | $39$ | $51$ | $49$ | $46$ | $\underline{26}$ | $12$ | $\underline{90}$ | ${\underline{17}}$ | $30$ | $13$ | ${\underline{8}}$ | $3$ | ${\underline{51}}$ | ${\underline{64}}$ | $\mathbf{\underline{78}}$ | $74$ | $57$ | $7$
VICReg |  | $65$ | $40$ | $46$ | $57$ | $55$ | ${57}$ | $46$ | $31$ | $45$ | $51$ | $52$ | $43$ | $21$ | $14$ | $89$ | $17$ | $28$ | $15$ | ${\underline{8}}$ | $\mathbf{\underline{4}}$ | $42$ | $\underline{64}$ | ${76}$ | $78$ | ${\underline{64}}$ | ${11}$
MoCo-v3 | ✓ | $\underline{82}$ | $\underline{62}$ | $\underline{57}$ | $63$ | $55$ | $55$ | $50$ | $51$ | $\underline{59}$ | $\mathbf{\underline{80}}$ | $\underline{64}$ | $47$ | $25$ | $\underline{29}$ | $72$ | $9$ | $55$ | $23$ | $4$ | ${\underline{4}}$ | $26$ | $59$ | $67$ | $70$ | $57$ | ${11}$
DINO | ✓ | $\underline{82}$ | $\underline{62}$ | $54$ | $62$ | $55$ | $54$ | $50$ | $50$ | $56$ | $\mathbf{\underline{80}}$ | $54$ | $46$ | $\underline{26}$ | $27$ | $74$ | $9$ | $\underline{55}$ | $22$ | $5$ | $3$ | $27$ | $58$ | $67$ | $74$ | $60$ | $8$
VICReg | ✓ | $\underline{82}$ | $59$ | $54$ | $61$ | $55$ | $53$ | $55$ | $49$ | $53$ | $79$ | $56$ | $45$ | $17$ | $26$ | $77$ | $9$ | $\underline{55}$ | $22$ | $5$ | $3$ | $26$ | $61$ | $70$ | $65$ | $63$ | $9$
ViT-B — Rand. |  | $2$ | $1$ | $10$ | $8$ | $7$ | $6$ | $1$ | $2$ | $14$ | $7$ | $9$ | $3$ | $3$ | $2$ | $17$ | $6$ | $2$ | $0$ | $1$ | $1$ | $18$ | $8$ | $31$ | $21$ | $24$ | $0$
X-Ent. |  | $81$ | $65$ | $76$ | ${72}$ | $53$ | $52$ | $48$ | $53$ | $58$ | $\underline{78}$ | $63$ | $46$ | $24$ | $29$ | $79$ | $12$ | $51$ | $24$ | $5$ | $3$ | $31$ | $55$ | $59$ | $78$ | $59$ | $\underline{6}$
MoCo-v3 |  | $74$ | $51$ | $72$ | $71$ | $56$ | $56$ | $45$ | $41$ | $50$ | $49$ | $58$ | ${48}$ | $19$ | $11$ | $88$ | ${17}$ | $39$ | $19$ | $8$ | $\mathbf{\underline{4}}$ | $49$ | ${64}$ | $72$ | $81$ | $59$ | $5$
DINO |  | $77$ | $56$ | $69$ | ${72}$ | ${\underline{57}}$ | ${\underline{57}}$ | $\mathbf{\underline{57}}$ | $50$ | $57$ | $56$ | ${68}$ | $\mathbf{\underline{49}}$ | ${28}$ | $12$ | $\mathbf{\underline{93}}$ | $\mathbf{\underline{20}}$ | $53$ | $25$ | ${8}$ | ${3}$ | ${51}$ | $\mathbf{\underline{65}}$ | ${\underline{76}}$ | $79$ | $58$ | $5$
MAE (CLS) |  | $38$ | $4$ | $66$ | $38$ | $29$ | $19$ | $6$ | $4$ | $57$ | $19$ | $41$ | $5$ | $6$ | $1$ | ${92}$ | ${18}$ | $18$ | $2$ | $\mathbf{\underline{22}}$ | ${3}$ | $\mathbf{\underline{62}}$ | $33$ | $71$ | $\mathbf{\underline{95}}$ | $\mathbf{\underline{73}}$ | $2$
MAE (avg) |  | $34$ | $16$ | $32$ | $33$ | $48$ | $52$ | $24$ | $14$ | $32$ | $25$ | $39$ | $30$ | $11$ | $7$ | $66$ | $9$ | $13$ | $4$ | $5$ | $3$ | $42$ | $45$ | $57$ | $76$ | $58$ | $2$
MoCo-v3 | ✓ | $84$ | ${68}$ | $\mathbf{\underline{79}}$ | $\mathbf{\underline{75}}$ | $54$ | $54$ | $52$ | $\mathbf{\underline{60}}$ | $\mathbf{\underline{64}}$ | $74$ | $\mathbf{\underline{69}}$ | ${48}$ | $\mathbf{\underline{29}}$ | $\mathbf{\underline{39}}$ | $81$ | $12$ | $54$ | ${27}$ | $6$ | $\mathbf{\underline{4}}$ | $32$ | $58$ | $62$ | $79$ | $59$ | $4$
DINO | ✓ | $\mathbf{\underline{84}}$ | $\mathbf{\underline{69}}$ | $73$ | $71$ | $53$ | $53$ | $50$ | $57$ | $63$ | $72$ | $62$ | $46$ | $27$ | $33$ | $81$ | $10$ | ${56}$ | $26$ | $6$ | $3$ | $31$ | $57$ | $60$ | $71$ | $62$ | $4$
MAE (avg) | ✓ | $\mathbf{\underline{84}}$ | ${68}$ | ${77}$ | $69$ | $54$ | $53$ | $52$ | ${59}$ | $\mathbf{\underline{64}}$ | $75$ | $62$ | $47$ | $24$ | ${37}$ | $87$ | $12$ | $\mathbf{\underline{57}}$ | $\mathbf{\underline{29}}$ | $7$ | ${4}$ | $36$ | $57$ | $62$ | $80$ | $61$ | $\underline{6}$

Table 16: Fraction of samples clustered by HDBSCAN. We indicate the fraction
of samples (%) which were placed into a cluster by HDBSCAN—the remaining
samples were rejected and placed in the “noise” category. Since every sample
in each of the datasets is labelled, such rejections are likely incorrect,
sampling a larger fraction of samples clustered is likely to indicate a better
clustering attempt. When dealing with curated datasets, we postulate it is
only plausible that a minority of the samples can truly be outliers.

|  | In-domain | Domain-shift | Near-OOD | Fine-grained | Far-OOD
---|---|---|---|---|---|---
Encoder | FT | IN1k | INv2 | C10 | C100 | IN9 | 9-FG | 9-MR | IN-R | IN-S | IN-O | LSU | P365 | Air | Cars | F102 | Bio | Birds | iNat | CelA | UTKF | BHis | DTD | ESAT | MNST | Fash | SVHN
Raw image |  | $26$ | $39$ | $37$ | $40$ | $44$ | $48$ | $45$ | $32$ | $39$ | $68$ | $42$ | $29$ | $55$ | $45$ | $41$ | $76$ | $32$ | $22$ | $76$ | ${80}$ | $59$ | $57$ | $58$ | $79$ | $67$ | $39$
RN50 — Rand. |  | $40$ | $54$ | $56$ | $57$ | $58$ | $64$ | $56$ | $50$ | $49$ | $77$ | $63$ | $41$ | $\mathbf{\underline{70}}$ | $50$ | $60$ | ${\underline{78}}$ | $49$ | $36$ | $\mathbf{\underline{79}}$ | $\mathbf{\underline{82}}$ | ${\underline{76}}$ | $71$ | $71$ | $56$ | $69$ | ${\underline{53}}$
X-Ent. |  | $\underline{80}$ | $71$ | $52$ | $\underline{64}$ | $\underline{90}$ | $84$ | $\underline{81}$ | $\underline{55}$ | $\underline{57}$ | $75$ | $58$ | $\underline{50}$ | $53$ | $46$ | $68$ | $73$ | $51$ | $\underline{40}$ | $74$ | $78$ | $53$ | $69$ | $79$ | $\underline{86}$ | ${70}$ | $33$
MoCo-v3 |  | $55$ | $53$ | $51$ | $54$ | $73$ | $66$ | $59$ | $44$ | $53$ | $62$ | $65$ | $45$ | $54$ | $46$ | $84$ | $72$ | $\underline{59}$ | $39$ | $74$ | $78$ | $54$ | $\underline{77}$ | $81$ | ${\underline{87}}$ | $64$ | $31$
DINO |  | $51$ | $50$ | $44$ | $48$ | $71$ | $70$ | $58$ | $45$ | $53$ | $57$ | $58$ | $42$ | $51$ | $52$ | ${\underline{84}}$ | $71$ | $50$ | $36$ | $74$ | $78$ | $67$ | $66$ | ${86}$ | $84$ | $65$ | $32$
VICReg |  | $53$ | $54$ | $41$ | $51$ | $66$ | $69$ | $59$ | $47$ | $54$ | $62$ | $60$ | $42$ | $55$ | $\underline{59}$ | ${\underline{85}}$ | $72$ | $47$ | $37$ | $73$ | $76$ | $59$ | $73$ | $\mathbf{\underline{88}}$ | $86$ | $67$ | $30$
MoCo-v3 | ✓ | $77$ | $71$ | $\underline{61}$ | $62$ | $89$ | $\underline{86}$ | $70$ | $\underline{55}$ | $56$ | $81$ | $\underline{71}$ | $49$ | $57$ | $48$ | $66$ | $73$ | $49$ | $39$ | $73$ | $77$ | $54$ | $74$ | $68$ | $76$ | $60$ | $29$
DINO | ✓ | $77$ | $71$ | $49$ | $58$ | $86$ | $81$ | $66$ | $52$ | $\underline{58}$ | $\underline{82}$ | $56$ | $48$ | $58$ | $46$ | $69$ | $74$ | $51$ | $38$ | $73$ | $78$ | $55$ | $74$ | $68$ | $79$ | $64$ | $29$
VICReg | ✓ | $76$ | $\underline{72}$ | $57$ | $60$ | $89$ | $81$ | $74$ | $51$ | $\underline{57}$ | $81$ | $67$ | $48$ | $59$ | $50$ | $72$ | $76$ | $54$ | $38$ | $75$ | $79$ | $52$ | $72$ | $73$ | $72$ | $\mathbf{\underline{72}}$ | $32$
ViT-B — Rand. |  | $45$ | $58$ | $58$ | $60$ | $66$ | $66$ | $64$ | $52$ | $45$ | $77$ | $67$ | $45$ | ${\underline{61}}$ | $57$ | $64$ | $\mathbf{\underline{80}}$ | $53$ | $\mathbf{\underline{44}}$ | $\mathbf{\underline{79}}$ | ${\underline{80}}$ | $\mathbf{\underline{77}}$ | $73$ | $75$ | $57$ | $62$ | $\mathbf{\underline{56}}$
X-Ent. |  | $90$ | $85$ | $85$ | $71$ | $\mathbf{\underline{95}}$ | $\mathbf{\underline{92}}$ | $84$ | $65$ | $61$ | $\mathbf{\underline{85}}$ | $78$ | $\mathbf{\underline{62}}$ | $57$ | $44$ | $78$ | $73$ | $53$ | $\mathbf{\underline{45}}$ | $71$ | $76$ | $52$ | $73$ | $70$ | $81$ | $64$ | $33$
MoCo-v3 |  | $69$ | $62$ | $78$ | $66$ | $82$ | $82$ | $68$ | $53$ | $58$ | $63$ | $71$ | $51$ | $57$ | ${59}$ | $83$ | $71$ | $55$ | $42$ | $73$ | $76$ | $59$ | $\mathbf{\underline{80}}$ | $69$ | $86$ | $69$ | $34$
DINO |  | $74$ | $69$ | $72$ | $66$ | $87$ | $84$ | $82$ | $52$ | $59$ | $68$ | ${83}$ | $51$ | $53$ | $\mathbf{\underline{61}}$ | $\mathbf{\underline{89}}$ | $71$ | ${61}$ | $43$ | $70$ | $77$ | $65$ | $\mathbf{\underline{80}}$ | $\underline{82}$ | $\mathbf{\underline{87}}$ | $63$ | $31$
MAE (CLS) |  | $6$ | $13$ | $3$ | $10$ | $21$ | $16$ | $7$ | $15$ | $18$ | $12$ | $11$ | $14$ | $28$ | $22$ | $20$ | $40$ | $20$ | $19$ | $33$ | $37$ | $14$ | $25$ | $30$ | $31$ | $35$ | $13$
MAE (avg) |  | $36$ | $45$ | $40$ | $43$ | $57$ | $62$ | $53$ | $40$ | $52$ | $54$ | $57$ | $33$ | $59$ | $56$ | $62$ | $73$ | $42$ | $28$ | $75$ | $79$ | $64$ | $62$ | $68$ | $82$ | $68$ | $39$
MoCo-v3 | ✓ | $90$ | ${86}$ | ${93}$ | $\mathbf{\underline{77}}$ | $\mathbf{\underline{95}}$ | $88$ | ${86}$ | $67$ | $\mathbf{\underline{64}}$ | $79$ | $\mathbf{\underline{91}}$ | $61$ | $48$ | $53$ | $76$ | $73$ | ${61}$ | $44$ | $71$ | $75$ | $46$ | $74$ | $61$ | $81$ | $61$ | $31$
DINO | ✓ | $\mathbf{\underline{92}}$ | ${86}$ | $81$ | $70$ | $\mathbf{\underline{95}}$ | $90$ | $85$ | ${68}$ | $\mathbf{\underline{64}}$ | $78$ | $70$ | $\mathbf{\underline{62}}$ | $58$ | $51$ | $74$ | $73$ | $55$ | $\mathbf{\underline{45}}$ | $71$ | $76$ | $50$ | $72$ | $67$ | $75$ | ${\underline{70}}$ | $28$
MAE (avg) | ✓ | ${91}$ | $\mathbf{\underline{87}}$ | $\mathbf{\underline{97}}$ | ${72}$ | $\mathbf{\underline{95}}$ | ${90}$ | $\mathbf{\underline{89}}$ | $\mathbf{\underline{69}}$ | $\mathbf{\underline{64}}$ | ${84}$ | $67$ | $59$ | $60$ | $53$ | $79$ | $72$ | $\mathbf{\underline{63}}$ | $44$ | $71$ | $75$ | $50$ | $73$ | $65$ | $\mathbf{\underline{87}}$ | $67$ | $30$

##  Appendix I Predicted Number of Clusters

We report the predicted number of clusters for the three clusterers which do
not require a number of clusters to be provided to the clusterer.

As shown in Tables 17–19, the number of clusters predicted varies greatly. We
found HDBSCAN usually generated the largest number of clusters, and AC w/o C
generated the fewest. The number of clusters predicted was often biased toward
the average magnitude (in the order of 100), such that datasets which had
fewer GT clusters (in the order of 10) were more likely to be clustered with
more clusters than were annotated, and datasets which had more GT clusterers
(in the order of 1000) were more likely to be clustered with fewer clusters
than were annotated. However, we note that for many datasets the number of
classes is ambiguous as the GT categories are hierarchical, and the clustered
embeddings may correspond to a coarser granularity than the finest-grained
annotations, as discussed in § 4.3. Similarly, for datasets which have few
annotated classes, it may be feasible to break the data down further into
subclasses.

Table 17:  Number of clusters generated using AC w/o C. Underlined (Bold):
encoder which generated clusters with numerosity closest to the GT per dataset
(across all clusterers). Background colour scale: logarithmic from smallest
underestimate (red) to largest overestimate (blue), centered around the GT
number of clusters (white).

|  | In-domain | Domain-shift | Near-OOD | Fine-grained | Far-OOD
---|---|---|---|---|---|---
Encoder | FT | IN1k | INv2 | C10 | C100 | IN9 | 9-FG | 9-MR | IN-R | IN-S | IN-O | LSU | P365 | Air | Cars | F102 | Bio | Birds | iNat | CelA | UTKF | BHis | DTD | ESAT | MNST | Fash | SVHN
№ GT classes |  | $1000$ | $1000$ | $10$ | $100$ | $9$ | $9$ | $9$ | $200$ | $1000$ | $200$ | $10$ | $365$ | $100$ | $196$ | $102$ | $2688$ | $555$ | $10000$ | $1000$ | $101$ | $32$ | $47$ | $10$ | $10$ | $10$ | $10$
Raw image |  | $1543$ | $533$ | $411$ | $398$ | $277$ | $126$ | $344$ | $595$ | $\underline{1221}$ | $137$ | $240$ | $1199$ | $\mathbf{\underline{101}}$ | $340$ | $393$ | $316$ | $\underline{914}$ | $\mathbf{\underline{3797}}$ | $324$ | $124$ | $250$ | $159$ | $179$ | $170$ | $47$ | $227$
RN50 — Rand. |  | $475$ | $213$ | $254$ | $228$ | $142$ | $198$ | $184$ | $342$ | $513$ | $98$ | $119$ | $\mathbf{\underline{351}}$ | $119$ | $176$ | $156$ | $460$ | $290$ | $735$ | $274$ | $198$ | $111$ | $87$ | $161$ | $55$ | $144$ | $317$
X-Ent. |  | $257$ | $104$ | $54$ | $111$ | $57$ | $52$ | $56$ | $262$ | $275$ | $58$ | $15$ | $321$ | $39$ | $126$ | $79$ | $296$ | $130$ | $436$ | $294$ | $71$ | $76$ | $41$ | $51$ | $73$ | $39$ | $748$
MoCo-v3 |  | $50$ | $20$ | $\mathbf{\underline{14}}$ | $22$ | $\mathbf{\underline{11}}$ | $\mathbf{\underline{13}}$ | $\mathbf{\underline{11}}$ | $54$ | $76$ | $9$ | $4$ | $58$ | $6$ | $29$ | $18$ | $37$ | $29$ | $85$ | $46$ | $11$ | $9$ | $9$ | $12$ | $\mathbf{\underline{11}}$ | $11$ | $79$
DINO |  | $90$ | $72$ | $66$ | $89$ | $26$ | $39$ | $68$ | $145$ | $141$ | $86$ | $3$ | $160$ | $4$ | $149$ | $32$ | $124$ | $72$ | $216$ | $269$ | $81$ | $\underline{54}$ | $53$ | $20$ | $27$ | $13$ | $1227$
VICReg |  | $72$ | $43$ | $93$ | $96$ | $26$ | $22$ | $56$ | $263$ | $132$ | $73$ | $3$ | $169$ | $4$ | $184$ | $37$ | $129$ | $61$ | $154$ | $344$ | $119$ | $113$ | $37$ | $17$ | $21$ | $\mathbf{\underline{10}}$ | $548$
MoCo-v3 | ✓ | $245$ | $95$ | $74$ | $123$ | $45$ | $46$ | $44$ | $228$ | $306$ | $46$ | $20$ | $315$ | $48$ | $112$ | $\underline{81}$ | $436$ | $131$ | $426$ | $271$ | $65$ | $106$ | $43$ | $58$ | $81$ | $38$ | $938$
DINO | ✓ | $241$ | $101$ | $60$ | $114$ | $48$ | $42$ | $47$ | $251$ | $264$ | $50$ | $20$ | $311$ | $49$ | $125$ | $74$ | $463$ | $133$ | $430$ | $274$ | $91$ | $76$ | $40$ | $58$ | $75$ | $45$ | $786$
VICReg | ✓ | $232$ | $96$ | $73$ | $116$ | $47$ | $44$ | $44$ | $253$ | $326$ | $46$ | $17$ | $313$ | $49$ | $151$ | $79$ | $414$ | $138$ | $455$ | $275$ | $96$ | $89$ | $42$ | $61$ | $101$ | $37$ | $876$
ViT-B — Rand. |  | $65$ | $56$ | $29$ | $35$ | $42$ | $62$ | $64$ | $57$ | $42$ | $36$ | $35$ | $108$ | $45$ | $95$ | $46$ | $28$ | $22$ | $45$ | $33$ | $21$ | $14$ | $34$ | $\mathbf{\underline{10}}$ | $6$ | $3$ | $\mathbf{\underline{15}}$
X-Ent. |  | $287$ | $143$ | $38$ | $89$ | $56$ | $60$ | $72$ | $183$ | $317$ | $79$ | $19$ | $250$ | $34$ | $108$ | $74$ | $298$ | $143$ | $378$ | $219$ | $64$ | $76$ | $42$ | $51$ | $48$ | $33$ | $399$
MoCo-v3 |  | $164$ | $382$ | $64$ | $249$ | $48$ | $61$ | $97$ | $755$ | $355$ | $76$ | $\mathbf{\underline{9}}$ | $864$ | $9$ | $898$ | $58$ | $\underline{957}$ | $161$ | $525$ | $\underline{1652}$ | $197$ | $454$ | $117$ | $85$ | $67$ | $15$ | $4729$
DINO |  | $213$ | $138$ | $204$ | $698$ | $48$ | $70$ | $102$ | $688$ | $634$ | $320$ | $26$ | $805$ | $19$ | $622$ | $\underline{81}$ | $671$ | $183$ | $614$ | $2275$ | $447$ | $694$ | $173$ | $74$ | $103$ | $36$ | $3804$
MAE (CLS) |  | $\underline{670}$ | $340$ | $244$ | $254$ | $175$ | $130$ | $236$ | $596$ | $375$ | $\mathbf{\underline{161}}$ | $106$ | $495$ | $49$ | $161$ | $176$ | $286$ | $204$ | $548$ | $351$ | $141$ | $70$ | $125$ | $31$ | $52$ | $36$ | $57$
MAE (avg) |  | $1906$ | $\mathbf{\underline{822}}$ | $333$ | $361$ | $317$ | $154$ | $266$ | $1235$ | $459$ | $268$ | $187$ | $1063$ | $73$ | $\mathbf{\underline{199}}$ | $247$ | $311$ | $198$ | $823$ | $481$ | $\mathbf{\underline{105}}$ | $69$ | $127$ | $25$ | $44$ | $18$ | $57$
MoCo-v3 | ✓ | $280$ | $147$ | $33$ | $88$ | $50$ | $63$ | $72$ | $192$ | $307$ | $68$ | $19$ | $241$ | $17$ | $121$ | $72$ | $321$ | $118$ | $315$ | $253$ | $60$ | $66$ | $\underline{45}$ | $53$ | $63$ | $38$ | $737$
DINO | ✓ | $299$ | $138$ | $49$ | $\mathbf{\underline{100}}$ | $56$ | $61$ | $69$ | $196$ | $328$ | $65$ | $20$ | $264$ | $23$ | $119$ | $75$ | $308$ | $122$ | $339$ | $250$ | $61$ | $73$ | $44$ | $49$ | $67$ | $42$ | $755$
MAE (avg) | ✓ | $290$ | $152$ | $46$ | $98$ | $53$ | $60$ | $67$ | $\mathbf{\underline{197}}$ | $319$ | $72$ | $15$ | $263$ | $11$ | $130$ | $71$ | $244$ | $122$ | $301$ | $227$ | $64$ | $80$ | $40$ | $52$ | $55$ | $35$ | $683$

Table 18:  Number of clusters generated using Affinity Prop.

|  | In-domain | Domain-shift | Near-OOD | Fine-grained | Far-OOD
---|---|---|---|---|---|---
Encoder | FT | IN1k | INv2 | C10 | C100 | IN9 | 9-FG | 9-MR | IN-R | IN-S | IN-O | LSU | P365 | Air | Cars | F102 | Bio | Birds | iNat | CelA | UTKF | BHis | DTD | ESAT | MNST | Fash | SVHN
№ GT classes |  | $1000$ | $1000$ | $10$ | $100$ | $9$ | $9$ | $9$ | $200$ | $1000$ | $200$ | $10$ | $365$ | $100$ | $196$ | $102$ | $2688$ | $555$ | $10000$ | $1000$ | $101$ | $32$ | $47$ | $10$ | $10$ | $10$ | $10$
Raw image |  | $\underline{1151}$ | $299$ | $317$ | $317$ | $153$ | $249$ | $143$ | $572$ | $2116$ | $73$ | $111$ | $828$ | $120$ | $310$ | $220$ | $913$ | $\underline{632}$ | $1969$ | $582$ | $212$ | $137$ | $66$ | $108$ | $784$ | $315$ | $428$
RN50 — Rand. |  | $804$ | $220$ | $244$ | $227$ | $113$ | $169$ | $119$ | $443$ | $818$ | $64$ | $103$ | $646$ | $110$ | $234$ | $145$ | $562$ | $423$ | $1238$ | $424$ | $173$ | $80$ | $48$ | $68$ | $452$ | $198$ | $316$
X-Ent. |  | $227$ | $108$ | $54$ | $59$ | $47$ | $33$ | $46$ | $179$ | $293$ | $65$ | $23$ | $167$ | $33$ | $90$ | $70$ | $121$ | $78$ | $179$ | $106$ | $64$ | $61$ | $43$ | $32$ | $40$ | $17$ | $202$
MoCo-v3 |  | $232$ | $102$ | $57$ | $87$ | $44$ | $51$ | $75$ | $241$ | $437$ | $67$ | $23$ | $219$ | $29$ | $96$ | $62$ | $129$ | $106$ | $313$ | $148$ | $75$ | $64$ | $41$ | $29$ | $\underline{20}$ | $538$ | $194$
DINO |  | $210$ | $86$ | $72$ | $\underline{96}$ | $35$ | $39$ | $78$ | $\underline{205}$ | $369$ | $59$ | $19$ | $204$ | $40$ | $101$ | $58$ | $98$ | $93$ | $295$ | $140$ | $77$ | $\underline{58}$ | $51$ | $25$ | $40$ | $31$ | $210$
VICReg |  | $205$ | $81$ | $68$ | $86$ | $33$ | $39$ | $73$ | $209$ | $364$ | $56$ | $\underline{13}$ | $191$ | $30$ | $82$ | $64$ | $112$ | $96$ | $293$ | $147$ | $83$ | $60$ | $40$ | $22$ | $33$ | $977$ | $175$
MoCo-v3 | ✓ | $169$ | $61$ | $41$ | $62$ | $28$ | $31$ | $\underline{34}$ | $181$ | $278$ | $46$ | $23$ | $161$ | $48$ | $88$ | $68$ | $154$ | $79$ | $172$ | $114$ | $66$ | $73$ | $35$ | $35$ | $74$ | $17$ | $265$
DINO | ✓ | $184$ | $64$ | $52$ | $62$ | $27$ | $\underline{27}$ | $40$ | $184$ | $265$ | $49$ | $21$ | $164$ | $37$ | $87$ | $67$ | $155$ | $81$ | $179$ | $120$ | $67$ | $67$ | $39$ | $33$ | $43$ | $26$ | $238$
VICReg | ✓ | $172$ | $72$ | $54$ | $60$ | $29$ | $31$ | $39$ | $188$ | $298$ | $51$ | $23$ | $164$ | $35$ | $91$ | $64$ | $145$ | $86$ | $182$ | $110$ | $73$ | $70$ | $38$ | $31$ | $53$ | $19$ | $251$
ViT-B — Rand. |  | $616$ | $203$ | $192$ | $199$ | $108$ | $177$ | $104$ | $397$ | $1252$ | $68$ | $79$ | $\underline{520}$ | $\underline{94}$ | $\underline{174}$ | $159$ | $439$ | $308$ | $963$ | $293$ | $132$ | $96$ | $65$ | $85$ | $184$ | $79$ | $254$
X-Ent. |  | $263$ | $141$ | $24$ | $52$ | $47$ | $53$ | $67$ | $164$ | $303$ | $83$ | $19$ | $157$ | $28$ | $62$ | $65$ | $144$ | $78$ | $176$ | $101$ | $55$ | $72$ | $41$ | $33$ | $28$ | $\underline{15}$ | $171$
MoCo-v3 |  | $1256$ | $99$ | $31$ | $60$ | $37$ | $43$ | $75$ | $187$ | $460$ | $46$ | $\underline{13}$ | $221$ | $31$ | $76$ | $62$ | $107$ | $70$ | $208$ | $104$ | $66$ | $60$ | $36$ | $24$ | $24$ | $4951$ | $169$
DINO |  | $363$ | $44$ | $43$ | $81$ | $\underline{24}$ | $\underline{27}$ | $40$ | $165$ | $322$ | $46$ | $19$ | $151$ | $33$ | $60$ | $58$ | $79$ | $57$ | $166$ | $91$ | $71$ | $63$ | $37$ | $\underline{20}$ | $33$ | $17$ | $\underline{155}$
MAE (CLS) |  | $2001$ | $\underline{505}$ | $426$ | $462$ | $254$ | $256$ | $248$ | $1220$ | $3670$ | $\underline{132}$ | $159$ | $1350$ | $172$ | $390$ | $357$ | $\underline{975}$ | $943$ | $\underline{2996}$ | $\mathbf{\underline{861}}$ | $302$ | $206$ | $143$ | $160$ | $397$ | $297$ | $872$
MAE (avg) |  | $428$ | $112$ | $99$ | $127$ | $57$ | $49$ | $67$ | $404$ | $\mathbf{\underline{943}}$ | $49$ | $64$ | $231$ | $35$ | $67$ | $\mathbf{\underline{103}}$ | $168$ | $1676$ | $1060$ | $208$ | $\underline{88}$ | $\underline{58}$ | $\mathbf{\underline{47}}$ | $38$ | $43$ | $7124$ | $182$
MoCo-v3 | ✓ | $255$ | $144$ | $\underline{23}$ | $57$ | $42$ | $65$ | $73$ | $188$ | $307$ | $76$ | $19$ | $165$ | $15$ | $65$ | $68$ | $117$ | $67$ | $138$ | $88$ | $55$ | $67$ | $46$ | $35$ | $39$ | $17$ | $235$
DINO | ✓ | $281$ | $129$ | $29$ | $52$ | $46$ | $63$ | $64$ | $184$ | $333$ | $69$ | $18$ | $158$ | $19$ | $58$ | $67$ | $126$ | $65$ | $158$ | $105$ | $58$ | $67$ | $44$ | $35$ | $40$ | $16$ | $259$
MAE (avg) | ✓ | $270$ | $155$ | $29$ | $52$ | $45$ | $58$ | $65$ | $184$ | $300$ | $81$ | $18$ | $161$ | $7$ | $67$ | $63$ | $109$ | $62$ | $141$ | $96$ | $58$ | $74$ | $41$ | $26$ | $34$ | $21$ | $241$

Table 19:  Number of clusters generated using HDBSCAN.

|  | In-domain | Domain-shift | Near-OOD | Fine-grained | Far-OOD
---|---|---|---|---|---|---
Encoder | FT | IN1k | INv2 | C10 | C100 | IN9 | 9-FG | 9-MR | IN-R | IN-S | IN-O | LSU | P365 | Air | Cars | F102 | Bio | Birds | iNat | CelA | UTKF | BHis | DTD | ESAT | MNST | Fash | SVHN
№ GT classes |  | $1000$ | $1000$ | $10$ | $100$ | $9$ | $9$ | $9$ | $200$ | $1000$ | $200$ | $10$ | $365$ | $100$ | $196$ | $102$ | $2688$ | $555$ | $10000$ | $1000$ | $101$ | $32$ | $47$ | $10$ | $10$ | $10$ | $10$
Raw image |  | $882$ | $226$ | $243$ | $245$ | $107$ | $118$ | $98$ | $572$ | $1237$ | $43$ | $88$ | $694$ | $96$ | $213$ | $156$ | $6154$ | $483$ | $1490$ | $5137$ | $1569$ | $82$ | $79$ | $95$ | $168$ | $284$ | $602$
RN50 — Rand. |  | $1310$ | $333$ | $322$ | $325$ | $166$ | $148$ | $150$ | $929$ | $1586$ | $69$ | $121$ | $1066$ | $123$ | $274$ | $207$ | $6374$ | $735$ | $2529$ | $5176$ | $1547$ | $120$ | $90$ | $168$ | $274$ | $448$ | $809$
X-Ent. |  | $1181$ | $481$ | $228$ | $196$ | $227$ | $214$ | $167$ | $533$ | $1714$ | $119$ | $77$ | $740$ | $\underline{98}$ | $230$ | $180$ | $6007$ | $526$ | $1617$ | $4939$ | $1503$ | $134$ | $76$ | $80$ | $81$ | $178$ | $617$
MoCo-v3 |  | $1302$ | $337$ | $222$ | $236$ | $172$ | $165$ | $141$ | $605$ | $2002$ | $104$ | $59$ | $728$ | $114$ | $242$ | $\underline{138}$ | $5905$ | $414$ | $1685$ | $4954$ | $1515$ | $121$ | $\underline{57}$ | $56$ | $81$ | $214$ | $544$
DINO |  | $1160$ | $329$ | $241$ | $251$ | $138$ | $130$ | $118$ | $578$ | $1967$ | $81$ | $76$ | $697$ | $111$ | $167$ | $143$ | $5791$ | $456$ | $1616$ | $4873$ | $1540$ | $140$ | $72$ | $49$ | $97$ | $183$ | $618$
VICReg |  | $1224$ | $344$ | $256$ | $240$ | $179$ | $150$ | $145$ | $582$ | $2020$ | $86$ | $71$ | $724$ | $115$ | $\underline{190}$ | $158$ | $5822$ | $594$ | $1659$ | $4901$ | $1493$ | $128$ | $69$ | $33$ | $90$ | $170$ | $575$
MoCo-v3 | ✓ | $1174$ | $474$ | $177$ | $190$ | $225$ | $193$ | $188$ | $517$ | $1592$ | $132$ | $60$ | $728$ | $86$ | $209$ | $182$ | $6062$ | $\mathbf{\underline{561}}$ | $1653$ | $4830$ | $1483$ | $119$ | $62$ | $81$ | $127$ | $215$ | $545$
DINO | ✓ | $1230$ | $482$ | $234$ | $236$ | $229$ | $198$ | $193$ | $608$ | $1615$ | $133$ | $95$ | $765$ | $94$ | $231$ | $161$ | $6067$ | $575$ | $1736$ | $4880$ | $1544$ | $118$ | $71$ | $92$ | $128$ | $188$ | $533$
VICReg | ✓ | $1247$ | $483$ | $224$ | $232$ | $217$ | $211$ | $194$ | $599$ | $1621$ | $133$ | $75$ | $805$ | $114$ | $235$ | $166$ | $6305$ | $493$ | $1842$ | $4980$ | $1519$ | $115$ | $67$ | $79$ | $190$ | $170$ | $599$
ViT-B — Rand. |  | $1454$ | $368$ | $383$ | $401$ | $159$ | $161$ | $172$ | $946$ | $1570$ | $77$ | $118$ | $1116$ | $136$ | $264$ | $256$ | $6498$ | $835$ | $\underline{3061}$ | $5184$ | $1511$ | $141$ | $74$ | $208$ | $356$ | $394$ | $956$
X-Ent. |  | $1102$ | $640$ | $88$ | $210$ | $254$ | $232$ | $211$ | $463$ | $1571$ | $\underline{145}$ | $51$ | $569$ | $86$ | $233$ | $162$ | $6026$ | $519$ | $1333$ | $4778$ | $1489$ | $104$ | $65$ | $96$ | $104$ | $207$ | $573$
MoCo-v3 |  | $1145$ | $416$ | $105$ | $235$ | $196$ | $180$ | $179$ | $567$ | $1873$ | $99$ | $60$ | $685$ | $97$ | $164$ | $162$ | $5927$ | $456$ | $1592$ | $4884$ | $1469$ | $139$ | $68$ | $93$ | $85$ | $171$ | $548$
DINO |  | $1131$ | $416$ | $154$ | $224$ | $191$ | $173$ | $161$ | $583$ | $1912$ | $89$ | $40$ | $726$ | $114$ | $167$ | $150$ | $5786$ | $452$ | $1548$ | $4681$ | $1484$ | $141$ | $60$ | $57$ | $78$ | $205$ | $576$
MAE (CLS) |  | $133$ | $19$ | $\underline{19}$ | $21$ | $\underline{15}$ | $\underline{14}$ | $\underline{5}$ | $48$ | $\underline{1095}$ | $6$ | $\underline{6}$ | $31$ | $10$ | $3$ | $52$ | $\mathbf{\underline{3671}}$ | $50$ | $40$ | $\underline{2712}$ | $\underline{845}$ | $\mathbf{\underline{50}}$ | $12$ | $\underline{20}$ | $\underline{17}$ | $\underline{14}$ | $\underline{51}$
MAE (avg) |  | $\mathbf{\underline{1029}}$ | $249$ | $274$ | $285$ | $137$ | $136$ | $130$ | $584$ | $1947$ | $70$ | $76$ | $698$ | $91$ | $154$ | $195$ | $5898$ | $614$ | $1984$ | $5000$ | $1476$ | $137$ | $78$ | $130$ | $131$ | $212$ | $720$
MoCo-v3 | ✓ | $1103$ | $651$ | $54$ | $\underline{181}$ | $250$ | $239$ | $208$ | $464$ | $1446$ | $130$ | $26$ | $\underline{555}$ | $89$ | $\underline{190}$ | $172$ | $6006$ | $398$ | $1271$ | $4750$ | $1502$ | $96$ | $61$ | $91$ | $104$ | $204$ | $458$
DINO | ✓ | $1102$ | $\underline{677}$ | $99$ | $202$ | $253$ | $252$ | $215$ | $436$ | $1478$ | $128$ | $65$ | $579$ | $70$ | $212$ | $155$ | $6054$ | $464$ | $1323$ | $4670$ | $1507$ | $110$ | $64$ | $93$ | $147$ | $177$ | $514$
MAE (avg) | ✓ | $1092$ | $649$ | $34$ | $197$ | $244$ | $237$ | $198$ | $\underline{423}$ | $1514$ | $129$ | $69$ | $621$ | $70$ | $221$ | $157$ | $6074$ | $395$ | $1357$ | $4728$ | $1461$ | $124$ | $66$ | $85$ | $81$ | $183$ | $490$

##  Appendix J Silhouette Scores

Our results on the silhouette score are broadly in line with our main finding
on the AMI between clusterings and annotation targets, reported in § 4. For
both the ResNet-50 and ViT-B encoders, the supervised model has the highest
silhouette score by a large margin of 0.25–0.3, but otherwise the clustering
quality across the encoders is very similar, achieving similar silhouette
scores to each other. There are some exceptions to this, such as the
silhouette scores for MAE which are near 0, illustrating the intrinsically-
poor quality of the clusters it exhibited and hence it is not well-suited to
this task.

Despite the very low AMI scores, we observe the silhouette scores for SVHN are
generally comparable to the silhouette scores of the other datasets. We
believe this is due to the heterogeneity within the classes in SVHN, where
house-numbers can be written in different formats, colours, etc., and thus the
encoded images can be appropriately grouped together, even if the semantic
meaning of the clusters does not correspond to the identity of the digit in
the center of the image.

Between the clusterers, K-Means and AC typically achieve the highest
silhouette scores. For HDBSCAN, the silhouette scores were often significantly
negative. This is because HDBSCAN builds clusters based on transitions in
density, and the non-convex clusters that result from this can score poor
silhouette scores (a known caveat to this evaluation metric). For Affinity
Propagation, we observe silhouette scores near 0, indicating the clusters it
discovered have high overlap with each other and are of low quality,
corresponding to its poor AMI performance.

##  Appendix K Detailed Comparison of Performances Across Clustering Methods

We sought to evaluate the clusterers to see which clustering methodology
produced the best results when using a pretrained encoder. For each set of
embeddings, created by passing one of the datasets listed in Appendix H
through one of the pretrained ResNet-50 or ViT-B encoders (X-Ent., MoCo-v3,
DINO, VICReg, or MAE), we compared the results of clustering that set of
embeddings with each of the clusterers (tabulated in Table 9–14).

Figure 6: Average clusterer rank (higher is better). For each set of
embeddings we apply each clusterer, compare the AMI of their clusters, and
rank them against each other (lowest AMI $\to$ rank 1, highest AMI $\to$ rank
6). Error bars: ±1 stderr; $N\\!=\\!225$.

We compared the performance of the clustering methods by ranking each
clusterer for each combination of pretrained encoder and dataset, shown in
Figure 6. The results show that AC w/ C performs best most often
($p\\!<\\!0.05$; Wilcoxon signed-rank test versus each other clusterer).
Spectral, K-Means, AC w/o C, and AP all perform similarly. HDBSCAN frequently
performed worst ($p\\!<\\!10^{-33}$).

Table 20:  Pearson correlation coefficient between clusterers. For each pair of clustering methods, we measure the Pearson correlation coefficient (%) between the AMI each attained when clustering the embeddings of a given dataset with a given encoder. We utilize datapoints across all datasets and all encoders, including fine-tuned, randomized (untrained), and raw pixels. Bold: for a given clustering method (column), the clustering method (row) that it is most correlated with.  | K-Means | Spectral | AC w/ C | AC w/o C | Affinity Prop | HDBSCAN
---|---|---|---|---|---|---
K-Means | – | $\mathbf{97.6}$ | $\mathbf{99.0}$ | $97.4$ | $97.4$ | $\mathbf{94.8}$
Spectral | $97.6$ | – | $97.1$ | $97.2$ | $96.0$ | $94.3$
AC w/ C | $\mathbf{99.0}$ | $97.1$ | – | $97.5$ | $96.9$ | $94.5$
AC w/o C | $97.4$ | $97.2$ | $97.5$ | – | $\mathbf{97.7}$ | $93.1$
Affinity Prop | $97.4$ | $96.0$ | $96.9$ | $\mathbf{97.7}$ | – | $93.6$
HDBSCAN | $94.8$ | $94.3$ | $94.5$ | $93.1$ | $93.6$ | –
Figure 7:  Correlation of AMI between clustering methods. For each pair of
clustering methods, we show a scatter plot of the AMI each attained when
clustering the embeddings of a given dataset with a given encoder. We show all
datasets and all encoders, including fine-tuned, randomized (untrained), and
raw pixels. Along the diagonal, the distribution of AMI values is shown for
each clusterer.

We investigated the correlation between the AMI for each pair of clustering
methods, shown in Table 20 and illustrated in Figure 7. We found the
correlation between clusterers was generally high ($0.931\leq r\leq 0.990$).
The performance of HDBSCAN was less correlated with the other clusterers
($r\leq 0.948$ vs $r\geq 0.960$).

##  Appendix L Detailed Comparison Between Encoders

We computed and evaluated the Pearson correlation coefficient between the
clusterings of pairs of encoders.

Looking across model architectures (Table 21) we find the performance of SSL
encoders are typically more correlated with other SSL models of the same
architecture than with the same pretraining loss but a different architecture.

As shown in Table 22 for ResNet-50 models, and Table 23 for ViT-B models, the
performance of the fine-tuned models ([FT]) were well correlated with each
other ($r\geq 0.989$) and with the supervised trained model (X-Ent.; $r\geq
0.978$). We also observed the performance of the whole-image SSL models were
highly correlated with each other ($r\geq 0.946$, and the two read-outs of the
MAE model were strongly correlated with each other ($r=0.912$). Outside of
these blocks, correlation scores were lower. In particular, we note the
performance of FT encoders was much more correlated with the X-Entropy models
than that of their original SSL-only pretrained encoder.

Table 21:  Pearson correlation coefficient between initial encoders. For each pair of pretrained encoders (without fine-tuning), we measure the Pearson correlation coefficient (%) between the AMI each attained when clustering the embeddings of a given dataset with a given clusterer. Bold: for a given encoder (column), the other encoder (row) that it is most correlated with.  |  | ResNet-50 | ViT-B
---|---|---|---
|  Raw image |  Rand. |  X-Ent. |  MoCo-v3 |  DINO |  VICReg |  Rand. |  X-Ent. |  MoCo-v3 |  DINO |  MAE (CLS) |  MAE (avg)
Raw image | – | $\mathbf{96.1}$ | $39.8$ | $58.6$ | $54.8$ | $57.7$ | $71.0$ | $39.0$ | $48.8$ | $43.0$ | $69.6$ | $66.8$
RN50 — Rand. | $\mathbf{96.1}$ | – | $39.3$ | $58.4$ | $58.0$ | $59.6$ | $\mathbf{78.5}$ | $36.5$ | $48.1$ | $44.4$ | $72.7$ | $68.1$
X-Ent. | $39.8$ | $39.3$ | – | $89.8$ | $86.2$ | $87.8$ | $41.2$ | $\mathbf{95.2}$ | $87.5$ | $93.6$ | $74.5$ | $73.2$
MoCo-v3 | $58.6$ | $58.4$ | $89.8$ | – | $96.1$ | $97.6$ | $60.4$ | $84.5$ | $93.1$ | $94.2$ | $88.5$ | $88.6$
DINO | $54.8$ | $58.0$ | $86.2$ | $96.1$ | – | $\mathbf{99.1}$ | $67.3$ | $77.7$ | $91.1$ | $92.9$ | $90.1$ | $91.6$
VICReg | $57.7$ | $59.6$ | $87.8$ | $\mathbf{97.6}$ | $\mathbf{99.1}$ | – | $67.0$ | $80.9$ | $92.5$ | $93.5$ | $90.3$ | $\mathbf{92.1}$
ViT-B — Rand. | $71.0$ | $78.5$ | $41.2$ | $60.4$ | $67.3$ | $67.0$ | – | $40.0$ | $59.5$ | $55.7$ | $71.8$ | $74.7$
X-Ent. | $39.0$ | $36.5$ | $\mathbf{95.2}$ | $84.5$ | $77.7$ | $80.9$ | $40.0$ | – | $86.7$ | $90.4$ | $65.7$ | $67.3$
MoCo-v3 | $48.8$ | $48.1$ | $87.5$ | $93.1$ | $91.1$ | $92.5$ | $59.5$ | $86.7$ | – | $\mathbf{94.6}$ | $81.8$ | $87.1$
DINO | $43.0$ | $44.4$ | $93.6$ | $94.2$ | $92.9$ | $93.5$ | $55.7$ | $90.4$ | $\mathbf{94.6}$ | – | $79.7$ | $81.5$
MAE (CLS) | $69.6$ | $72.7$ | $74.5$ | $88.5$ | $90.1$ | $90.3$ | $71.8$ | $65.7$ | $81.8$ | $79.7$ | – | $91.2$
MAE (avg) | $66.8$ | $68.1$ | $73.2$ | $88.6$ | $91.6$ | $92.1$ | $74.7$ | $67.3$ | $87.1$ | $81.5$ | $\mathbf{91.2}$ | –
Table 22:  Pearson correlation coefficient between ResNet-50 encoders. For each pair of pretrained encoders, we measure the Pearson correlation coefficient (%) between the AMI each attained when clustering the embeddings of a given dataset with a given clusterer. [FT]: fine-tuned with cross-entropy on IN-1k. Bold: for a given encoder (column), the other encoder (row) that it is most correlated with.  |  Rand. |  X-Ent. |  MoCo-v3 |  DINO |  VICReg |  MoCo-v3 [FT] |  DINO [FT] |  VICReg [FT]
---|---|---|---|---|---|---|---|---
Rand. | – | $39.3$ | $58.4$ | $58.0$ | $59.6$ | $28.5$ | $34.1$ | $32.0$
X-Ent. | $39.3$ | – | $89.8$ | $86.2$ | $87.8$ | $97.8$ | $98.9$ | $98.4$
MoCo-v3 | $58.4$ | $89.8$ | – | $96.1$ | $97.6$ | $84.7$ | $87.0$ | $86.9$
DINO | $58.0$ | $86.2$ | $96.1$ | – | $\mathbf{99.1}$ | $80.8$ | $82.7$ | $83.3$
VICReg | $\mathbf{59.6}$ | $87.8$ | $\mathbf{97.6}$ | $\mathbf{99.1}$ | – | $82.0$ | $84.4$ | $84.5$
MoCo-v3 [FT] | $28.5$ | $97.8$ | $84.7$ | $80.8$ | $82.0$ | – | $99.2$ | $\mathbf{99.5}$
DINO [FT] | $34.1$ | $\mathbf{98.9}$ | $87.0$ | $82.7$ | $84.4$ | $99.2$ | – | $99.5$
VICReg [FT] | $32.0$ | $98.4$ | $86.9$ | $83.3$ | $84.5$ | $\mathbf{99.5}$ | $\mathbf{99.5}$ | –
Table 23:  Pearson correlation coefficient between ViT-B encoders. For each
pair of pretrained encoders, we measure the Pearson correlation coefficient
(%) between the AMI each attained when clustering the embeddings of a given
dataset with a given clusterer. [FT]: fine-tuned with cross-entropy on IN-1k.
Bold: for a given encoder (column), the other encoder (row) that it is most
correlated with.

Rand. X-Ent. MoCo-v3 DINO MAE (CLS) MAE (avg) MoCo-v3 [FT] DINO [FT] MAE (avg)
[FT] Rand. – $40.0$ $59.5$ $55.7$ $71.8$ $74.7$ $37.2$ $35.5$ $39.9$ X-Ent.
$40.0$ – $86.7$ $90.4$ $65.7$ $67.3$ $97.9$ $97.9$ $98.4$ MoCo-v3 $59.5$
$86.7$ – $\mathbf{94.6}$ $81.8$ $87.1$ $85.8$ $85.5$ $87.2$ DINO $55.7$ $90.4$
$\mathbf{94.6}$ – $79.7$ $81.5$ $88.8$ $89.1$ $90.8$ MAE (CLS) $71.8$ $65.7$
$81.8$ $79.7$ – $\mathbf{91.2}$ $62.4$ $63.1$ $65.2$ MAE (avg) $\mathbf{74.7}$
$67.3$ $87.1$ $81.5$ $\mathbf{91.2}$ – $65.4$ $65.0$ $67.9$ MoCo-v3 [FT]
$37.2$ $97.9$ $85.8$ $88.8$ $62.4$ $65.4$ – $99.2$ $98.9$ DINO [FT] $35.5$
$97.9$ $85.5$ $89.1$ $63.1$ $65.0$ $\mathbf{99.2}$ – $\mathbf{99.2}$ MAE (avg)
[FT] $39.9$ $\mathbf{98.4}$ $87.2$ $90.8$ $65.2$ $67.9$ $98.9$ $\mathbf{99.2}$
–

##  Appendix M ImageNet-9 Examples

As described in § 4.5, we used the variants of the ImageNet-9 backgrounds
challenge dataset (Xiao et al.,, 2020) to evaluate whether SSL-encoded
clusters prioritized foreground and background components of the stimulus
differently to clusters using embeddings from supervised models. In Figure 8,
we provide illustrative example stimuli from the variants of this dataset.

| OG | FG | FG${}^{\text{C}}$ | BG | MS | MR
---|---|---|---|---|---|---
Bird |  (a) |  (b) |  (c) |  (d) |  (e) |  (f)
Bird |  (g) |  (h) |  (i) |  (j) |  (k) |  (l)
Insect |  (m) |  (n) |  (o) |  (p) |  (q) |  (r)
Wheeled vehicle |  (s) |  (t) |  (u) |  (v) |  (w) |  (x)
Figure 8: Example images from the ImageNet-9 dataset. For three classes (bird,
insect, and wheeled vehicle) we show a sample from each of the variant
datasets: original images (OG), foreground only (FG), foreground removed and
replaced with black (FG${}^{\text{C}}$), background only (bounding box
replaced with background texture; BG), mixed-same (foreground overlaid on the
background of a sample of the same class; MS), and mixed-random (foreground
overlaid on the background of a random sample; MR). We note that MS places the
foreground object on an appropriate background, whereas MR places the
foreground on a background which may be out-of-context for the foreground.
ImageNet-9 labels are coarse-grained superclasses, each spanning multiple
IN-1k classes, hence images of toucan and flamingo are both labelled “bird”.

##  Appendix N ImageNet-Rendition Information Breakdown

We sought to better understand what information about the stimulus is being
captured in the clusters. By using a dataset which possesses more than one
annotation per image, we can investigate the agreement between the clusterings
and each annotation type. The ImageNet-Rendition dataset in particular has
primary annotations for the object class represented in the image (goldfish,
great white shark, cowboy hat, volcano, etc.), but also annotations for the
style of rendition (cartoon, graffiti, embroidery, origami, etc.), see Figure
9. We compute the AMI for each annotation stream, see Table 24.

| Cartoon | Embroid. | Graffiti | Origami | Painting | Sketch | Tattoo
---|---|---|---|---|---|---|---
Toucan |  (a) |  (b) |  (c) |  $\varnothing$ |  (d) |  (e) |  (f)
Flamingo |  (g) |  (h) |  (i) |  (j) |  (k) |  (l) |  (m)
Bee |  (n) |  (o) |  (p) |  (q) |  (r) |  (s) |  (t)
Mushroom |  (u) |  (v) |  (w) |  (x) |  (y) |  (z) |  $\varnothing$
Figure 9:  Example images from ImageNet-R by both class and artform style. $\varnothing$ indicates no images in that artform for this class. In our experiments, we measure the AMI between the clusterings and the labels pooled across each row (Object), each column (Artform), or using only the labels per row–column combination/cell (Both).  Table 24:  ImageNet-Rendition Breakdown. Information (AMI, %) about different aspects of images in ImageNet-R: the IN-1k object class represented, the style of rendition, and their combinations. Bold: best encoder per aspect. Underlined: best encoder per arch. Background: from median AMI (white) to max (blue) per aspect. FT: fine-tuned with x-ent. on IN-1k.  Arch. | Encoder | FT | Class | Artform | Both
---|---|---|---|---|---
RN50 | X-Ent. |  | $34$ | $19$ | $\underline{29}$
| MoCo-v3 |  | $26$ | $19$ | $23$
| DINO |  | $18$ | $\mathbf{\underline{24}}$ | $20$
| VICReg |  | $20$ | $23$ | $21$
| MoCo-v3 | ✓ | $\underline{35}$ | $18$ | $\underline{29}$
| DINO | ✓ | $34$ | $18$ | $28$
| VICReg | ✓ | $33$ | $19$ | $28$
ViT-B | X-Ent. |  | $38$ | $19$ | $32$
| MoCo-v3 |  | $26$ | $\mathbf{\underline{25}}$ | $26$
| DINO |  | $33$ | $23$ | $30$
| MAE (CLS) |  | $10$ | $16$ | $11$
| MAE (avg) |  | $10$ | $19$ | $13$
| MoCo-v3 | ✓ | $\mathbf{\underline{44}}$ | $18$ | ${36}$
| DINO | ✓ | $43$ | $18$ | $35$
| MAE (avg) | ✓ | $\mathbf{\underline{44}}$ | $18$ | $\mathbf{\underline{36}}$

Our results indicate there is generally a trade-off between the two:
embeddings which are grouped according to object class identities are not
grouped according to the artform, and vice versa. This trend is true across
all ResNet-50 encoders and supervised ViT-B, but MoCo-v3 and DINO ViT-B
embeddings can capture information about both aspects.

##  Appendix O BreakHis Information Breakdown

BreakHis (Spanhol et al.,, 2016) is a medical dataset containing images of
microscopic images of breast tumor tissue collected from 81 patients. At a
coarse level, the tumor can be malignant (cancerous) or benign (normal cells).
Within each of these categories, the dataset contains samples for four
distinct types of benign tumor and four types of malignant tumor. Images were
taken for each slide (one slide per subject) at varying zoom levels (40x,
100x, 200x, 400x).

We investigated how much information the clustered embeddings contained about
each of these labels, shown in Table 25. We found that SSL pretrained encoders
were much better at encoding the medically relevant information about the
tumor’s malignancy and specific type, with up to twice as much AMI than the
supervised and fine-tuned models. The embeddings from the SSL encoders were
generally also superior for encoding the magnification level, and the slide
ID. However, the MAE model’s clusters were worst at encoding the
magnification, and the MoCo-v3 model worst at encoding the slide ID. We
hypothesize that MoCo-v3’s poor performance on slide ID may be because the
types of differences between subjects may be comparable to the augmentations
it is tasked with being _robust_ to during training.

Across all label types for this dataset, SSL pretrained models produced the
best clusters. Within these, DINO was the best performing model with either
ResNet-50 or ViT-B architecture. The DINO training paradigm features multi-
crop training, which may have helped the encoder to produce encodings which
work well on this dataset which includes images at a variety of zoom levels
and hence features at varying apparent scales.

Table 25:  BreakHis Breakdown. Information (AMI, %) about different aspects of images in BreakHis: Bold: best encoder per aspect. Underlined: best encoder per arch. Background: from median AMI (white) to max (blue) per aspect. FT: fine-tuned with x-ent. on IN-1k.  Arch. | Encoder | FT |  Malignancy |  Tumor type |  Magnification |  Tumor type x Magnifn. |  Slide ID
---|---|---|---|---|---|---|---
RN50 | X-Ent. |  | $7$ | $12$ | $23$ | $26$ | $23$
| MoCo-v3 |  | $5$ | $10$ | $31$ | $30$ | $18$
| DINO |  | $\mathbf{\underline{14}}$ | $\mathbf{\underline{22}}$ | $\mathbf{\underline{35}}$ | $\mathbf{\underline{43}}$ | ${\underline{35}}$
| VICReg |  | $9$ | $15$ | ${33}$ | $36$ | $25$
| MoCo-v3 | ✓ | $6$ | $10$ | $19$ | $22$ | $18$
| DINO | ✓ | $7$ | $11$ | $18$ | $22$ | $20$
| VICReg | ✓ | $6$ | $10$ | $19$ | $22$ | $19$
ViT-B | X-Ent. |  | $7$ | $13$ | $20$ | $25$ | $23$
| MoCo-v3 |  | $12$ | $19$ | $30$ | $37$ | $31$
| DINO |  | $\mathbf{\underline{14}}$ | $\mathbf{\underline{22}}$ | $\underline{32}$ | ${\underline{40}}$ | $\mathbf{\underline{36}}$
| MAE (CLS) |  | $\mathbf{\underline{14}}$ | $19$ | $19$ | $28$ | $32$
| MAE (avg) |  | $11$ | $16$ | $25$ | $30$ | $26$
| MoCo-v3 | ✓ | $6$ | $10$ | $22$ | $24$ | $19$
| DINO | ✓ | $7$ | $12$ | $21$ | $24$ | $21$
| MAE (avg) | ✓ | $8$ | $13$ | $22$ | $26$ | $22$

##  Appendix P Correlation Between Clustering and kNN

Classically, SSL encoders have been evaluated by determining their
classification performance through e.g. kNN-probing (Balestriero et al.,,
2023). We propose that the quality of the clusters measured using AMI can
function as an orthogonal measure. Therefore, we compare the AMI score of the
different clustering methods with the accuracy obtained using kNN-probing with
$k=\\{1,10,20,100,200\\}$ aggregating across all encoders and datasets. kNN-
probing was chosen due to the computational restriction from running linear
probing across all datasets for all methods. Following Caron et al., (2021) we
use a weighted kNN-probing approach. Using the Spearman’s rank correlation
coefficient we find that there is a moderate positive correlation between kNN-
probing accuracy and AMI ($0.33\leq\rho\leq 0.54$), see Table 26.
Specifically, we find that HDBSCAN and Spectral Clustering correlates the most
with kNN-probing, while AP and AC w/o C correlates the least. From this we can
conclude that measuring the clustering performance of the SSL encoders is not
redundant, but instead is an orthogonal manner of measuring the performance of
the encoders.

Table 26:  Correlation between clustering AMI and kNN accuracy. We measure the Spearman’s rank correlation coefficient (%) between the AMI score of the clustered embeddings and the accuracy of the classes predicted using kNN-probing, aggregated across datasets and encoders. The two measures are consistently less correlated when using the ViT-B embeddings, and for all clustering methods we find that the two evaluation methods are only moderately correlated.  Arch. | Clusterer | $k\\!=\\!1$ | $k\\!=\\!10$ | $k\\!=\\!20$ | $k\\!=\\!100$ | $k\\!=\\!200$
---|---|---|---|---|---|---
RN50 | K-Means | 51 | 48 | 48 | 48 | 49
| Spectral | 54 | 52 | 52 | 53 | 53
| AC w/ C | 49 | 47 | 47 | 47 | 47
| AC w/o C | 45 | 44 | 44 | 44 | 44
| Affinity Prop. | 49 | 47 | 47 | 47 | 47
| HDBSCAN | 51 | 49 | 49 | 49 | 49
ViT-B | K-Means | 37 | 37 | 37 | 39 | 39
| Spectral | 37 | 38 | 38 | 40 | 40
| AC w/ C | 33 | 34 | 34 | 35 | 36
| AC w/o C | 39 | 40 | 40 | 41 | 41
| Affinity Prop. | 36 | 37 | 36 | 38 | 38
| HDBSCAN | 38 | 39 | 38 | 40 | 40

##  Appendix Q Correlation Between AMI and Silhouette Score

In addition to the scatter plot of the ranked AMI and silhouette scores shown
in the main paper (Figure 4), we also provide the scatter plot of the actual
AMI and $S$ values in Figure 10. We observe that in the UMAP-reduced embedding
space a larger extent of the silhouette score’s range is used, making the
correlation between AMI and $S$ more clear. This increases the usability of
the silhouette score as a proxy.

Figure 10: AMI–Silhouette scatter plots. The AMI and silhouette score ($S$)
per clusterer, across datasets and encoders. The silhouette scores are
measured in the original (top) and UMAP-reduced 50-d (bottom) feature spaces.
We indicate the per-clustering-method Spearman’s rank correlation ($\rho$).