TAPUDD [13] | 0.01 | 0.99 | 0.01 | 0.00 | 0.96 | 0.04
OpenMax [3] | 0.00 | 0.00 | 1.00 | 0.02 | 0.00 | 0.98
MC-Dropout [16] | 0.64 | 0.26 | 0.11 | 0.01 | 0.45 | 0.55
Deep Ensemble [33] | 0.17 | 0.83 | 0.00 | 0.00 | 0.98 | 0.02
DenseNet [28] |  |  |  |  |  |
SP | 0.76 | 0.07 | 0.17 | 0.02 | 0.18 | 0.80
ASP | 0.10 | 0.90 | 0.00 | 0.00 | 0.93 | 0.07
MD [36] | 0.00 | 1.00 | 0.00 | 0.00 | 0.96 | 0.04
KNN [70] | 0.02 | 0.90 | 0.09 | 0.01 | 0.73 | 0.26
TAPUDD [13] | 0.04 | 0.96 | 0.00 | 0.00 | 0.97 | 0.03
OpenMax [3] | 0.01 | 0.00 | 0.98 | 0.01 | 0.00 | 0.99
MC-Dropout [16] | 0.58 | 0.28 | 0.14 | 0.01 | 0.39 | 0.60
Deep Ensemble [33] | 0.03 | 0.97 | 0.00 | 0.00 | 0.99 | 0.01
ConvNeXt [41] |  |  |  |  |  |
SP | 0.66 | 0.14 | 0.20 | 0.02 | 0.33 | 0.65
ASP | 0.35 | 0.59 | 0.06 | 0.01 | 0.87 | 0.12
MD [36] | 0.05 | 0.93 | 0.03 | 0.00 | 0.94 | 0.06
KNN [70] | 0.02 | 0.95 | 0.04 | 0.00 | 0.86 | 0.13
TAPUDD [13] | 0.10 | 0.90 | 0.00 | 0.00 | 0.97 | 0.03
OpenMax [3] | 0.01 | 0.00 | 0.99 | 0.00 | 0.00 | 1.00
MC-Dropout [16] | 0.62 | 0.24 | 0.14 | 0.02 | 0.46 | 0.52
Deep Ensemble [33] | 0.00 | 1.00 | 0.00 | 0.00 | 1.00 | 0.00
MLP-Mixer [71] |  |  |  |  |  |
SP | 0.62 | 0.09 | 0.29 | 0.01 | 0.19 | 0.80
ASP | 0.47 | 0.45 | 0.08 | 0.00 | 0.74 | 0.26
MD [36] | 0.02 | 0.95 | 0.04 | 0.00 | 0.87 | 0.13
KNN [70] | 0.04 | 0.89 | 0.07 | 0.00 | 0.79 | 0.20
TAPUDD [13] | 0.31 | 0.61 | 0.08 | 0.00 | 0.68 | 0.32
OpenMax [3] | 0.01 | 0.00 | 0.99 | 0.05 | 0.00 | 0.95
MC-Dropout [16] | 0.60 | 0.14 | 0.26 | 0.00 | 0.26 | 0.74
Deep Ensemble [33] | 0.06 | 0.94 | 0.00 | 0.00 | 0.99 | 0.01

###  Appendix H Detailed Comparisons with Previous Works

In this section, we will explain in detail how our work differs from related
previous works. Our ultimate goal is to create a rigorous test (similar to
tests that humans take such as college entrance exams) to quantitatively
measure the visual perception gap between the models and humans across various
categories. Our main interest does not lie in training but on rigorously
testing the visual perception alignment. For that purpose, a dataset should
satisfy the four requirements we mentioned in Section 3 and use a proper
metric that reflects the visual perception alignment.

Peterson et al. [54] and Schmarje et al. [68] utilized their datasets mainly
for training and did not thoroughly verify whether the model actually achieved
visual alignment. Peterson et al. [54] only tested their models on in-class
samples (in our case, Category 1) and out-of-class samples (in our case,
Category 4 and Category 6) and they showed only accuracy and cross entropy,
which is analogous to KL divergence. Therefore, they did not test their models
on various possible scenarios and did not use proper measurement, as KL
divergence is not an optimal choice for visual perception alignment as we
described in Section 4.1. Schmarje et al. [68] only tested their models on in-
class samples (in our case, Category 1) and showed accuracy and KL divergence.
Therefore, although previous works trained their models with the goal of
achieving visual perception alignment, none of the works have thoroughly
verified how much the models have actually achieved visual perception
alignment under diverse situations with an appropriate measurement.

Zhang et al. [79] and Bomatter et al. [5] are similar to our work since they
show that both models and humans have better object recognition when given
more contextual information, but it is difficult to say that they have
comprehensively evaluated visual perception alignment. These two works only
tested their models on partial aspects (in our case, Category 1, Category 2,
and Category 8). Thus, these works did not test on Must-Abstain samples, which
makes it difficult to claim that they "comprehensively" evaluated visual
perception alignment. Zhang et al. [79] and Bomatter et al. [5] simply showed
that both models and humans exhibit similar performance trends based on
context (_i_._e_., when given more context, both human’s and model’s visual
recognition performance increases), and they provided human-model correlations
to describe their relative trends across conditions. However, our study on
visual perception alignment is not about following human trends, but about
measuring how well the model replicates human perception sample-wise. Hence,
considering our research scope and criteria, it’s challenging to assert that
Zhang et al. [79] and Bomatter et al. [5] rigorously measured visual
perception alignment.

In contrast, we quantitatively measured visual perception alignment across
various scenarios with multiple human annotations on uncertain images. In
addition, we borrowed Hellinger distance to precisely calculate the visual
perception alignment after careful consideration of other distance-based
metrics such as KL divergence and Total Variation distance. Furthermore, we
incorporated specialized elements (sampling theory, statistical theories
related to survey design, and experts in the related fields) in creating our
dataset.

There are three key differences that distinguish our dataset compared to
existing datasets that also focus on uncertainty in object recognition. First,
we applied corruption and cropping with different intensities ranging from 1
to 10 to reflect the continuity of uncertainty mentioned in Appendix I.1.
Uncertainty is continuous and it is critical to test models on samples where
uncertainty may increase in stages. In this sense, we tested models visual
perception alignment on varying degrees of uncertainty. Second, we obtained
134 human annotations per image to accurately estimate the ground truth visual
perception distribution. We borrowed statistical sampling theory to achieve an
error bound of lower than 5%, of which the details are in Section 3.3. Third,
while our uncertain samples include uncertainty that confuses between classes,
refer refer to as "inter-class uncertainty" (soft labels distributed only
among target classes), we also include "recognizability uncertainty" (soft
labels distributed among classes + abstention), namely whether an image itself
is recognizable or not. If an image is moderately brightened (_i_._e_.,
intermediate phase between a clear image and a complete white image), then the
object itself may or may not be recognizable. Visual perception includes not
only object identification (predicting that it is an elephant) but also object
recognizability (the object itself is recognizable). In this sense, we cover
broader scenarios compared to previous works as we include object
recognizability uncertainty in our uncertain category.

We also want to highlight that VisAlign does only contain Uncertain but also
Must-Act and Must-Abstain to cover diverse scenarios as possible. In order to
evaluate a model’s visual perception alignment, a model should be tested under
Must-Act (whether it predicts a correct class with high confidence), Must-
Abstain (whether it abstains out-of-class samples with high confidence), and
Uncertain (whether it reflects the human uncertainty). However, previous works
are limited in that they test their model on partial cases (Category 1 and
Category 4 in Peterson et al. [54], and Category 1 in Schmarje et al. [68])
which does not truly reflect visual perception alignment on various
situations. It is especially important to test models on samples from out of
distributions (_i_._e_., Category 5 and Category 7), but previous works
overlook these samples thus did not quantitately evalute from visual
perception alignment. Therefore, their dataset cannot be utilized as a
benchmark to evaluate visual perception alignment. While previous papers and
our work have in common with handling uncertainty, in our case, uncertain
samples are a subset of our final dataset and we cover more diverse necessary
situations, which previous works do not, as possible to measure the visual
perception alignment.

###  Appendix I Discussion on Uncertainty

####  I.1 Continuity of Uncertainty

In this section, we will discuss a critical aspect of uncertainty which is
continuity. Uncertainty is continuous and it is challenging to draw clear
distinctions among classes (_i_._e_., as we mentioned in the main paper that
it is hard to distinguish between "car" and "truck") and between clear and
uncertain images (_i_._e_., if at least one person claims an image as
"uncertain", then it becomes an uncertain image). However, as our ultimate
goal is to construct a universal testing benchmark that quantitatively
measures visual perception alignment between models and humans, our classes
should have clear defintions so that model developers can easily prepare their
models and training strategy. Therefore, after careful consideration, we used
the taxonomy classification in biology which is the meticulous product of
decades of efforts by countless domain experts to hierarchically distinguish
each species as accurately as possible with clear definitions. Also, in order
to comprehensively measure the visual perception alignment between models and
humans, the models should be tested under various conditions including clear
in-class images (Must-Act), clear out-of-class images (Must-Abstain) and
confusing images (Uncertain). As there is no clear boundary between clear and
uncertain images, the best scenario would be to survey all images in our
dataset to 134 people per image to obtain numerically reliable annotations.
However, surveying all images is not always feasible as it requires tremendous
amount of time and money considering that there are 1800 images (900 each in
the open and closed test sets) in our dataset. Therefore, due to the realistic
reasons, we put significant effort to include only clear images that anyone
can agree on in Must-Act and Must-Abstain and obtained human annotations on
Uncertain images. Nevertheless, we also recognize that continuity is an
essential characteristic of uncertainty that should be carefully considered
and there is always a possibility of corner cases that may be disagreeable by
at least one person. We have done our best to remove those corner case samples
and cross-validated our final selection. Further detailed analysis and
benchmark dataset on the continuity of uncertainty is highly needed and we
will leave this as a future work.

####  I.2 Coverage of Uncertainty

"Uncertainty" is a broad concept and it is hard to define with one clear line
and list all possible cases. In this paper, we chose 15 different types of
corruptions to generate uncertainty in various ways following a concrete
previous work [23]. Furthermore, to better represent the continuity of
uncertainty explained in Appendix I.1, we apply the corruptions varying
severity ranging from 1 to 10. Many types of corruption we used resemble the
reality in their own way. For example, adjusting the brightness of the image
is certainly realistic, and changing its resolution is similar to viewing an
object beyond a filter (_e_._g_., semi-transparent glass), and weather changes
are also certainly realistic. These corruptions result in some of realistic
uncertain images, precisely 8.5% in the case of the open test set, where MTurk
survey participants were struggling with differentiating between two or more
animals (rather than being confused between one animal and abstention).
Despite our meticulous effort, we are well aware that those corruptions
certainly do not cover all possible uncertainties that arise in the real
world. However, "uncertainty" is too broad to specify and diffcult to collect
or generate, and hence for now we use corruptions (but sufficiently divserse
types of corruptions).