11institutetext: Bosch Center for Artificial Intelligence 22institutetext:
University of Freiburg

# Multi-Attribute Open Set Recognition

Piyapat Saranrittichai 11 2 2 0000-0003-0620-7945 Chaithanya Kumar Mummadi
1122 0000-0002-1173-2720 Claudia Blaiotta 11 0000-0003-2314-3939 Mauricio
Munoz 11 0000-0002-9520-4430 Volker Fischer 11 0000-0001-5437-4030

###### Abstract

Open Set Recognition (OSR) extends image classification to an open-world
setting, by simultaneously classifying known classes and identifying unknown
ones. While conventional OSR approaches can detect Out-of-Distribution (OOD)
samples, they cannot provide explanations indicating which underlying visual
attribute(s) (e.g., shape, color or background) cause a specific sample to be
unknown. In this work, we introduce a novel problem setup that generalizes
conventional OSR to a multi-attribute setting, where multiple visual
attributes are simultaneously recognized. Here, OOD samples can be not only
identified but also categorized by their unknown attribute(s). We propose
simple extensions of common OSR baselines to handle this novel scenario. We
show that these baselines are vulnerable to shortcuts when spurious
correlations exist in the training dataset. This leads to poor OOD performance
which, according to our experiments, is mainly due to unintended cross-
attribute correlations of the predicted confidence scores. We provide an
empirical evidence showing that this behavior is consistent across different
baselines on both synthetic and real world datasets.

###### Keywords:

Open Set Recognition Multi-Task Learning Shortcut Learning.

##  1 Introduction

In recent years, deep learning techniques have been applied to a variety of
complex classification tasks with great success. However, the majority of
these methods rely on the closed-set assumption, where all classes in the test
data are _known_ during training. As a result, such methods are poorly suited
to more realistic scenarios where it is not possible to observe all classes
during training. Open Set Recognition (OSR) [23] extends the standard
classification task to handle _unknown_ classes that are not included in the
training set - that is, an OSR model must discriminate known from unknown
samples as well as classify known ones.

Previous research in the OSR domain has only considered a single
classification task [3, 30, 32, 4, 13, 26], where a network takes an image as
an input and produces a label prediction, possibly accompanied by a confidence
score (to make a decision on whether the input belongs to a known or unknown
class). Predicting a single label, however, is not sufficient in some real
world applications. For example, a robot picking up a red shirt needs to
recognize multiple attributes of an object, such as its shape (”shirt”) and
its color (”red”).

Figure 1: Illustration of our multi-attribute Open Set Recognition. Dataset
contains images with attribute annotations (e.g., garment type and color).
Samples with some known attribute combinations are available during training.
During inference, networks need to be able to handle images containing both
known and unknown attribute values. In case of input with unknown attribute
values, the networks have to identify which attributes are actually of unknown
values.

The task of predicting multiple attributes simultaneously is closely related
to Compositional Zero-Shot Learning (CZSL) [17, 19, 22, 2], whose goal is to
recognize unknown combinations of known attributes. However, one limitation of
CZSL is that it assumes labels of input images to be already known. Therefore,
when the model gets an input containing unknown attribute labels, it would
incorrectly map them to known ones. In this work, we are interested in multi-
attribute predictions, similar to CZSL, but we allow attributes of input
images to be of unknown labels.

The scenario described above, can be cast as a multi-attribute OSR problem
illustrated in Figure 1. Given an image, the goal is to recognize multiple
attributes (e.g., shape, color, background) at once. Additionally, if an
attribute has an unknown value, the model should be able to identify it as
such, irrespectively of whether the other attributes were seen or not before.
We use the term _attribute_ to denote a generic visual property of image’s
which can be either low-level (i.e., shape, color, etc.) or high-level (i.e.,
object type, background, etc.) depending on the target tasks. Please note that
our work is conducted under the assumption that the available attributes in a
task are semantically independent (cannot describe the same aspects of an
image). For example, in an object recognition task, texture and material are
not semantically independent attributes, since the material attribute also
contain texture information.

Several OSR approaches can naturally be extended to this novel setting.
However, additional challenges arise in the case that certain combinations of
known attributes are not available during training. This is especially likely
when the number of feasible combinations is large. In case of missing
combinations, spurious correlations among attributes are introduced leading to
shortcut learning [9]. Our setting enables analyzing the effect of shortcut
learning on OSR. In contrast to previous works in the shortcut learning
literature, which analyze the effect of shortcuts only the recognition of
known classes, our setting extends the analysis to unknown classes. We will
also demonstrate that the multi-attribute extensions of common OSR techniques
are shortcut vulnerable consistently across different approaches and datasets.

The main contributions of this work are as follows: (1) We introduce the
multi-attribute OSR setup, as a novel problem formulation promoting
explainability. (2) We extend several OSR baselines to handle the multi-
attribute setting. (3) We demonstrate, with extensive empirical analyses, on
both synthetic and realistic datasets, that many state-of-the-art OSR methods
are vulnerable to shortcut learning. Specifically, we found OSR performance to
be inversely correlated to the strength of dataset biases.

##  2 Related Works

###  2.1 Open Set Recognition

Open-Set Recognition (OSR) is the problem of simultaneously classifying known
samples and detecting unknown samples. OSR is conventionally a single-task
problem, considering only one aspect of images (e.g., object type). With deep
learning, the straightforward approach to solve OSR is Maximum Softmax
Probability (MSP) applying softmax thresholding on the output of
classification networks [20]. [3] shows that direct softmax thresholding does
not minimize open space risk and proposes OpenMax as an alternative activation
function that explicitly estimates the probability of a sample being unknown.
[24] models prototypes of individual object types to detect unknown
categories. Other approaches instead rely on generative models to improve OSR
[13, 32, 27, 21, 25]. [8, 20] generate synthetic unknown images to be used
during training. [30] directly estimates the likelihood of a sample being of a
known class using a flow-based model. [5, 4] formulate constraints to maximize
the difference between image features of known and unknown samples. Recently,
[26] shows that network performance in OSR is heavily correlated with close-
set accuracy and propose using Maximum Logit Score (MLS) for unknown class
detection. These lines of work consider only object recognition but cannot
recognize multiple attributes or infer which properties cause certain input
images to be unknown. Some studies decompose feature space into multiple
components based on image regions [12] or attributes [6] to improve
performance on fine-grained datasets. Still, their final task is that of
single-task OSR. On the other hand, we study a generalized version of the OSR
problem, in which multiple attributes are recognized simultaneously.

###  2.2 Compositional Zero-Shot Learning

Compositional Zero-Shot Learning (CZSL) is the the task of predicting both
seen and unseen combinations of known attributes. VisProd is a naïve solution,
which trains deep networks with cross-entropy losses to predict all attributes
[19]. Recent works aim to model compatibility functions between attribute
combinations and image features [17, 19, 22, 2, 18, 16]. Our problem is
closely related to CZSL as we also recognize multiple attributes at the same
time. While CZSL assumes input images to have only known attribute values, we
allow input images with unknown attribute values and would like to identify
the unknown ones during inference.

###  2.3 Shortcut Learning

Shortcut Learning studies degradation of the recognition performance caused by
spurious correlations [9, 7]. [10] shows that real-world objects, in general,
have shape-texture correlations and that deep networks trained with natural
images tend to classify objects using the simpler visual clue, which is, in
general, texture. [14] also shows that a trained model tends to extract
information from visual features that are more linearly decodable. In contrast
to most works on shortcut learning, which study its effects on the recognition
of the known classes, we additionally study its effects on the recognition of
the unknown classes.

##  3 Problem Formulation

In this work, we define _attributes_ as semantically independent visual
properties of an image. For each attribute, we aim to predict its value or to
recognize it as unknown if the value was never seen during training.

Our task formulation can be formally defined as follows. A sample from the
training set contains an image $I^{i}\in\mathcal{I}^{train}$ as well as its
attribute annotation
$\bm{y}^{i}=(a_{1}^{i},a_{2}^{i},\ldots,a_{M}^{i})\in\mathcal{Y}^{train}=\mathcal{A}^{k}_{1}\times\mathcal{A}^{k}_{2}\times\ldots\times\mathcal{A}^{k}_{M}$
where $M$ is the number of attributes and each $\mathcal{A}^{k}_{j}$
represents the set of all _attribute values_ of the $j$-th attribute seen
during training. The superscript $k$ indicates that all attribute values
available in the training set are of known values. On the other hand, a test
sample can belong to either known or unknown values. In other words,
$\mathcal{Y}^{test}=\left(\mathcal{A}^{k}_{1}\cup\mathcal{A}^{u}_{1}\right)\times\left(\mathcal{A}^{k}_{2}\cup\mathcal{A}^{u}_{2}\right)\times\ldots\times\left(\mathcal{A}^{k}_{M}\cup\mathcal{A}^{u}_{M}\right)$,
where the superscript $u$ represents ones containing unknown values.

Given the training set, our goal is to estimate a function
$f(I)=\left(\bm{\hat{y}},\bm{\hat{s}}\right)$ that takes an image $I$ from a
test sample as an input and predicts attribute values
$\bm{\hat{y}}\in\mathcal{Y}^{train}$ as well as attribute-wise confidence
scores $\bm{\hat{s}}=\left(\hat{s}_{1},\hat{s}_{2},\ldots,\hat{s}_{M}\right)$.
These confidence scores will be used to distinguish between known and unknown
attribute values by thresholding. In our work, higher confidence score
indicates higher likelihood for a sample to be of a known attribute value.

During training, some attribute combinations can be missing (see Figure 5). On
the other hand, any attribute combinations can be seen during testing,
including the combinations containing unknown attribute values. We define the
term _Open-Set OOD for $\mathcal{A}_{m}$_ ($OOD^{O}_{m}$) to represent a
sample whose attribute values are partially unknown on the $m$-th attribute.
Additional terms indicating samples with different categories of attribute
combinations are defined in Figure 2.

Figure 2: Definitions of samples which can be found during inference. Samples
with (dark or light) green colors represent samples whose attribute values are
all _known_. These values can be either the seen combinations during training
(dark green) or the unseen ones (light green). Yellow, blue and red boxes
represent attribute combinations containing _unknown_ attribute values. The
yellow and blue boxes are partially unknown as only the values of attribute 1
($OOD^{O}_{1}$) or attribute 2 ($OOD^{O}_{2}$) are unknown respectively.

##  4 Extension from common OSR approaches

(a) Conventional OSR

(b) Multi-attribute OSR ($M=2$)

Figure 3: Illustration of generic architectures for single- and multi-
attribute OSR.

For conventional single-task OSR, which is a special case of our setting where
$M=1$, model solves a single task and predicts one class label and one
confidence score. These predictions are typically computed by a network
combining a feature extractor (to compute intermediate representations) with a
prediction head (to compute final predictions) as shown in Figure 3(a). Given
a feature extractor $g$ and a prediction head $h$, the function $f$ can be
written as $f(I)=h(g(x))$, where $g(I)=z$ is the intermediate feature
representation.

In this work, we study a simple generalization of single-task OSR methods,
which is directly applicable to several state-of-the-art approaches. To extend
a single-task model, we opt to use only a single feature extractor $g$ similar
to the case of conventional OSR. However, $h$ is extended to have multiple
heads $(h_{1},h_{2},\ldots,h_{M})$ to support simultaneous prediction of
multiple attributes. This extension can be seen in Figure 3(b).

##  5 Experiments

In this section, we present a set of experiments and analysis to understand
the behavior of different methods on multi-attribute OSR setting. We begin by
describing our experimental setup and then present quantitative evaluation of
various approaches on synthetic datasets. Additionally, we uncover a possible
mechanism by which shortcuts can affect OSR performance, which relates to
undesired correlation of confidence scores across attributes. Lastly, we show
that our findings can also be observed in a more realistic dataset.

###  5.1 Experiment Setup

In this subsection, details of our experiment setup will be described. In
particular we will present the datasets, baselines and metrics used in our
study.

####  5.1.1 Datasets

(a) Color-MNIST

(b) Color-Object

(c) Scene-Object

(d) UT-Zappos

Figure 4: Images of example samples from datasets used in this work.

Datasets with multiple attribute annotations are required for multi-attribute
OSR setting. Additionally, to enable shortcut analysis, we choose datasets
where the available attributes have different complexity (i.e., in term of
linear decodability as in [14]). In the presence of shortcuts, we expect
neural networks to rely heavily on simpler but less robust visual clues. In
this work, we consider datasets where two attributes are annotated ($M=2$).
The first attribute $\mathcal{A}_{c}$ is intended to be complex (e.g., shape,
object type) while the second attribute $\mathcal{A}_{s}$ is intended to be
simple (e.g., color, background). The datasets used in this work are as
follows (example images are shown in Figure 4):

  * •

Color-MNIST: This dataset contains MNIST digits with different colors. We
construct this dataset based on the DiagVib framework [7]. Attributes
available in this dataset are digit shape and color.

  * •

Color-Object: This dataset is inspired by [1] to simulate more realistic
settings while full-control over dataset biases is still maintained. Each
image sample contains an object from COCO [15] over a background of single
color. Attributes available in this dataset are object type and background
color respectively.

  * •

Scene-Object: Similar to Color-Object, each image sample still contains a COCO
object. However, instead of using color as background, scene images from [31]
are used.

  * •

UT-Zappos: This dataset contains images of shoes annotated with two attributes
(shoe material and type) [28, 29] which is commonly used as a CZSL benchmark.

We split datasets mentioned above so that some attribute values are kept
unknown during training for OSR evaluation. Details of each dataset split will
be presented in appendix C.

####  5.1.2 Baselines

All baselines considered in this work comprise a feature extraction backbone
($g$) and multiple prediction heads $(h_{1},h_{2},\ldots,h_{M})$, as
introduced in section 4. The main difference among baselines lies in the
modeling of loss functions and retrieval of confidence scores. Baselines
considered in this work are:

  * •

Maximum Softmax Probability (MSP) [20]: This standard baseline trains networks
by minimizing a sum of cross-entropy losses across all prediction heads. The
confidence score for each attribute is simply the maximum softmax value from
its prediction head.

  * •

OpenMax [3]: The training procedure of this baseline is similar to MSP.
However, the final confidence score of each attribute will be calibrated to
reduce probability of the predicted labels based on the distance between the
estimated logit vector and its corresponding mean vector.

  * •

ARPL [4]: This baseline trains the model so that feature representations of
potential unknown samples are far away from the ones of known samples. The
suffix +CS is appended when _confusing samples_ are also generated and used
for training.

  * •

Maximum Logit Score (MLS) [26]: This baseline uses maximum logit value before
softmax activation from each prediction head as the confidence score. This
prevents certain information being discarded by softmax normalization.

Model architectures and hyperparameters used for each dataset will be
presented in appendix B.

####  5.1.3 Metrics

For the evaluation, we extend conventional OSR metrics for multi-task setting.
In contrast to conventional OSR, we have multiple prediction heads so that the
evaluation will be performed per head. We consider the average metrics over
all heads to represent overall performance. Two main metrics adopted here are
Open-Set Classification Rate (OSCR) and Area Under the Receiver Operating
Characteristic (AUROC) similar to [4, 26]. Quantitative results reported in
this section are averaged across three different random seeds.

###  5.2 Shortcut Analysis in Multi-attribute OSR

Figure 5: Training dataset configurations. Each box represents an attribute
combination whose yellow color indicates its presence during training. These
three configurations provide different degrees of spurious correlations to
simulate different shortcut opportunities.

In this section, we study effects of shortcut learning on synthetic datasets
in which we can control the attribute combinations seen during training. A low
number of seen combinations can amplify the effects of shortcut learning. As
illustrated in Figure 5, we utilize three dataset configurations: (1)
Correlated (C): With this configuration, the number of seen combination is
minimal. In other words, there is a one-to-one mapping between each available
attribute. (2) Semi-Correlated (SC): With this configuration, additional
combinations are introduced so that a trained network has enough information
to distinguish two visual attributes. The number of combinations in this
configuration is two times higher than in the Correlated configuration. (3)
Uncorrelated (UC): All combinations are available.

We control dataset generation such that the number of samples are balanced
across all attribute combinations. During testing, for each attribute, the
number of the unknown attribute values is the same as the number of the known
attribute values. In addition, the test combinations are also balanced so that
the random chance of detecting an unknown attribute is 50% per attribute. More
details of the splits are presented in appendix C.

Table 1: Average OSCR (from all attributes) on Color-MNIST, Color-Object and Scene-Object. Approach | Color-MNIST | Color-Object | Scene-Object
---|---|---|---
UC | SC | C | UC | SC | C | UC | SC | C
MSP | $80.5$ | $56.0$ | $42.9$ | $76.2$ | $58.4$ | $53.0$ | $39.7$ | $32.1$ | $29.3$
OpenMax | $79.1$ | $52.9$ | $45.2$ | $70.2$ | $45.8$ | $25.0$ | $35.5$ | $18.4$ | $13.1$
ARPL | $80.8$ | $54.0$ | $49.2$ | $77.3$ | $54.4$ | $36.0$ | $42.5$ | $34.5$ | $29.9$
ARPL+CS | $78.7$ | $52.7$ | $49.9$ | $77.6$ | $53.9$ | $38.5$ | $44.7$ | $34.8$ | $28.3$
MLS | $76.5$ | $44.5$ | $36.0$ | $74.9$ | $57.0$ | $51.6$ | $40.4$ | $32.2$ | $29.4$
MSP-D | $80.0$ | $58.3$ | $46.4$ | $75.0$ | $61.7$ | $52.4$ | $42.9$ | $35.5$ | $28.8$
OpenMax-D | $76.8$ | $49.1$ | $43.8$ | $75.2$ | $48.5$ | $22.6$ | $38.6$ | $16.6$ | $12.2$
ARPL-D | $83.6$ | $60.3$ | $45.8$ | $79.1$ | $59.2$ | $49.8$ | $44.0$ | $33.1$ | $26.3$
|  |  |  |  |  |  |  |  |

(a) Complex Attribute

(b) Simple Attribute

Figure 6: Attribute-wise OSCR evaluation on Color-MNIST with different dataset
configurations.

Table 1 shows OSCR across different approaches and dataset configurations
(i.e., Uncorrelated (UC), semi-correlated (SC) and correlated (C)). We notice
that, in uncorrelated cases, OSCR values are consistently the highest across
all baselines and datasets. These values correspond to the ideal case that
there is no dataset bias caused by spurious attribute correlations (low
shortcut opportunities). In contrast, OSR performance degrades as soon as
correlations are introduced, such as in the SC and C configurations. This
suggests that attribute correlations in the SC and C cases can induce shortcut
learning during training. Similar trends can also be observed with AUROC
metrics (see appendix A.2).

In order to investigate models’s behavior in the presence of shortcut
opportunities, we look at attribute-wise OSCR for the complex attribute
($\mathcal{A}_{c}$) and the simple attribute ($\mathcal{A}_{s}$). Figure 6
presents OSCR of $\mathcal{A}_{c}$ and $\mathcal{A}_{s}$ from different
baselines on Color-MNIST dataset. According to the results in the Figure,
spurious correlations do not have the same effects to both attributes.
Specifically, the reduction of OSCR mainly occurs on $\mathcal{A}_{c}$ but not
on $\mathcal{A}_{s}$ (For some baselines, OSCR values of $\mathcal{A}_{s}$ do
not even reduce). A possible explanation of this behavior is that, during
correlated training, a network tends to rely more heavily on the simple
attribute compared to the complex one when both attributes are predictive of
the model’s target. Similar insights were reported by previous works on
shortcut learning [9, 14]. More evaluation details and discussions regarding
to the attribute-wise evaluations will be presented in the appendix A.1.

In addition to the baseline extension method described in section 4, we also
study another simple approach that simply duplicates the whole model for each
attribute (represented with a -D appended after the approach names) resulting
in models with two times many parameters (in the case of two attributes).
According to the results in Table 1, the general trend of the performance
degradation due to spurious correlations is still hold. Throughout this work,
we opt to perform our analysis mainly based on the extension introduced in
section 4.

###  5.3 Effects of Shortcut on Confidence Scores

The results presented in the previous section indicate that spurious
correlations can degrade multi-attribute OSR performance, as measured using
common OSR metrics. Even though these metrics are useful for comparing models,
they do not uncover the underlying mechanism by which the detection of unknown
samples degrades. Since confidence scores are the main output used to
distinguish between known and unknown samples, we will explore how shortcut
learning affects the confidence scores in this section.

Figure 7: Visualization of cross-attribute confidence scores

We will visualize the behavior of confidence scores as a heatmap $\mathcal{M}$
similar to Figure 7. A value $\mathcal{M}_{ij}$ in the $(i,j)$ cell is the
average confidence score from the prediction head of the $i$-th attribute
given input sample of type $j$ (1=Known, 2=Open-Set OOD for attribute 1
($\mathcal{A}_{c}$), 3=Open-Set OOD for attribute 2 ($\mathcal{A}_{s}$)). The
ideal behaviour is that, if an input sample has a known attribute value for
the attribute $i$, the confidence score of the $i$-th head should be similar
to the case of known inputs (i.e., $\mathcal{M}_{11}\approx\mathcal{M}_{13}$
and $\mathcal{M}_{21}\approx\mathcal{M}_{22}$). On the other hand, if an input
sample is unknown with respect to the $i$-attribute, the $i$-th confidence
score should be lower (i.e., $\mathcal{M}_{11}>\mathcal{M}_{12}$ and
$\mathcal{M}_{21}>\mathcal{M}_{23}$), in which case by thresholding the score,
one can identify the unknown attribute value. Ideally, the confidence score of
one attribute should not be impacted by the attribute value of another
attribute. We called this property of the confidence score as _cross-attribute
independence_.

(a) Uncorrelated (UC)

(b) Semi-Correlated (SC)

(c) Correlated (C)

Figure 8: Cross-attribute confidence scores of MSP on different configurations
of the Color-MNIST dataset.

(a) Uncorrelated (UC)

(b) Semi-Correlated (SC)

(c) Correlated (C)

Figure 9: Cross-attribute confidence scores of MSP on different configurations
of Color-Object dataset.

We test the standard MSP baseline on uncorrelated, semi-correlated and
correlated setting. The result of the Color-MNIST and Color-Object dataset
shown in Figure 8 and Figure 9. Firstly, we notice that, in the uncorrelated
case, confidence scores are more or less similar to the ideal behaviour
mentioned above (e.g., see Figure 8(a) and 9(a)). However, as the correlation
is introduced, a cross-attribute dependency of the confidence scores can be
observed. For example, considering the semi-correlated case in Figure 9(b),
confidence score of the first attribute (object type) drops when the second
attribute (background) is unknown ($\mathcal{M}_{11}>\mathcal{M}_{13}$). This
indicates unintended dependency between confidence score of the first
attribute and the attribute value of the second attribute. In other words, the
network incorrectly uses the information of second attribute to determine if
the first attribute is unknown or not. This demonstrates one scenario that
shortcut can affect the recognition performance of OOD. In addition,
considering edge cases in which all attributes are correlated in Figure 8(c)
and 9(c), average confidence scores of the first attribute are unchanged
regardless of whether actual attribute values are unknown or not (i.e.,
$\mathcal{M}_{11}=\mathcal{M}_{12}$). The network only use the information of
the second attribute to detect the unknown of both attributes. This is an
obvious sign of shortcuts similar to the semi-correlated case mentioned
before.

###  5.4 Measuring Explainability for Unknown Samples

(a) Uncorrelated (UC)

(b) Semi-Correlated (SC)

(c) Correlated (C)

Figure 10: Open-Set Explainability Matrix of MSP on different configurations
of the Color-MNIST dataset.

(a) Uncorrelated (UC)

(b) Semi-Correlated (SC)

(c) Correlated (C)

Figure 11: Open-Set Explainability Matrix of MSP on different configurations
of the Color-Object dataset.

As mentioned earlier, one benefit of our task is that, in contrast to
conventional OSR or anomaly detection, models will be able to provide
meaningful attribute-level explanations of why certain samples are identified
as unknown. In other words, because models categorize each attribute
separately as known or unknown, they can also explain which underlying
attribute(s) are responsible for an input sample likely being unknown.

In this regard, we evaluate explainability using a confusion matrix
$\mathcal{C}$, similarly to closed-set evaluation. However, instead of
grouping samples by class labels, we group samples based on whether they are
known or unknown with respect to specific attributes. Each row and column
index correspond to a type of sample, either known, Open-Set OOD for attribute
1, Open-Set OOD for attribute 2 or Open-Set OOD (refer to the definitions in
Figure 2). The horizontal and vertical axes correspond to prediction and
ground truth respectively. A value in each cell indicates the percentage of
samples from the ground truth group corresponded to the row index which the
network predicts to the group corresponded to the column index. The sum of
values in each row is normalized to 100. Ideally, we would like to get high
values along the diagonal and low values on off-diagonal cells which indicates
the prediction of the correct groups. It should be noted that, unlike previous
experiments, the evaluation in this section needs thresholding to identify the
unknowns. Here, we pick thresholds that maximize micro-F-Measure metric [11].

Results on the Color-MNIST and Color-Object datasets are shown in Figure 10
and 11. In the uncorrelated case, we obtain high values along the diagonal,
thus indicating good explainability of OOD samples (see Figure 10(a) and
11(a)). Shortcuts do, as to be expected, degrade model’s ability to explain an
OOD outcome, as shown by the results in the semi-correlated and correlated
cases. In Figure 11(b), the high value in the off-diagonal cell
$\mathcal{C}_{12}$ indicates poor performance to recognize unknown for the
complex attribute (i.e., object type in this case). This result is consistent
as in section 5.3 since shortcuts make confidence scores of complex attributes
less distinguishable for its own attribute. Recognition performance of the
complex attribute is also poor even for the known attribute values, therefore,
the optimal threshold get less reward from identifying a sample as the known.
Thresholding is then likely to output more unknown predictions. This behaviour
can be more clearly seen in the fully-correlated case (Figure 10(c) and
11(c)), where complex attributes for all samples are identified as unknown.
From all of our experiments, we can conclude that shortcuts affect performance
to recognize OOD and explanability of the models.

###  5.5 Evaluation on a Realistic Dataset

Approach | $\mathcal{A}_{c}$ | $\mathcal{A}_{s}$ | $\bar{\mathcal{A}}$
---|---|---|---
MSP | $45.3$ | $67.7$ | $56.5$
OpenMax | $45.2$ | $68.3$ | $56.8$
ARPL | $51.0$ | $69.6$ | $60.3$
ARPL+CS | $51.9$ | $67.5$ | $59.7$
MLS | $44.7$ | $71.2$ | $58.0$
|  |  |
(a) AUROC evaluation

(b) Confidence scores

Figure 12: Qualitative Evaluation on UT-Zappos (a) AUROC for material
($\mathcal{A}_{c}$), type ($\mathcal{A}_{s}$) predictions and their average
($\bar{\mathcal{A}}$) (b) Cross-attribute confidence scores of MSP.

In previous sections, we conducted experiments on well-controlled synthetic
datasets and empirically shows effects of shortcut learning on multi-attribute
OSR. In this section, we will demonstrate that our findings on synthetic
datasets can also be observed in realistic settings. In this regard, we use an
open set split based on UT-Zappos dataset [28, 29], a common dataset for CZSL.
The dataset contains shoe images annotated with shoe materials and types. In
the context of UT-Zappos, shoe material is relatively fine-grained compared to
shoe type (as seen in Figure 4(d)). Therefore, we consider shoe material as
the complex attribute ($\mathcal{A}_{c}$) and shoe type as the simple
attribute ($\mathcal{A}_{s}$).

Figure 12(a) shows AUROC on UT-Zappos from various baselines. The general
trend follows synthetic scenarios that AUROC values tend to be lower for
$\mathcal{A}_{c}$ and higher for $\mathcal{A}_{s}$. We also look at the cross-
attribute confidence scores for MSP as in Figure 12(b). According to the
Figure, the scores are similar for the first and second column
($\mathcal{M}_{11}\approx\mathcal{M}_{12}$ and
$\mathcal{M}_{21}\approx\mathcal{M}_{22}$) where shoe types are known, while,
the scores drop on the third column ($\mathcal{M}_{13}$ and
$\mathcal{M}_{23}$) where shoe types are unknown. This indicates that shoe
types are mainly determined the output of unknown detection even for the shoe
material prediction head. In other words, models take $\mathcal{A}_{s}$ as
shortcuts for unknown detection of $\mathcal{A}_{c}$ demonstrating the same
observation as in the semi-correlated cases of section 5.3. This shows that
our findings are also generalized to realistic scenarios.

##  6 Conclusion

In this work, we generalize the single-task OSR to a multi-attribute OSR
setup. This enables trained models to identify unknown properties of input
images in an attribute-wise manner. We propose simple extensions of common OSR
baselines for our task. These extended baselines are evaluated across multiple
datasets. We found that these naïve extensions are shortcut vulnerable
especially when higher degrees of attribute correlations are introduced in
training sets. An investigation of how the models are affected by shortcuts is
presented. We found that confidence scores predicted by the models are likely
to be affected by unrelated attributes when spurious correlations are
introduced. Lastly, we also evaluate the explainability of the models trained
on our task. While the problem we investigated in this work remains open
challenge in the domain of visual perception, we hope that it can inspire
further researches in this direction.

## References

  * [1] Ahmed, F., Bengio, Y., van Seijen, H., Courville, A.: Systematic generalisation with group invariant predictions. In: International Conference on Learning Representations (2020)
  * [2] Atzmon, Y., Kreuk, F., Shalit, U., Chechik, G.: A causal view of compositional zero-shot recognition. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 1462–1473. Curran Associates, Inc. (2020)
  * [3] Bendale, A., Boult, T.E.: Towards open set deep networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1563–1572 (2016)
  * [4] Chen, G., Peng, P., Wang, X., Tian, Y.: Adversarial reciprocal points learning for open set recognition. arXiv preprint arXiv:2103.00953 (2021)
  * [5] Chen, G., Qiao, L., Shi, Y., Peng, P., Li, J., Huang, T., Pu, S., Tian, Y.: Learning open set network with discriminative reciprocal points. In: European Conference on Computer Vision. pp. 507–522. Springer (2020)
  * [6] Du, S., Hong, C., Chen, Y., Cao, Z., Zhang, Z.: Class-attribute inconsistency learning for novelty detection. Pattern Recognition 126, 108582 (2022)
  * [7] Eulig, E., Saranrittichai, P., Mummadi, C.K., Rambach, K., Beluch, W., Shi, X., Fischer, V.: Diagvib-6: A diagnostic benchmark suite for vision models in the presence of shortcut and generalization opportunities. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10655–10664 (2021)
  * [8] Ge, Z., Demyanov, S., Chen, Z., Garnavi, R.: Generative openmax for multi-class open set classification. arXiv preprint arXiv:1707.07418 (2017)
  * [9] Geirhos, R., Jacobsen, J.H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., Wichmann, F.A.: Shortcut learning in deep neural networks. Nature Machine Intelligence 2(11), 665–673 (2020)
  * [10] Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F.A., Brendel, W.: Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In: ICLR (2018)
  * [11] Geng, C., Huang, S.j., Chen, S.: Recent advances in open set recognition: A survey. IEEE transactions on pattern analysis and machine intelligence 43(10), 3614–3631 (2020)
  * [12] Gillert, A., von Lukas, U.F.: Towards combined open set recognition and out-of-distribution detection for fine-grained classification. In: VISIGRAPP (5: VISAPP). pp. 225–233 (2021)
  * [13] Guo, Y., Camporese, G., Yang, W., Sperduti, A., Ballan, L.: Conditional variational capsule network for open set recognition. arXiv preprint arXiv:2104.09159 (2021)
  * [14] Hermann, K., Lampinen, A.: What shapes feature representations? exploring datasets, architectures, and training. In: NeurIPS. vol. 33, pp. 9995–10006. Curran Associates, Inc. (2020)
  * [15] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)
  * [16] Mancini, M., Naeem, M.F., Xian, Y., Akata, Z.: Open world compositional zero-shot learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5222–5230 (2021)
  * [17] Misra, I., Gupta, A., Hebert, M.: From red wine to red tomato: Composition with context. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1792–1801 (2017)
  * [18] Naeem, M.F., Xian, Y., Tombari, F., Akata, Z.: Learning graph embeddings for compositional zero-shot learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 953–962 (2021)
  * [19] Nagarajan, T., Grauman, K.: Attributes as operators: factorizing unseen attribute-object compositions. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 169–185 (2018)
  * [20] Neal, L., Olson, M., Fern, X., Wong, W.K., Li, F.: Open set learning with counterfactual images. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 613–628 (2018)
  * [21] Oza, P., Patel, V.M.: C2ae: Class conditioned auto-encoder for open-set recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2307–2316 (2019)
  * [22] Purushwalkam, S., Nickel, M., Gupta, A., Ranzato, M.: Task-driven modular networks for zero-shot compositional learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3593–3602 (2019)
  * [23] Scheirer, W.J., de Rezende Rocha, A., Sapkota, A., Boult, T.E.: Toward open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 35(7), 1757–1772 (2013). https://doi.org/10.1109/TPAMI.2012.256
  * [24] Shu, Y., Shi, Y., Wang, Y., Huang, T., Tian, Y.: p-odn: prototype-based open deep network for open set recognition. Scientific reports 10(1), 1–13 (2020)
  * [25] Sun, X., Yang, Z., Zhang, C., Ling, K.V., Peng, G.: Conditional gaussian distribution learning for open set recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13480–13489 (2020)
  * [26] Vaze, S., Han, K., Vedaldi, A., Zisserman, A.: Open-set recognition: A good closed-set classifier is all you need. In: International Conference on Learning Representations (2021)
  * [27] Yoshihashi, R., Shao, W., Kawakami, R., You, S., Iida, M., Naemura, T.: Classification-reconstruction learning for open-set recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4016–4025 (2019)
  * [28] Yu, A., Grauman, K.: Fine-grained visual comparisons with local learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 192–199 (2014)
  * [29] Yu, A., Grauman, K.: Semantic jitter: Dense supervision for visual comparisons via synthetic images. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 5570–5579 (2017)
  * [30] Zhang, H., Li, A., Guo, J., Guo, Y.: Hybrid models for open set recognition. In: European Conference on Computer Vision. pp. 102–117. Springer (2020)
  * [31] Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence 40(6), 1452–1464 (2017)
  * [32] Zhou, D.W., Ye, H.J., Zhan, D.C.: Learning placeholders for open-set recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4401–4410 (2021)

##  Appendix A Additional Quantitative Evaluation

###  A.1 Attribute-wise Open-Set Classification Rate (OSCR)

Table 2: Average OSCR (from the complex attribute $\mathcal{A}_{c}$) on Color-MNIST, Color-Object and Scene-Object. Approach | Color-MNIST | Color-Object | Scene-Object
---|---|---|---
UC | SC | C | UC | SC | C | UC | SC | C
MSP | $76.2$ | $30.3$ | $11.4$ | $55.3$ | $23.0$ | $10.4$ | $29.9$ | $22.0$ | $18.9$
OpenMax | $76.1$ | $29.0$ | $11.3$ | $47.4$ | $14.2$ | $11.1$ | $27.6$ | $13.1$ | $11.1$
ARPL | $79.0$ | $27.3$ | $11.2$ | $57.7$ | $22.7$ | $9.7$ | $34.5$ | $24.2$ | $19.6$
ARPL+CS | $81.4$ | $28.3$ | $11.1$ | $59.5$ | $22.5$ | $9.3$ | $36.3$ | $24.7$ | $18.5$
MLS | $74.2$ | $26.2$ | $10.9$ | $53.5$ | $22.2$ | $10.3$ | $29.4$ | $21.7$ | $19.0$
MSP-D | $81.8$ | $33.3$ | $11.4$ | $58.2$ | $27.4$ | $10.7$ | $33.6$ | $24.5$ | $18.6$
OpenMax-D | $81.4$ | $32.7$ | $13.0$ | $53.4$ | $17.0$ | $10.4$ | $31.5$ | $14.5$ | $11.1$
ARPL-D | $81.6$ | $31.2$ | $11.8$ | $59.4$ | $28.8$ | $12.1$ | $33.0$ | $22.4$ | $16.8$
|  |  |  |  |  |  |  |  |
Table 3: Average OSCR (from the simple attribute $\mathcal{A}_{s}$) on Color-MNIST, Color-Object and Scene-Object. Approach | Color-MNIST | Color-Object | Scene-Object
---|---|---|---
UC | SC | C | UC | SC | C | UC | SC | C
MSP | $84.9$ | $81.8$ | $74.4$ | $97.1$ | $93.8$ | $95.7$ | $49.5$ | $42.3$ | $39.6$
OpenMax | $82.2$ | $76.7$ | $79.1$ | $93.0$ | $77.3$ | $38.9$ | $43.5$ | $23.8$ | $15.0$
ARPL | $82.6$ | $80.8$ | $87.1$ | $97.0$ | $86.2$ | $62.4$ | $50.4$ | $44.7$ | $40.3$
ARPL+CS | $75.9$ | $77.1$ | $88.7$ | $95.6$ | $85.2$ | $67.7$ | $53.0$ | $44.8$ | $38.1$
MLS | $78.8$ | $62.9$ | $61.0$ | $96.4$ | $91.7$ | $93.0$ | $51.4$ | $42.8$ | $39.9$
MSP-D | $78.3$ | $83.3$ | $81.5$ | $91.8$ | $96.0$ | $94.2$ | $52.2$ | $46.4$ | $39.0$
OpenMax-D | $72.1$ | $65.6$ | $74.6$ | $97.0$ | $80.0$ | $34.8$ | $45.8$ | $18.7$ | $13.3$
ARPL-D | $85.7$ | $89.4$ | $79.8$ | $98.9$ | $89.6$ | $87.5$ | $55.0$ | $43.8$ | $35.8$
|  |  |  |  |  |  |  |  |

In section 5.2, we have investigated the effects of spurious correlations on
multi-attribute OSR performance. We mentioned that spurious correlations do
not have the same effects on all attributes. Specifically, the correlations
mainly degrade recognition performance of the the complex attribute
$\mathcal{A}_{c}$ more significantly. In this section, we would like to
provide quantitative evaluation in more details.

OSCR evaluation for $\mathcal{A}_{c}$ and $\mathcal{A}_{s}$ are presented in
Table 2 and 3 respectively. It can be clearly observed from the tables that,
the degradation of OSR performance due to spurious correlations on
$\mathcal{A}_{c}$ is more severe. Considering the evaluation of Color-MNIST as
an example, the OSCR of MSP for $\mathcal{A}_{c}$ is decreased from 76.2 in
the uncorrelated case to 11.4 in the correlated case (see Table 2). While, for
$\mathcal{A}_{s}$, the difference between OSCR in the uncorrelated case and
the correlated case is much smaller (84.9 and 74.4 for MSP from Table 3). This
evidence infers that, spurious correlations encourage deep networks to focus
more on encoding the information of simple attribute leading to shortcut
learning that degrades recognition performance of the complex attribute.

While, in Table 2, OSCR metrics of the complex attribute $\mathcal{A}_{c}$
monotonically reduce with respect to the correlations consistently, we observe
that, for the simple attribute $\mathcal{A}_{s}$, some baselines perform
better in more correlated cases. For example, according to Table 3, on Color-
MNIST dataset, ARPL has OSCR of 87.1 in the correlated case which is higher
than 82.6 in the uncorrelated case. This is not a surprise behavior as, in the
correlated case, both $\mathcal{A}_{c}$ and $\mathcal{A}_{s}$ are equally
predictive for target labels during training. In this regard, a model is
likely to based its decision on $\mathcal{A}_{s}$ more than $\mathcal{A}_{c}$
as $\mathcal{A}_{s}$ is easier to learn resulting in $\mathcal{A}_{s}$ being a
shortcut feature. Consequently, extensively emphasizing on $\mathcal{A}_{s}$
in the correlated cases can, for some baselines, even lead to increased
performance on $\mathcal{A}_{s}$ as observed in Table 3.

###  A.2 Evaluation with AUROC Metric

Table 4: Average AUROC (from all attributes) on Color-MNIST, Color-Object and Scene-Object. Approach | Color-MNIST | Color-Object | Scene-Object
---|---|---|---
UC | SC | C | UC | SC | C | UC | SC | C
MSP | $81.9$ | $69.6$ | $61.5$ | $82.4$ | $73.9$ | $74.0$ | $58.3$ | $57.3$ | $57.2$
OpenMax | $80.8$ | $66.8$ | $64.0$ | $78.8$ | $67.8$ | $55.4$ | $57.2$ | $52.2$ | $52.0$
ARPL | $83.3$ | $69.8$ | $67.7$ | $82.5$ | $69.1$ | $57.4$ | $59.2$ | $57.7$ | $58.2$
ARPL+CS | $79.8$ | $67.1$ | $68.4$ | $82.3$ | $68.9$ | $59.4$ | $61.5$ | $57.8$ | $57.4$
MLS | $78.2$ | $58.2$ | $54.7$ | $81.5$ | $72.6$ | $72.6$ | $59.4$ | $58.1$ | $57.7$
MSP-D | $80.8$ | $71.4$ | $65.0$ | $80.3$ | $74.9$ | $72.9$ | $60.1$ | $59.1$ | $57.2$
OpenMax-D | $77.4$ | $61.5$ | $62.1$ | $80.1$ | $65.1$ | $55.5$ | $57.9$ | $51.6$ | $50.5$
ARPL-D | $85.3$ | $73.3$ | $64.1$ | $83.9$ | $72.1$ | $70.5$ | $61.2$ | $57.7$ | $57.1$
|  |  |  |  |  |  |  |  |
Table 5: Average AUROC (from the complex attribute $\mathcal{A}_{c}$) on Color-MNIST, Color-Object and Scene-Object. Approach | Color-MNIST | Colored-Objects | Scene-Objects
---|---|---|---
UC | SC | C | UC | SC | C | UC | SC | C
MSP | $78.9$ | $57.4$ | $48.6$ | $67.4$ | $53.9$ | $52.2$ | $54.4$ | $55.3$ | $53.2$
OpenMax | $79.4$ | $56.9$ | $48.3$ | $63.1$ | $52.1$ | $50.0$ | $53.6$ | $51.1$ | $51.5$
ARPL | $84.1$ | $58.9$ | $48.3$ | $67.9$ | $51.9$ | $52.3$ | $56.8$ | $54.5$ | $54.7$
ARPL+CS | $83.7$ | $57.1$ | $48.0$ | $68.9$ | $52.5$ | $51.1$ | $58.8$ | $54.4$ | $53.2$
MLS | $77.5$ | $53.6$ | $48.3$ | $66.4$ | $53.3$ | $52.0$ | $53.7$ | $55.4$ | $53.2$
MSP-D | $83.3$ | $59.6$ | $48.5$ | $68.7$ | $53.8$ | $51.5$ | $56.8$ | $56.1$ | $54.1$
OpenMax-D | $82.7$ | $57.3$ | $49.2$ | $64.5$ | $49.6$ | $50.6$ | $55.7$ | $51.3$ | $50.8$
ARPL-D | $84.9$ | $57.2$ | $48.4$ | $68.9$ | $54.4$ | $52.5$ | $56.1$ | $54.0$ | $54.0$
|  |  |  |  |  |  |  |  |
Table 6: Average AUROC (from the simple attribute $\mathcal{A}_{s}$) on Color-MNIST, Color-Object and Scene-Object. Approach | Color-MNIST | Colored-Objects | Scene-Objects
---|---|---|---
UC | SC | C | UC | SC | C | UC | SC | C
MSP | $85.0$ | $81.8$ | $74.4$ | $97.3$ | $93.9$ | $95.7$ | $62.2$ | $59.3$ | $61.2$
OpenMax | $82.2$ | $76.8$ | $79.8$ | $94.4$ | $83.6$ | $60.8$ | $60.9$ | $53.3$ | $52.4$
ARPL | $82.6$ | $80.8$ | $87.1$ | $97.2$ | $86.4$ | $62.5$ | $61.5$ | $60.8$ | $61.6$
ARPL+CS | $75.9$ | $77.1$ | $88.7$ | $95.7$ | $85.3$ | $67.8$ | $64.2$ | $61.2$ | $61.6$
MLS | $78.8$ | $62.9$ | $61.0$ | $96.6$ | $91.9$ | $93.1$ | $65.2$ | $60.7$ | $62.3$
MSP-D | $78.2$ | $83.3$ | $81.6$ | $91.9$ | $96.1$ | $94.3$ | $63.3$ | $62.0$ | $60.3$
OpenMax-D | $72.1$ | $65.7$ | $74.9$ | $95.8$ | $80.6$ | $60.5$ | $60.1$ | $51.8$ | $50.2$
ARPL-D | $85.7$ | $89.4$ | $79.8$ | $98.9$ | $89.8$ | $88.6$ | $66.3$ | $61.4$ | $60.2$
|  |  |  |  |  |  |  |  |

In section 5.2, we conduct experiments on multi-attribute OSR and observe
relationship between spurious correlations and OSR performance evaluated with
OSCR metric. In this section, we would like to show that using AUROC metric
for evaluation provides also consistent results. According to the results in
Table 4, 5 and 6, the following observations can still be observed:

  * •

Spurious correlations cause degradation of multi-attribute OSR performance
(Table 4 provides similar observations as Table 1).

  * •

Performance degradation on the complex attribute is more significant compared
to the one on the simple attribute (5 and 6 provide similar observations as
Table 2 and 3).

##  Appendix B Implementation Details

In this section, we would like to provide additional details of our training
and network hyperparameters. We would begin with the details of MSP, OpenMax
and MLS baselines. These baselines have common architectures and loss terms
during training. The only difference among these baselines is confident score
estimation. For multi-attribute OSR, their network architectures are based on
the illustration in Figure 3(b). For Color-MNIST, the feature extraction $g$
is LeNet followed by a linear layer producing a feature representation $z$ of
128 dimensions. Each prediction head $h_{i}$ is a simple multilayer perceptron
(MLP) with 2 hidden layers (each has a linear layer of 128 hidden units
followed by Batch normalization and ReLU activation). We use Adam optimization
for training with the learning rate of $0.001$. The maximum number of training
epoch is 400. For Color-Object, Scene-Object and UT-Zappos, most
hyperparameters are the same as in the case of Color-MNIST except that the
feature extraction $g$ is based on ResNet18 instead of LeNet. Also, learning
rate of $0.0001$ is used. Our experiments are implemented using PyTorch. The
code for our implementation is publicly available at
https://github.com/boschresearch/multiosr.

For ARPL variations, we perform experiments based on the code and
hyperparameters published by the author [5]. However, for fair comparison, we
have modified its feature extraction to LeNet (for Color-MNIST) and ResNet-18
(for Color-Object, Scene-Object and UT-Zappos).

At the end of each training epoch, the loss will be computed also on the hold-
out validation set. The final model evaluated on the test set will be chosen
across all training epochs as the one having the minimum total loss on the
validation set.

##  Appendix C Details of the Dataset Details

Table 7: Details of Color-MNIST dataset splits. Property |  Details
---|---
Complex/Simple Attributes |  Digit / Color
Known Attribute Values |  {0, 1, 2, 3, 4} /
{red, yellow, green, cyan, blue}
Unknown Attribute Values |  {5, 6, 7, 8, 9} /
{magenta, orange, violet, azure, rose}
Attribute Combinations (Correlated Training) |  {(0, red), (1, yellow), (2, green), (3, cyan), (4, blue)}
Attribute Combinations (Semi-Correlated Training) |  {(0, red), (0, yellow), (1, yellow), (1, green), (2, green), (2, cyan), (3, cyan), (3, blue), (4, blue), (4, yellow)}
|
Table 8: Details of Color-Object dataset splits. Property |  Details
---|---
Complex/Simple Attributes |  Object Type / Color
Known Attribute Values |  {boat, airplane, truck, dog, zebra, horse} /
{$C_{1}$,$C_{2}$, $C_{3}$, $C_{4}$, $C_{5}$, $C_{6}$}
Unknown Attribute Values |  {bird, motorcycle, elephant, bear, bed, giraffe} /
{$C_{7}$, $C_{8}$, $C_{9}$, $C_{10}$, $C_{11}$, $C_{12}$}
Attribute Combinations (Correlated Training) |  {(boat, $C_{1}$), (airplane, $C_{2}$), (truck, $C_{3}$), (dog, $C_{4}$), (zebra, $C_{5}$), (horse, $C_{6}$)}
Attribute Combinations (Semi-Correlated Training) |  {(boat, $C_{1}$), {(boat, $C_{2}$), (airplane, $C_{2}$), (airplane, $C_{3}$), (truck, $C_{3}$), (truck, $C_{4}$), (dog, $C_{4}$), (dog, $C_{5}$), (zebra, $C_{5}$), (zebra, $C_{6}$), (horse, $C_{6}$), (horse, $C_{1}$)}
|
Table 9: Details of Scene-Object dataset splits. Property |  Details
---|---
Complex/Simple Attributes |  Object Type / Scene Type
Known Attribute Values |  {boat, airplane, truck, dog, zebra, horse} /
{beach, canyon, building, stair, desert, crevasse}
Unknown Attribute Values |  {bird, motorcycle, elephant, bear, bed, giraffe} /
{ball_pit, oast_house, kasbah, lighthouse, pagoda, rock_arch}
Attribute Combinations (Correlated Training) |  {(boat, beach), (airplane, canyon), (truck, building), (dog, stair), (zebra, desert), (horse, crevasse)}
Attribute Combinations (Semi-Correlated Training) |  {(boat, beach), {(boat, canyon), (airplane, canyon), (airplane, building), (truck, building), (truck, stair), (dog, stair), (dog, desert), (zebra, desert), (zebra, crevasse), (horse, crevasse), (horse, beach)}
|
Table 10: Details of UT-Zappos dataset splits. Property |  Details
---|---
Complex/Simple Attributes |  Shoe Material / Shoe Type
Known Attribute Values |  {Faux.Leather, Full.grain.leather, Leather, Rubber, Suede} /
{Boots.Knee.High, Boots.Mid-Calf, Shoes.Flats, Shoes.Heels, Shoes.Loafers}
Unknown Attribute Values |  {Canvas, Nubuck, Patent.Leather, Satin, Synthetic} /
{Boots.Ankle, Sandals, Shoes.Oxfords, Shoes.Sneakers.and.Athletic.Shoes}
Attribute Combinations (Training) |  {(Faux.Leather, Boots.Knee.High), (Faux.Leather, Boots.Mid-Calf), (Faux.Leather, Shoes.Flats), (Full.grain.leather, Boots.Mid-Calf), (Full.grain.leather, Shoes.Loafers), (Leather, Shoes.Flats), (Leather, Shoes.Heels), (Leather, Shoes.Loafers), (Rubber, Boots.Knee.High), (Rubber, Boots.Mid-Calf), (Suede, Boots.Knee.High), (Suede, Shoes.Flats), (Suede, Shoes.Heels)}
|

In this section, we would like to provide details of the specific splits used
in different dataset configurations. The details of Color-MNIST, Color-Object,
Scene-Object and UT-Zappos datasets are provided in Table 7, 8, 9 and 10
respectively. For Color-Object, the attribute values of color are defined in
RGB space as follows: $C_{1},C_{2},\ldots,C_{12}$ = (0, 100, 0), (188, 143,
143), (255, 0, 0), (255, 215, 0), (0, 255, 0), (65, 105, 225), (255, 20, 147),
(135, 188, 191), (27, 145, 68), (48, 101, 53), (20, 235, 154), (187, 33, 227).