# Neural Concept Binder

Wolfgang Stammer1,2,∗ Antonia Wüst1,∗ David Steinmann1,∗ Kristian
Kersting1,2,3,4 1Computer Science Department, TU Darmstadt; 2Hessian Center
for AI (hessian.AI);
3German Research Center for AI (DFKI); 4Centre for Cognitive Science, TU
Darmstadt

###### Abstract

The challenge in object-based visual reasoning lies in generating descriptive
yet distinct concept representations. Moreover, doing this in an unsupervised
fashion requires human users to understand a model’s learned concepts and
potentially revise false concepts. In addressing this challenge, we introduce
the Neural Concept Binder, a new framework for deriving discrete concept
representations resulting in what we term “concept-slot encodings”. These
encodings leverage both "soft binding" via object-centric block-slot encodings
and "hard binding" via retrieval-based inference. The Neural Concept Binder
facilitates straightforward concept inspection and direct integration of
external knowledge, such as human input or insights from other AI models like
GPT-4. Additionally, we demonstrate that incorporating the hard binding
mechanism does not compromise performance; instead, it enables seamless
integration into both neural and symbolic modules for intricate reasoning
tasks, as evidenced by evaluations on our newly introduced CLEVR-Sudoku
dataset.111Code and data available here.††∗These authors share equal
contribution.††Correspondence to: Wolfgang Stammer<EMAIL_ADDRESS>darmstadt.de¿.

##  1 Introduction

Figure 1: Unsupervised learning of concepts for visual reasoning. Models that
learn concepts from unlabeled data require inspectable and revisable concept
representations.

An essential aspect of visual reasoning is obtaining a proper conceptual
understanding of the world by learning visual concepts and processing these
into a suitable representation (_cf._ Fig. 1). The majority of current machine
learning (ML) approaches that focus on visual concept-based processing utilize
forms of supervised [28, 59, 26], weakly-supervised [34, 60, 41, 54, 4, 72] or
text-guided [23] learning of concepts. These approaches all require some form
of additional (prior) knowledge about the relevant domain. An attractive
alternative, though much more challenging, is to learn concepts in an
unsupervised fashion. This comes with several challenges: (i) learning an
expressive concept representation without concept supervision is intrinsically
difficult [33], and (ii) there is no guarantee that learned concepts align
with general domain knowledge [30, 75, 5] and can therefore be utilized for
complex downstream tasks. Moreover, (iii) to trust that the learned concept
representations are reliable for high stakes scenarios [10], it is necessary
to make the model’s concept representations human-inspectable and -revisable
[59, 60, 25] (_cf._ Fig. 1).

These challenges raise questions about the nature of the unsupervised learned
concept representations. Continuous encodings [51, 50, 68, 70] are easier to
learn and more expressive. However, they are difficult to interpret and suffer
from problems related to poor generalization [71] and information leakage [40,
42]. On the other hand, discrete encodings [60, 70, 20, 2] are hard to learn
[37, 63, 5], but are easier to understand and thus align, _e.g._ , to a task
at hand.

This work proposes the Neural Concept Binder (NCB) framework to learn
expressive yet inspectable and revisable concepts from unlabeled data. NCB
combines continuous encodings, obtained via block-slot-based soft-binding,
with discrete concept representations, obtained via retrieval-based hard-
binding. Thus, to tackle the challenges of unsupervised concept learning, NCB
harnesses the advantages of continuous and discrete concept representations.
Moreover, NCB features a straightforward inspection of learned concepts and
facilitates easy revision procedures to align the obtained concept knowledge,
_e.g._ , with prior knowledge. In our evaluations, we show that NCB’s discrete
concept-slot encodings retain the expressiveness of their continuous
counterparts. Furthermore, they can easily be used in downstream applications
via symbolic and transparent neural computations. In this context we introduce
our novel CLEVR-Sudoku dataset that represents a challenging visual puzzle
requiring both perception and reasoning capabilities (_cf._ Fig. 4).

In summary, our contributions are the following: (i) we introduce the Neural
Concept Binder framework (NCB) for unsupervised concept learning, (ii) we show
the possibilities to integrate NCB with symbolic and subsymbolic modules in
challenging downstream tasks, achieving performance on par with supervised
trained models, (iii) we highlight the possibilities of easy concept
inspection and revision via NCB, and (iv) we introduce the novel CLEVR-Sudoku
dataset, which combines challenging visual perception and symbolic reasoning.

##  2 Related Work

Unsupervised visual concept learning is concerned with obtaining concept-level
representations from unlabeled images [19]. Where some works have tackled this
only for specific domains, _e.g._ , to extract “teachable” concepts for chess
[53] or manipulation concepts via videos of task demonstrations [32] others
depend on object-level concept guidance from initial image segmentations [21]
or “natural supervision” [41]. In contrast, Vedantam et al. [68] and Wüst et
al. [71] consider learning higher-level, relational concepts, _i.e._ ,
assuming basic-level concepts have been provided. Related to their task-
supervised manner of learning concepts, Wang et al. [69] obtain concepts via
the training signal from their image classification task, thereby focusing on
image-region-based concepts. Recently, several works have also investigated
the potential of leveraging the knowledge stored in large pretrained models,
_e.g._ , via combinations of large-language models and CLIP embeddings [72,
43] or weakly-supervised queries to a vision-language model [4]. These works
contain some form of supervision via text, class labels, or on the prompt
level. In contrast, in this work we learn unsupervised concepts on both object
and factor levels, which remain inherently inspectable and revisable.

The motivation for inherently inspectable and revisable concept
representations is to allow human stakeholders to investigate and potentially
revise a model’s internal concepts. A majority of research focuses on post-hoc
approaches based on distilling concept knowledge from pretrained models [73,
16, 74, 48, 13, 17]. In contrast, Lage and Doshi-Velez [29] investigate how to
learn inspectable concept representations via human feedback, focusing hereby
on tabular data and higher-level concepts. Additionally, Stammer et al. [60]
learn inherently inspectable visual concepts via weak supervision and a
prototype-based binding mechanism. No single work develops inherently
inspectable and revisable concept representations in the context of
unsupervised visual learning.

The different properties of discrete vs. continuous encodings is a vibrant
research topic that is very relevant for learning suitable concept
representations. Continuous encodings allow for easier and more flexible
optimization and information binding [36, 55, 56, 3]. Discrete representations
are, however, considered to be vital for understanding AI models [25] and for
mitigating shortcut learning [59], but also benefitial for solving complex
visual reasoning tasks [20, 57]. However, learning discrete representations
via neural modules is a challenging problem [37, 18, 15, 65]. Where some works
have focused on categorical-distribution-based discretization [2, 22, 39],
others have investigated the potential of retrieval-based discretization of
continuous encodings via different forms of inherent “codebooks” [64, 62].
Only a few have investigated how to explicitly bind semantic visual
information to specific discrete representations [60]. Where previous works
mainly focus on one of the two representation types, we see great potential in
the recent trend of explicitly integrating both types of representations [12].

Figure 2: The Neural Concept Binder (NCB) combines (i) continuous, block-slot
encodings via slot-attention based image processing with (ii) discrete,
concept-slot encodings via retrieval-based inference. The retrieval corpus
(distilled from the block-slot encodings) allows for easy concept inspection
and revision by human stakeholders. Moreover, the resulting concept-slot
encodings can be easily integrated into complex downstream tasks.

##  3 Neural Concept Binder (NCB): Extracting Hard from Soft Concepts

Our proposed Neural Concept Binder (NCB) framework tackles the challenge of
learning inspectable and revisable object-factor level concepts from unlabeled
images by combining two key elements: (i) continuous representations via
(block-)slot-attention [36] based image processing with (ii) discrete
representations via retrieval-based inference. Fig. 2 provides an overview of
NCB’s inference, training, concept inspection, and revision processes. Let us
now formally introduce these processes.

In this work, we refer to a concept as "the label of a set of things that have
something in common" [1]. This definition can be applied on different scales
of a visual scene: on an image level (_e.g._ , an image of a park), an object
level (_e.g._ , a tree vs. a bird) or an object-factor level (_e.g._ , the
color of a bird). Overall, we consider a set of unlabeled images
$X:=(x_{1},\cdots,x_{N})\in\mathbb{R}^{N\times D}$ with
$x_{i}\in\mathbb{R}^{D}$, $N\in\mathbb{N}$ and $D\in\mathbb{N}$ (for
simplicity, we drop the image index notation in the following). Briefly, given
an image $x$, NCB derives a symbolic representation, $c$, which expresses the
concepts of the objects in the image, _i.e._ , object-factor level concepts.
Herefore, NCB infers a block-slot encoding, $z$, of the image and performs a
retrieval-based discretization step to finally infer concept-slot encodings,
$c$. We begin by introducing the inference procedure of NCB. We hereby assume
that NCB’s components have already been trained and will introduce details of
the training procedure subsequently.

###  3.1 Inferring Concept-Slot Representations

Obtaining Continuous Block-Slot-Encodings. Consider an image $x\in X$. The
first component of NCB, the soft binder, is represented via a block-slot
encoder, $g_{\theta}:x\rightarrow z\in\mathbb{R}^{N_{S}\times N_{B}\times
D_{B}}$, where $g$ is parameterized by $\theta$ (unless required, this
notation is omitted in the following). The soft binder transforms an input
image into a latent, continuous block-slot representation, where $N_{S}$
represents the number of slots, $N_{B}$ the number of blocks per slot, and
$D_{B}$ the dimension of a block. It hereby utilizes two important types of
binding mechanisms: spatial and factor binding. Spatial binding ensures
spatial modularity across the entire scene and is achieved via slot attention
[36]. As a result, each object in the image is represented in a specific slot,
$z_{i}$, for slot $i$. Factor binding, on the other hand, as introduced by
[56] ensures that different object factors (_e.g._ , object attributes like
color) are separately encoded in different blocks of a slot encoding, _i.e._ ,
$z_{i}^{j}$ for block $j$. These two binding mechanisms jointly perform
valuable object- and factor-based image processing. Overall, the resulting
block-slot encodings represent continuous object-centric encodings of an input
image (_cf._ Fig. 2), with objects encoded in slots and object factors encoded
in blocks within the slots.

Obtaining Discrete Concept-Slot-Encodings. The role of NCB’s second processing
component, the hard binder, is to transform these continuous block-slot
encodings into expressive, yet discrete concept-slot encodings. Specifically,
the hard binder is represented via a retrieval encoder, $f$, (_cf._ Fig. 2).
It processes the block-slot encodings, $z$, into a set of discrete concept-
slot encodings, $c$. We refer to its underlying binding mechanism as concept
binding. In detail, $f$ represents a function $f_{\mathcal{R}}:z\rightarrow
c\in\mathbb{N}^{N_{S}\times N_{B}}$ that is parameterized by a retrieval
corpus $\mathcal{R}$. This retrieval corpus consists of a tuple of sets
$\mathcal{R}:=[\mathcal{R}^{1},\cdots,\mathcal{R}^{N_{B}}]$ where, in turn,
each
$\mathcal{R}^{j}:=\\{(\texttt{enc}_{l}^{j},v_{l}):l\in\\{1,...,|R^{j}|\\}\\}$
contains tuples of encodings, $\texttt{enc}_{l}^{j}\in\mathbb{R}^{D_{B}}$, and
corresponding discrete values, $v_{l}\in\\{1,\cdots,N_{C}\\}$. Importantly,
$\texttt{enc}_{l}^{j}$ consists of a representative block encoding of a
specific concept. Such a concept is determined in NCB’s training phase and
represents a distinct group of similar block representations (_cf._ Fig. 2,
formal details below). The value $v_{l}$ thus represents the symbol identifier
that corresponds to the specific concept that $\texttt{enc}_{l}^{j}$ is
associated with. Hereby, each block contains up to $N_{C}\in\mathbb{N}$
different concepts. To infer the concept symbol of a sample’s block-slot
encoding, NCB compares $z_{i}^{j}$ to the encodings in the corresponding
block’s retrieval corpus, $\mathcal{R}^{j}$, and selects the most likely
concept. Specifically, given a distance metric $d(\cdot,\cdot)$ and the block-
slot encoding, $z_{i}^{j}$, the selection function
$s_{\mathcal{R}}:z_{i}^{j}\rightarrow l\in\mathbb{N}$ finds the index $l$ of
the closest encoding in the retrieval corpus:
$s_{\mathcal{R}}(z_{i}^{j})=\operatorname*{argmin}_{l}d(\texttt{enc}_{l}^{j},z_{i}^{j})$
such that $(\texttt{enc}_{l}^{j},v_{l})\in\mathcal{R}^{j}$. This results in
the concept representation for slot $i$ and block $j$ denoted as
$c_{i}^{j}:=v_{s_{\mathcal{R}}(z_{i}^{j})}$. For the slot $i$, we thus obtain
$c_{i}:=[c_{i}^{1},\cdots,c_{i}^{N_{B}}]$. The final concept-slot encoding is
given by $c:=f_{\mathcal{R}}(z)=[c_{1},\cdots,c_{N_{S}}]$. We further refer to
Sec. A.1 for details on an alternative, top-k selection function. Let us now
move on to NCB’s training procedure.

###  3.2 Unsupervised Concept Learning via NCB

The training procedure of the Neural Concept Binder is separated into two
subsequent steps where we provide an overview here and details in Sec. A.2. We
formally describe these steps using the pseudo-code in Alg. 1. The first step
consists of optimizing the encoder, $g$, to provide object-factorised block-
slot encodings. It is optimized for unsupervised image reconstruction based on
the decoder model,
$g^{\prime}_{\theta^{\prime}}:z\rightarrow\tilde{x}\in\mathbb{R}^{D}$ (_cf._
Fig. 2) and utilising a mean squared error loss:
$L=L_{\mathrm{MSE}}(x,g^{\prime}(g(x)))$. The goal of NCB’s second training
step is to obtain the retrieval corpus, $\mathcal{R}$. This procedure is based
on obtaining an optimal clustering of block encodings via an unsupervised
clustering model, $h$, and distilling the resulting information from $h$ into
explicit representations in the retrieval corpus. Thus, for each block $j$ a
clustering model $h_{\phi^{j}}$ (_cf._ Fig. 2) is fit to identify a
(potentially overparameterised) optimal set of clusters within a set of block
encodings. This is based on an unsupervised criteria, _e.g._ , a density-based
score [44], and results in $N_{C}\in\mathbb{N}$ clusters. Next, for each
cluster, $v\in\\{1,\cdots,N_{C}\\}$, representative block encodings,
$\texttt{enc}^{j}$, are extracted from $h$. Such an encoding represents either
an averaged prototype or instance-based exemplar encoding. The corresponding
tuples $(\texttt{enc}^{j},v)$ are explicitly stored in the retrieval corpus
$\mathcal{R}^{j}$, and the final retrieval corpus consists of the set of
individual corpora for each block, $R=[R^{1},\cdots,R^{N_{B}}]$. Through this
training procedure, NCB learns to unsupervisedly categorize the object-factor
information from the latent encoding space of the soft binder and stores this
information in a structured, symbolic, and accessible way in the hard binder’s
retrieval corpus. We refer to the resulting clusters of each block as NCB’s
concepts. In the following we denote concepts with a capital letter for the
block and a natural number for the category id, _e.g._ , $A3$. We note that in
practice, it is further possible to finetune the block-slot encoder, $g$,
through supervision from the hard binder (_cf._ Fig. 2), _e.g._ , once initial
categories have been identified, and can be achieved via a standard supervised
approach. Ultimately, this allows for dynamically finetuning NCB’s concept
representations. Let us now introduce how human stakeholders can inspect and
revise NCB’s learned concepts.

Figure 3: NCB’s concept space is inherently inspectable. A human stakeholder
can easily inspect the concept space by asking a diverse set of questions. For
example, NCB answers interventional questions (iii) via generating images with
selectively modified concepts.

###  3.3 Inspecting NCB’s Concepts

One key advantage of NCB’s concept representations is their inherent
readability and inspectability. Specifically, NCB inherently enables: (i)
implicit, (ii) comparative, (iii) interventional and (iv) similarity-based
inspection (_cf._ Fig. 3). Where the first three aim at investigating NCB’s
explicit, symbolic concept space (stored in $\mathcal{R}$), the last one aims
at investigating its latent, continuous concept space (stored in $\theta$).

(i) Implicit inspection queries the model to provide a set of examples for a
specific concept. Essentially, this answers the question "What are examples of
this concept?". NCB answers this question in two ways: by providing samples
from the retrieval corpus corresponding to exemplars of the concept or by
identifying additional data samples belonging to the concept at hand.

(ii) Comparative inspection, on the other hand, allows comparing two
specifically different concepts, _e.g._ , "Why does this object depict concept
H5 and not concept H1?". NCB hereby provides examples for both concepts for
the user to compare and potentially identify dissimilar properties.
Ultimately, this form of inspection allows to answer questions of the form
"Why not …?" and represents a valuable tool for in-depth and targeted concept
inspection.

(iii) Interventional inspection allows to answer questions such as "What if
this object would have concept H1?" To answer this question, NCB utilizes its
decoder $g^{\prime}$. Specifically, by swapping the block $z_{i}^{j}$ of a
data sample’s block-slot encoding with that of a representative sample,
$(\texttt{enc}_{l}^{j},v_{l})\in\mathcal{R}^{j}$, NCB can provide an
interventional image reconstruction, from which the effect of the swapped
concept can be observed. Ultimately, this form of inspection allows to answer
important questions of the form "What if …?".

Finally, (iv) Similarity inspection allows inspecting NCB’s continuous
encoding space on a more global level (in comparison to the more symbolic,
sample-based inspection above), _e.g._ , "What are similar concepts to this
concept?". Specifically, NCB’s distance metric $d$ directly provides
information about the similarity between concepts in the continuous
representation. Inspecting the block-slot encoding space thus allows to
identify a suboptimal soft binding, _e.g._ , when block encodings are similar
according to $g$ but not according to the human stakeholder. Overall, these
inspection mechanisms allow a human stakeholder to ask a diverse set of
questions concerning a model’s learned concepts.

###  3.4 Revising NCB’s Concepts

Let us now describe how a human stakeholder can revise NCB’s concept space.
Below, we provide details on the three main actions for symbolic revision
(_i.e._ , revision on the representations in $\mathcal{R}$): (i) merging, (ii)
deleting, or (iii) adding information. These actions can be performed on a
single encoding or on a concept level and essentially represent a form of
"reorganization" of information stored in $\mathcal{R}$. Furthermore, we
provide details on how to (iv) revise the continuous latent space, which
essentially requires finetuning of $g$’s parameters.

(i) Merge Concepts: In the case that $\mathcal{R}$ contains multiple concepts
that, according to human knowledge, represent a joint underlying concept
(_e.g._ , two concepts for purple in Fig. 3 (right)) it is easy to update the
model’s internal representations by replacing the concept symbols of one
concept with those of the second concept. Formally, (in the case of two
concepts) if $v_{l}$ and $v_{m}$ of block $j$ should be merged, the
information stored in $\mathcal{R}^{j}$ is updated via
$\forall(\texttt{enc}_{l}^{j},v_{l}):v_{l}\rightarrow v_{m}$.

(ii) Delete Encodings or Concepts: If $\mathcal{R}$ contains an encoding for a
specific concept that does not match the other encodings of that concept,
_e.g._ , a misplaced exemplar, this encoding can simply be removed via
$\texttt{remove}((\texttt{enc}_{l}^{j},v_{l}))$. Accordingly, if an entire
concept is identified as suboptimal, it is possible to delete all
corresponding encodings of concept $m$ in $\mathcal{R}^{j}$ as
$\forall(\texttt{enc}_{l}^{j},v_{l}):v_{l}=m\implies\texttt{remove}((\texttt{enc}_{l}^{j},v_{l}))$.

(iii) Add Encodings or Concepts: In a similar fashion, it is also possible to
add encodings to $\mathcal{R}$. For instance, if a specific concept is not
sufficiently well captured via the encodings in $\mathcal{R}^{j}$, one can add
an additional encoding, $\hat{\texttt{enc}}_{l}^{j}$, for the concept, $m$,
via $\texttt{add}((\hat{\texttt{enc}}_{l}^{j},m))$. Accordingly, it is also
possible to add encodings for an entire concept, $\hat{v}_{l}$:
$\forall(\hat{\texttt{enc}}_{l}^{j},\hat{v}_{l})\implies\texttt{add}((\texttt{enc}_{l}^{j},m))$

(iv) Revise the (Continuous) Latent Space: Lastly, if the soft binder provides
suboptimal object- and factor-level block-slot encodings, it is further
possible to integrate revisory feedback on the soft binder’s continuous latent
space. This can be achieved via additional finetuning of the soft binder’s
parameters, $\theta$, _e.g._ , via standard forms of weak supervision [35, 60]
or interactive learning [59, 52].

Table 1: Comparison of different approaches for concept learning. Hereby, we
differentiate based on the following categories: whether a method (1) is
learned in an unsupervised fashion, (2) provides object-level concepts (_i.e._
, can explicitly process multiple objects), (3) provides factor-level concepts
(_i.e._ , the color green), (4) provides continuous concept encodings, (5)
provides discrete concept encodings, (6) provides inherently inspectable and
(7) revisable concept representations.

Method | Unsupervised | Obj. level | Factor level | Cont. encs | Disc. encs | Inspectable | Revisable
---|---|---|---|---|---|---|---
CBM [28] | ✗ | ✗ | ✓ | ✗ | ✓ | ✓ | ✓
NeSyCL [59] | ✗ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓
GlanceNets [42] | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓
VAE [27] | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗
VQ-VAE [66] | ✓ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗
SA [36] | ✓ | ✓ | ✗ | ✓ | ✗ | (✓) | ✗
SysBinder [56] | ✓ | ✓ | ✓ | ✓ | ✗ | (✓) | ✗
Neural Concept Binder | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓

In summary, our novel Neural Concept Binder framework fulfills several
important desiderata for concept learning (_cf._ Tab. 1). Specifically, NCB
learns concepts in an unsupervised fashion that are structured on both an
object (_i.e._ , can explicitly process multiple objects) and factor-level.
Furthermore, next to standard continuous encodings, NCB also provides discrete
concept representations, which are crucial for interpretability and
integration into symbolic computations. Lastly, NCB’s concept space is
inspectable and revisable, essential for unsupervised learned concept
representations.

##  4 Experimental Evaluations

In our evaluations, we investigate the potential of NCB’s soft and hard
binding mechanisms in unsupervised concept learning and its integration into
downstream tasks. Notably, NCB encompasses concept processing between both of
its components (soft binder and hard binder) whereby the direction "soft
binder $\leftarrow$ hard binder" (_cf._ Fig. 2) represents a standard approach
(_i.e._ , supervised learning of the soft binder’s encoding space via symbolic
concept labels, _e.g._ , [28, 59]). Therefore, we focus our evaluations on
NCB’s more novel processing direction, "soft binder $\rightarrow$ hard
binder". We aim to answer the following research questions: (Q1) Does NCB
provide expressive and distinct encodings? (Q2) Can NCB be combined with
symbolic methods to solve complex downstream tasks? (Q3) Can NCB’s learned
concepts be revised to improve suboptimal behaviour? (Q4) Can NCB be combined
with subsymbolic methods to transparently solve complex downstream tasks?

Data. We focus our evaluations on different variations of the popular CLEVR
dataset. Specifically, we investigate (Q1 & Q3) in the context of the CLEVR
[24] and CLEVR-Easy [56] datasets. For investigating the integration of NCB
into symbolic modules (Q2), we utilize our novel CLEVR-Sudoku puzzles
introduced in the following. Finally, to evaluate the integration of NCB into
subsymbolic modules (Q4), we evaluate on confounded and non-confounded
variants of the CLEVR-Hans3 dataset [59]. We provide further details on these
datasets in the supplements (_cf._ Suppl. C).

CLEVR-Sudoku. To investigate the potential of integrating NCB’s discrete
concept representations into symbolic downstream tasks, we introduce the novel
CLEVR-Sudoku dataset. It represents a challenging visual puzzle requiring both
visual object perception and reasoning capabilities. Each sample of the
dataset (_cf._ Fig. 4 for an example puzzle) consists of a Sudoku puzzle that
is (partially filled) with CLEVR-based images [24] and additional example
images that depict the mapping of relevant object properties to digits.
Specifically, each digit of the Sudoku is replaced by an image of an object.
All objects representing the same digit share a set of common properties,
_e.g._ , in Fig. 4 all objects replacing "1"s are yellow spheres. We introduce
two variants of CLEVR-Sudoku: Sudoku CLEVR-Easy and Sudoku CLEVR. In the first
one, all objects are large and metallic, leaving only shape and color as
properties for the digits. In Sudoku CLEVR, the size and material of the CLEVR
objects are also relevant for the digits. Moreover, up to 10 example images
are provided per digit mapping; the fewer examples provided, the harder it

Figure 4: Example from CLEVR-Sudoku. Each digit is represented by CLEVR
objects with the same attribute combination. The objective is to solve the
Sudoku only based on the initial grid of CLEVR images and the digit mapping of
candidate examples.

is to learn the mapping. The initial state and digit-attribute mapping vary
for each sample. One specific intricacy of CLEVR-Sudoku is that a Sudoku can
only be solved if all subcell images are correctly mapped to the digits. Even
one mistake can lead to an unsolvable Sudoku. Thus, in comparison to standard
Sudoku puzzles, which mainly require deductive reasoning, solving CLEVR-Sudoku
additionally requires complex object recognition and the ability to map these
visual concept perceptions to the task concepts (_i.e._ the 9 digits of
Sudoku). We refer to Suppl. B for further details.

Models. For our evaluations, we instantiate Neural Concept Binder based on the
SysBinder model [56] for the soft binder encoder, $g$, and hdbscan [8, 9] for
the clustering model, $h$. We refer to Sec. A.3 for all additional details. In
the context of (Q1), we compare NCB’s results to those of four different
variations of the SysBinder model [56]. We denote the original SysBinder
configuration as SysBinder (cont.) (provides continuous block-slot encodings).
For SysBinder, we discretize SysBinder’s continuous encodings at inference
time (via an $\operatorname*{argmin}$ operation over SysBinder’s internal
codebooks). SysBinder (hard) is trained from the start to provide discrete
encodings (via a low codebook softmax temperature). SysBinder (step) is
trained by step-wise decreasing this temperature (_cf._ Suppl. D for details).
For evaluations on CLEVR-Sudoku (Q2+Q3), we first infer NCB’s discrete
concept-slot encodings for a puzzle’s candidate examples. These encodings
(together with the corresponding digit labels) are passed to a symbolic
classifier, which is fit to predict the digits from the encodings. Next, this
classifier infers the digits from the image of each subcell in the puzzle’s
initial state. These are used by a constraint propagation and search-based
algorithm [46, 6] to solve the puzzle (_cf._ Sec. E.2 for details). We denote
the combination of the symbolic classifier and constraint solver as solver. We
compare the solver’s performance when provided ground-truth (GT) object-
property labels (GT concepts), encodings from a supervised trained slot
attention encoder [36] (SA (supervised)) and the discrete encodings from
SysBinder (denoted here as SysBinder (unsupervised)). For classification
evaluations (Q4), we evaluate a configuration in which a set transformer
classifier [31] is provided NCB’s concept encodings (NCB + NN) and makes a
final class prediction based on these (_cf._ Sec. E.4). We compare to SA + NN,
_i.e._ , where a supervised trained slot attention encoder [36] provides
object-property predictions.

Metrics. We evaluate all models based on their accuracies on held-out test
splits, each with 3 seeded reruns. We provide average accuracies and standard
deviations over these. When assessing the expressiveness of NCB’s concept-slot
encodings (Q1), we evaluate the accuracy for object-property prediction. When
evaluating the performance of the downstream tasks, we provide the percentage
of solved CLEVR-Sudokus (Q2) and the classification accuracy on the test set
of CLEVR-Hans3 (Q4).

Table 2: NCB’s concept encodings are expressive despite information
bottleneck. Classifying object properties from different continuous and
discrete encodings. The classifier is provided with different amounts of
training sample encodings. The best (“$\bullet$”) and runner-up (“$\circ$”)
results are bold.

Dataset | N Train |  SysBinder $($cont.$)$ | SysBinder | SysBinder (hard) | SysBinder (step) | Neural Concept Binder
---|---|---|---|---|---|---
CLEVR- Easy | N=2000 | $\mathbf{\bullet\ 99.83}\mbox{\scriptsize$\pm 0.24$}$ | $92.49\mbox{\scriptsize$\pm 5.45$}$ | $22.92\mbox{\scriptsize$\pm 0.00$}$ | $95.76\mbox{\scriptsize$\pm 4.92$}$ | $\mathbf{\circ\ 99.02}\mbox{\scriptsize$\pm 1.00$}$
N=200 | $\mathbf{\bullet\ 99.20}\mbox{\scriptsize$\pm 0.41$}$ | $87.90\mbox{\scriptsize$\pm 8.05$}$ | $22.92\mbox{\scriptsize$\pm 0.00$}$ | $92.42\mbox{\scriptsize$\pm 7.32$}$ | $\mathbf{\circ\ 98.50}\mbox{\scriptsize$\pm 1.80$}$
N=50 | $\mathbf{\circ\ 91.13}\mbox{\scriptsize$\pm 4.21$}$ | $78.41\mbox{\scriptsize$\pm 8.69$}$ | $22.92\mbox{\scriptsize$\pm 0.00$}$ | $70.64\mbox{\scriptsize$\pm 11.89$}$ | $\mathbf{\bullet\ 95.87}\mbox{\scriptsize$\pm 2.93$}$
N=20 | $\mathbf{\circ\ 64.88}\mbox{\scriptsize$\pm 10.89$}$ | $62.61\mbox{\scriptsize$\pm 7.18$}$ | $22.92\mbox{\scriptsize$\pm 0.00$}$ | $54.61\mbox{\scriptsize$\pm 9.57$}$ | $\mathbf{\bullet\ 94.22}\mbox{\scriptsize$\pm 4.11$}$
CLEVR | N=2000 | $\mathbf{\bullet\ 98.86}\mbox{\scriptsize$\pm 1.15$}$ | $86.22\mbox{\scriptsize$\pm 10.40$}$ | $36.46\mbox{\scriptsize$\pm 0.00$}$ | $88.90\mbox{\scriptsize$\pm 14.81$}$ | $\mathbf{\circ\ 97.26}\mbox{\scriptsize$\pm 2.67$}$
N=200 | $\mathbf{\bullet\ 97.61}\mbox{\scriptsize$\pm 2.58$}$ | $81.13\mbox{\scriptsize$\pm 12.39$}$ | $36.46\mbox{\scriptsize$\pm 0.00$}$ | $83.17\mbox{\scriptsize$\pm 17.05$}$ | $\mathbf{\circ\ 96.80}\mbox{\scriptsize$\pm 3.01$}$
N=50 | $\mathbf{\circ\ 93.25}\mbox{\scriptsize$\pm 4.62$}$ | $61.67\mbox{\scriptsize$\pm 8.51$}$ | $36.46\mbox{\scriptsize$\pm 0.00$}$ | $68.81\mbox{\scriptsize$\pm 17.74$}$ | $\mathbf{\bullet\ 94.67}\mbox{\scriptsize$\pm 4.65$}$
N=20 | $\mathbf{\circ\ 79.11}\mbox{\scriptsize$\pm 8.75$}$ | $49.79\mbox{\scriptsize$\pm 6.73$}$ | $36.46\mbox{\scriptsize$\pm 0.00$}$ | $58.58\mbox{\scriptsize$\pm 16.09$}$ | $\mathbf{\bullet\ 88.57}\mbox{\scriptsize$\pm 4.68$}$

###  4.1 Evaluations

Discrete, yet expressive representations (Q1). In (Q1), we investigate how
much valuable information NCB’s discrete concept-slot encodings contain,
despite NCB’s inherent information bottleneck. For this purpose, we train a
classifier on NCB’s encodings to predict corresponding object-property labels,
_e.g._ , the color green (_cf._ Sec. E.1 for details). In Tab. 2 we provide
results for the CLEVR-Easy and CLEVR dataset with a classification training
set of 2000, 200, 50, or 20 encodings. Focusing first on the results for
$N=2000$, we observe that, as expected, the continuous representation of the
original Sysbinder model contains the most information compared to all
discrete encodings. Remarkably, however, NCB’s discrete concept
representations are nearly on par with the continuous encodings. This is
particularly remarkable given the immense information bottleneck in NCB
222SysBinder (cont.) provides $2048$-sized continuous encodings, whereas NCB
provides discrete encodings of size $\leq 16$.. Moreover, we observe that
NCB’s encodings greatly outperform all other forms of discrete
representations. Focusing now on the results when the classifier is trained on
data subsets, we observe a substantial degradation in performance via
encodings of any of the discrete baselines, and the continuous encodings. In
stark contrast, when classifying based on NCB’s encodings, the accuracy
remains nearly constant even with 1% of the initial training samples. We
provide additional ablations on the effect of concept encoding types and NCB’s
selection function in Sec. F.1 as well as an ablation analysis on the effect
of suboptimal behavior from NCB’s individual components in Sec. F.2. We
further refer to Sec. F.3 for an analysis of NCB’s obtained concept space as
well as Suppl. G for examples of learned concepts. Overall, our results
provide evidence for the expressiveness of NCB’s concept encodings despite
their enormous information bottleneck. Furthermore, our results for $N\leq
200$ suggest that the information stored in the NCB’s encodings is easier to
generalize compared to the baselines. We thus answer Q1 affirmatively.

Figure 5: NCB’s unsupervised concepts allow solving symbolic puzzles. Accuracy
of solved Sudokus via different discrete concept encodings on Sudoku CLEVR-
Easy and Sudoku CLEVR (left sides). Additional revision on NCB’s concepts
leads to improved performances (right sides).

Utilising unsupervised concepts for solving visual Sudoku (Q2). In our
following evaluations, we investigate the potential of NCB’s representations
for solving complex reasoning tasks via their integration into symbolic
computations. We base these evaluations on our novel CLEVR-Sudoku dataset. We
report the percentage of solved puzzles for CLEVR-Sudoku in Fig. 5. Hereby, it
is important to note that the solver can only solve a puzzle if each of the
images of the initial state has been classified correctly. Thus, the results
of Fig. 5 represent "all-or-nothing" results. Focusing on the results left of
the dashed lines, we observe that the symbolic module can solve every puzzle
with GT concepts, even when only one example image is provided. Interestingly,
performance drops drastically when provided with encodings of SA (supervised).
This highlights the difficulty of the CLEVR-Sudoku puzzles: minor errors in
digit prediction can lead to major errors in solving the underlying puzzle.
When comparing the performance via encodings from the two unsupervised models,
we observe quite strong performances via NCB’s concept encodings. _e.g._ ,
they allow to solve $\approx 50\%$ for the 10 example configurations (over
both data settings) in comparison to $\approx 61\%$ for SA (supervised). In
contrast, when provided SysBinder’s encodings the solver fails over all Sudoku
variations. This demonstrates the effectiveness of NCB’s binding mechanisms
over solely those of the SysBinder approach. We refer to Sec. F.4 for further
discussions as well as quantitative digit classification results. Overall, our
evaluations highlight the potential of NCB’s unsupervised concept encodings
for solving complex, symbolic downstream tasks. We therefore answer Q2
affirmatively.

Easily revising NCB’s concepts (Q3). In our next evaluations, we illustrate
the potential of NCB’s revision procedures. As revision on the continuous
latent space of NCB’s soft binder is analogous to existing approaches (_e.g._
, [52, 49, 60]), we focus these evaluations on the novel, NCB-specific forms
of symbolic revision, _i.e._ , on the hard binder’s concept space. We hereby
illustrate the first two forms of symbolic revision (removing and merging
concept information) via two forms of feedback: based on a pretrained vision-
language model (here via GPT-4 [47]) and simulated human feedback. In both
cases, we ask the revisory agent to identify which concepts in each block
should be removed or merged based on exemplar images of each concept, _i.e._ ,
implicit concept inspection (_cf._ Sec. E.3 for details). In Fig. 5 we provide
CLEVR-Sudoku performances when the NCB’s retrieval corpus has been updated
based on the different revisory agents (_i.e._ , NCB revised (GPT-4) and NCB
revised (human)). Interestingly, although we observe improvements in the few
example settings via GPT-4’s revisions, we observe a negative effect in the
settings with more digit examples. This is due to the suboptimal behavior of
GPT-4 in providing consistent object descriptions, such that too much concept
information is removed or merged and highlights the effect of “ill-informed”
feedback (_cf._ Sec. F.5). In contrast, when provided with human revision, we
observe a substantial boost in Sudoku performance, particularly in puzzle
configurations with few candidate examples. Moreover, via NCB’s similarity
inspection mechanism (_cf._ Sec. 3.3), a human stakeholder can easily identify
models that provide suboptimal soft binding processing. In this case, such
models can be excluded from further downstream evaluations (_cf._ NCB revised
(human)*) and must be refined via finetuning $g$’s parameters (_e.g._ , via
approaches of [52, 49, 60]). In Sec. F.6, we further investigate concept
revision via adding information. Overall, our results highlight the potential
and ease of revising NCB’s concept space. We thus answer Q3 positively.

Table 3: NCB’s unsupervised concept representations facilitate interpretable
neural computations. Explanations of a NN classifier trained on the
unsupervised concepts of NCB. Via NCB’s inherent inspection procedures a human
stakeholder can identify which concepts the classifier focuses on to make its
predictions and thus interpret the NN’s underlying decision rule.

GT Class Rule |  NN Expl. |  Human Inspection |  Human Interpretation
---|---|---|---
Large, gray cube |  C3 $\land$ H4 $\land$ K4 $\land$ O12 $\land$ P5 |  (Gray1) $\land$ (Red $\lor$ Gray2) $\land$ (Large) $\land$ (Gray3) $\land$ (Gray4) |  “A large gray object”
Small, metal cube |  B3 $\land$ D3 $\land$ H0 $\land$ I0 $\land$ K0 |  (Cube) $\land$ (Small1) $\land$ (Small2) $\land$ (Small3) $\land$ (Small4) |  “A small cube”
Large, blue sphere |  B0 $\land$ C6 $\land$ H3 $\land$ O0 $\land$ P1 |  (Sphere) $\land$ (Blue1) $\land$ (Blue2) $\land$ (Small $\lor$ Blue3) $\land$ (Blue4 $\lor$ Green $\lor$ Purple) |  “A blue sphere”

Utilising unsupervised concepts for understanding neural computations (Q4). In
our final evaluations, we investigate whether NCB’s concept encodings can make
subsymbolic computations more transparent. Therefore, we consider the task of
image classification on variations of the benchmark CLEVR-Hans3 dataset [59].
While the concept encodings in NCB + NN are trained unsupervised, they perform
on par with those of the supervised approach of [59] (_cf._ Sec. F.7). Even
more important, integrating NCB’s inherently inspectable concept
representations into neural computations can ellicit more transparent decision
processes. We illustrate this in Tab. 3, where we exemplify class-level

Figure 6: NCB’s unsupervised concept representations facilitate shortcut
mitigation. Test accuracy for classification via NN predictor when trained on
confounded images.

explanations of the classifier of NCB + NN (_cf._ Sec. E.4 for details). Via
NCB’s inspection mechanisms, a human stakeholder can now easily identify the
classifier’s internal decision rule for a class (_e.g._ , "a large gray
object"). Overall, this is a vital feature for the deployment of trustworthy
AI models in real-world scenarios and the significance of our results is, that
we can achieve this even when utilising unsupervised concept encodings.
Moreover, in Fig. 6 we investigate whether a NCB-based neural classifier can
be revised to mitigate the confounders in CLEVR-Hans3 (_cf._ Sec. E.4 and Sec.
F.8 for details). The confounding factor in the training set is the color
gray. We present the non-confounded test set accuracy in Fig. 6. We observe
that standard loss-based feedback via explanatory interactive learning (XIL)
[59] on the NN classifier’s explanations (\+ XIL on NN) can greatly reduce the
effect of the confounder. Alternatively, by simply zeroing the activations of
the undesired concept gray (\+ XIL on concepts) we even achieve improved
confounding mitigation results without the usual issues of joint optimization.
Overall, our results highlight the potential of integrating NCB’s unsupervised
concept representations for eliciting transparent and trustworthy subsymbolic
computations. We thus answer Q4 affirmatively.

Limitations. NCB largely benefits from qualitative initial block-slot
encodings. If these are lacking, the extracted concept-slot encodings lose
quality as well. An important next step to handle more complex visual inputs,
_e.g._ , video data, is the integration of recent approaches, _e.g._ , [11,
14]. Furthermore, due to the unsupervised nature of NCB’s training, additional
alignment of NCB’s concepts is inevitable for deployment in downstream tasks
[5]. Further, to build trust in NCB’s concept knowledge, human inspection is
required. Lastly, revisions are an integral part of NCB. However they rely on
humans to provide "good" feedback, as a malicious user can manipulate NCB’s
concepts. However, inspecting the concept space allows to track this
effectively.

##  5 Conclusions

In this work, we introduce the Neural Concept Binder framework for learning
visual object-factor concepts in an unsupervised fashion. Our evaluations
suggest that NCB’s specific binding mechanisms benefit learning expressive yet
discrete concept representations. Moreover, our evaluations highlight the
potential of integrating NCB’s inherently inspectable and revisable concept-
slot encodings into symbolic and neural modules. Interesting avenues for
future research are investigating the benefits of NCB’s concept
representations in the context of continual learning settings [7] as well as
integrating NCB’s basic level concepts into high-level concept learning
approaches [71], but also probabilistic logic programming approaches [57, 58].
Additionally, integrating learning signals from a downstream task into NCB’s
concept representations, _e.g._ , via differentiable clustering [67], is
important for improving the quality of initial concept encodings.

### Acknowledgments

The authors wish to thank Gautam Singh for initial help with SysBinder.
Furthermore, the authors wish to thank Cyprien Dzialo for his preliminary
results and insights on this project. This work was supported by the Priority
Program (SPP) 2422 in the subproject “Optimization of active surface design of
high-speed progressive tools using machine and deep learning algorithms“
funded by the German Research Foundation (DFG), the ”ML2MT” project from the
Volkswagen Stiftung and the ”The Adaptive Mind” project from the Hessian
Ministry of Science and Arts (HMWK). It has further benefited from the HMWK
projects ”The Third Wave of Artificial Intelligence - 3AI”, and Hessian.AI, as
well as the Hessian research priority program LOEWE within the project
WhiteBox, the ICT-48 Network of AI Research Excellence Center “TAILOR” (EU
Horizon 2020, GA No 952215) and the EU-funded “TANGO” project (EU Horizon
2023, GA No 57100431).

## References

  * Archer [1966] E James Archer.  The psychological nature of concepts.  In _Analyses of concept learning_ , pages 37–49. Elsevier, 1966.
  * Asai and Fukunaga [2018] Masataro Asai and Alex Fukunaga.  Classical planning in deep latent space: Bridging the subsymbolic-symbolic boundary.  In _Conference on Artificial Intelligence (AAAI)_ , pages 6094–6101. AAAI Press, 2018.
  * Barbiero et al. [2023] Pietro Barbiero, Gabriele Ciravegna, Francesco Giannini, Mateo Espinosa Zarlenga, Lucie Charlotte Magister, Alberto Tonda, Pietro Lio, Frédéric Precioso, Mateja Jamnik, and Giuseppe Marra.  Interpretable neural-symbolic concept reasoning.  In _International Conference on Machine Learning (ICML)_ , 2023\.
  * Barua et al. [2024] Adrita Barua, Cara Widmer, and Pascal Hitzler.  Concept induction using llms: a user experiment for assessment.  _CoRR_ , abs/2404.11875, 2024.
  * Bembenek and Murray [2024] Aaron Bembenek and Toby Murray.  Symbol correctness in deep neural networks containing symbolic layers.  _CoRR_ , abs/2402.03663, 2024.
  * Bessiere [2006] Christian Bessiere.  Constraint propagation.  In _Foundations of Artificial Intelligence_ , volume 2, pages 29–83. Elsevier, 2006.
  * Busch et al. [2024] Florian Peter Busch, Roshni Kamath, Rupert Mitchell, Wolfgang Stammer, Kristian Kersting, and Martin Mundt.  Where is the truth? the risk of getting confounded in a continual world.  _CoRR_ , abs/2402.06434, 2024.
  * Campello et al. [2013] Ricardo J. G. B. Campello, Davoud Moulavi, and Jörg Sander.  Density-based clustering based on hierarchical density estimates.  In _Advances in Knowledge Discovery and Data Mining (PAKDD)_ , 2013\.
  * Campello et al. [2015] Ricardo J. G. B. Campello, Davoud Moulavi, Arthur Zimek, and Jörg Sander.  Hierarchical density estimates for data clustering, visualization, and outlier detection.  _ACM Transactions on Knowledge Discovery from Data_ , 10(1):5:1–5:51, 2015.
  * DeGrave et al. [2021] Alex J. DeGrave, Joseph D. Janizek, and Su-In Lee.  AI for radiographic COVID-19 detection selects shortcuts over signal.  _Nature Machine Intelligence_ , 3(7):610–619, 2021.
  * Delfosse et al. [2023] Quentin Delfosse, Wolfgang Stammer, Thomas Rothenbacher, Dwarak Vittal, and Kristian Kersting.  Boosting object representation learning via motion and object continuity.  In _European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (PKDD / ECML)_ , 2023.
  * Dinu et al. [2024] Marius-Constantin Dinu, Claudiu Leoveanu-Condrei, Markus Holzleitner, Werner Zellinger, and Sepp Hochreiter.  Symbolicai: A framework for logic-based approaches combining generative models and solvers.  _CoRR_ , abs/2402.00854, 2024.
  * Dreyer et al. [2023] Maximilian Dreyer, Reduan Achtibat, Wojciech Samek, and Sebastian Lapuschkin.  Understanding the (extra-)ordinary: Validating deep model decisions with prototypical concept-based explanations.  _CoRR_ , abs/2311.16681, 2023.
  * Elsayed et al. [2022] Gamaleldin F. Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C. Mozer, and Thomas Kipf.  Savi++: Towards end-to-end object-centric learning from real-world videos.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Feldman [2013] Jerome Feldman.  The neural binding problem (s).  _Cognitive neurodynamics_ , 7:1–11, 2013.
  * Ghandeharioun et al. [2022] Asma Ghandeharioun, Been Kim, Chun-Liang Li, Brendan Jou, Brian Eoff, and Rosalind W. Picard.  DISSECT: disentangled simultaneous explanations via concept traversals.  In _International Conference on Learning Representations (ICLR)_ , 2022.
  * Ghosh et al. [2023] Shantanu Ghosh, Ke Yu, Forough Arabshahi, and Kayhan Batmanghelich.  Dividing and conquering a blackbox to a mixture of interpretable models: Route, interpret, repeat.  In _International Conference on Machine Learning (ICML)_ , 2023\.
  * Greff et al. [2020] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber.  On the binding problem in artificial neural networks.  _CoRR_ , abs/2012.05208, 2020.
  * Gupta and Narayanan [2024] Avani Gupta and P. J. Narayanan.  A survey on concept-based approaches for model improvement.  _CoRR_ , abs/2403.14566, 2024.
  * Hersche et al. [2023] Michael Hersche, Mustafa Zeqiri, Luca Benini, Abu Sebastian, and Abbas Rahimi.  A neuro-vector-symbolic architecture for solving raven’s progressive matrices.  _Nature Machine Intelligence_ , 5(4):363–375, 2023.
  * Huang et al. [2022] Haiyang Huang, Zhi Chen, and Cynthia Rudin.  Segdiscover: Visual concept discovery via unsupervised semantic segmentation.  _CoRR_ , abs/2204.10926, 2022.
  * Jang et al. [2017] Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  In _International Conference on Learning Representations (ICLR)_ , 2017.
  * Jin et al. [2023] Chen Jin, Ryutaro Tanno, Amrutha Saseendran, Tom Diethe, and Philip Teare.  An image is worth multiple words: Learning object level concepts using multi-concept prompt learning.  _CoRR_ , abs/2310.12274, 2023.
  * Johnson et al. [2017] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick.  CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Kambhampati et al. [2022] Subbarao Kambhampati, Sarath Sreedharan, Mudit Verma, Yantian Zha, and Lin Guan.  Symbols as a lingua franca for bridging human-ai chasm for explainable and advisable AI systems.  In _Conference on Artificial Intelligence (AAAI)_ , 2022.
  * Kim et al. [2023] Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon.  Probabilistic concept bottleneck models.  In _International Conference on Machine Learning (ICML)_ , 2023\.
  * Kingma and Welling [2019] Diederik P. Kingma and Max Welling.  An introduction to variational autoencoders.  _Foundations and Trends in Machine Learning_ , 12(4):307–392, 2019.
  * Koh et al. [2020] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang.  Concept bottleneck models.  In _International Conference on Machine Learning (ICML)_ , 2020\.
  * Lage and Doshi-Velez [2020] Isaac Lage and Finale Doshi-Velez.  Learning interpretable concept-based models with human feedback.  _CoRR_ , abs/2012.02898, 2020.
  * Laugel et al. [2019] Thibault Laugel, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and Marcin Detyniecki.  The dangers of post-hoc interpretability: Unjustified counterfactual explanations.  In _International Joint Conference on Artificial Intelligence (IJCAI)_ , 2019.
  * Lee et al. [2019] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh.  Set transformer: A framework for attention-based permutation-invariant neural networks.  In _International Conference on Machine Learning (ICML)_ , 2019\.
  * Liu et al. [2024] Ruizhe Liu, Qian Luo, and Yanchao Yang.  Infocon: Concept discovery with generative and discriminative informativeness.  2024\.
  * Locatello et al. [2019] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem.  Challenging common assumptions in the unsupervised learning of disentangled representations.  In _International Conference on Machine Learning (ICML)_ , 2019\.
  * Locatello et al. [2020a] Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen.  Weakly-supervised disentanglement without compromises.  In _International conference on machine learning (ICML)_ , 2020a.
  * Locatello et al. [2020b] Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem.  Disentangling factors of variations using few labels.  In _International Conference on Learning Representations (ICLR)_ , 2020b.
  * Locatello et al. [2020c] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf.  Object-centric learning with slot attention.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020c.
  * Lorello and Lippi [2023] Luca Salvatore Lorello and Marco Lippi.  The challenge of learning symbolic representations.  In _International Workshop on Neural-Symbolic Learning and Reasoning_ , 2023.
  * MacQueen et al. [1967] James MacQueen et al.  Some methods for classification and analysis of multivariate observations.  In _Berkeley Symposium on Mathematical Statistics and Probability_ , 1967.
  * Maddison et al. [2017] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh.  The concrete distribution: A continuous relaxation of discrete random variables.  In _International Conference on Learning Representations (ICLR)_ , 2017.
  * Mahinpei et al. [2021] Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan.  Promises and pitfalls of black-box concept learning models.  _CoRR_ , abs/2106.13314, 2021.
  * Mao et al. [2019] Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu.  The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision.  In _International Conference on Learning Representations (ICLR)_ , 2019.
  * Marconato et al. [2022] Emanuele Marconato, Andrea Passerini, and Stefano Teso.  Glancenets: Interpretable, leak-proof concept-based models.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Moayeri et al. [2023] Mazda Moayeri, Keivan Rezaei, Maziar Sanjabi, and Soheil Feizi.  Text-to-concept (and back) via cross-model alignment.  In _International Conference on Machine Learning (ICML)_ , 2023\.
  * Moulavi et al. [2014] Davoud Moulavi, Pablo A. Jaskowiak, Ricardo J. G. B. Campello, Arthur Zimek, and Jörg Sander.  Density-based clustering validation.  In _International Conference on Data Mining_ , 2014.
  * Nielsen [2016] Frank Nielsen.  Hierarchical clustering.  _Introduction to HPC with MPI for Data Science_ , pages 195–211, 2016\.
  * Norvig [2006] Peter Norvig.  Solving every sudoku puzzle.  _URL http://norvig. com/sudoku. html_ , 2006.
  * OpenAI [2023] OpenAI.  GPT-4 technical report.  _CoRR_ , abs/2303.08774, 2023.
  * Pan et al. [2023] Bo Pan, Zhenke Liu, Yifei Zhang, and Liang Zhao.  Surrocbm: Concept bottleneck surrogate models for generative post-hoc explanation.  _CoRR_ , abs/2310.07698, 2023.
  * Ross et al. [2017] Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Velez.  Right for the right reasons: Training differentiable models by constraining their explanations.  In _International Joint Conference on Artificial Intelligence (IJCAI)_ , 2017.
  * Santoro et al. [2018] Adam Santoro, Felix Hill, David G. T. Barrett, Ari S. Morcos, and Timothy P. Lillicrap.  Measuring abstract reasoning in neural networks.  In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 4477–4486. PMLR, 2018.
  * Sawada and Nakamura [2022] Yoshihide Sawada and Keigo Nakamura.  Concept bottleneck model with additional unsupervised concepts.  _IEEE Access_ , 10:41758–41765, 2022.
  * Schramowski et al. [2020] Patrick Schramowski, Wolfgang Stammer, Stefano Teso, Anna Brugger, Franziska Herbert, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, and Kristian Kersting.  Making deep neural networks right for the right scientific reasons by interacting with their explanations.  _Nature Machine Intelligence_ , 2(8):476–486, 2020.
  * Schut et al. [2023] Lisa Schut, Nenad Tomasev, Tom McGrath, Demis Hassabis, Ulrich Paquet, and Been Kim.  Bridging the human-ai knowledge gap: Concept discovery and transfer in alphazero.  _CoRR_ , abs/2310.16410, 2023.
  * Shu et al. [2020] Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Weakly supervised disentanglement with guarantees.  In _International Conference on Learning Representations (ICLR)_ , 2020.
  * Singh et al. [2022] Gautam Singh, Fei Deng, and Sungjin Ahn.  Illiterate DALL-E learns to compose.  In _International Conference on Learning Representations (ICLR)_ , 2022.
  * Singh et al. [2023] Gautam Singh, Yeongbin Kim, and Sungjin Ahn.  Neural systematic binder.  In _International Conference on Learning Representations (ICLR)_ , 2023.
  * Skryagin et al. [2022] Arseny Skryagin, Wolfgang Stammer, Daniel Ochs, Devendra Singh Dhami, and Kristian Kersting.  Neural-probabilistic answer set programming.  In _International Conference on Principles of Knowledge Representation and Reasoning (KR)_ , 2022.
  * Skryagin et al. [2023] Arseny Skryagin, Daniel Ochs, Devendra Singh Dhami, and Kristian Kersting.  Scalable neural-probabilistic answer set programming.  _Journal of Artificial Intelligence Research_ , 78:579–617, 2023.
  * Stammer et al. [2021] Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting.  Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Stammer et al. [2022] Wolfgang Stammer, Marius Memmel, Patrick Schramowski, and Kristian Kersting.  Interactive disentanglement: Learning concepts by interacting with their prototype representations.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Sundararajan et al. [2017] Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In _International Conference on Machine Learning (ICML)_ , 2017\.
  * Tamkin et al. [2023] Alex Tamkin, Mohammad Taufeeque, and Noah D. Goodman.  Codebook features: Sparse and discrete interpretability for neural networks.  _CoRR_ , abs/2310.17230, 2023.
  * Topan et al. [2021] Sever Topan, David Rolnick, and Xujie Si.  Techniques for symbol grounding with satnet.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * Träuble et al. [2023] Frederik Träuble, Anirudh Goyal, Nasim Rahaman, Michael Curtis Mozer, Kenji Kawaguchi, Yoshua Bengio, and Bernhard Schölkopf.  Discrete key-value bottleneck.  In _International Conference on Machine Learning (ICML)_ , 2023\.
  * Treisman [1999] Anne Treisman.  Solutions to the binding problem: progress through controversy and convergence.  _Neuron_ , 24(1):105–125, 1999.
  * van den Oord et al. [2017] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.  Neural discrete representation learning.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.
  * Vardakas and Likas [2023] Georgios Vardakas and Aristidis Likas.  Neural clustering based on implicit maximum likelihood.  _Neural Computing and Applications_ , 35(29):21511–21524, 2023.
  * Vedantam et al. [2021] Ramakrishna Vedantam, Arthur Szlam, Maximilian Nickel, Ari Morcos, and Brenden M. Lake.  CURI: A benchmark for productive concept learning under uncertainty.  In _International Conference on Machine Learning (ICML)_ , 2021\.
  * Wang et al. [2023] Bowen Wang, Liangzhi Li, Yuta Nakashima, and Hajime Nagahara.  Learning bottleneck concepts in image classification.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Webb et al. [2021] Taylor Whittington Webb, Ishan Sinha, and Jonathan D. Cohen.  Emergent symbols through binding in external memory.  In _International Conference on Learning Representations (ICLR)_ , 2021.
  * Wüst et al. [2024] Antonia Wüst, Wolfgang Stammer, Quentin Delfosse, Devendra Singh Dhami, and Kristian Kersting.  Pix2code: Learning to compose neural visual concepts as programs.  _CoRR_ , abs/2402.08280, 2024.
  * Yang et al. [2023] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar.  Language in a bottle: Language model guided concept bottlenecks for interpretable image classification.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Yeh et al. [2020a] Chih-Kuan Yeh, Been Kim, Sercan Ömer Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar.  On completeness-aware concept-based explanations in deep neural networks.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020a.
  * Yeh et al. [2020b] Chih-Kuan Yeh, Been Kim, Sercan Ömer Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar.  On completeness-aware concept-based explanations in deep neural networks.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020b.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018.

Supplementary Materials

In the following, we provide details on Neural Concept Binder, experimental
evaluations as well as additional evaluations.

### Impact Statement

Our work provides a new framework for unsupervised concept learning for visual
reasoning. It improves the reliability of the unsupervised concept learning by
explicitly including both inspection and revision of the concept space in the
framework. NCB thus makes an important step towards more reliable and
transparent AI, by providing an interpretable symbolic concept representation.
This representation can be utilized within reliable and proven symbolic
methods, or to improve transparency of neural modules. However, as the
concepts are learned unsupervised, one has to keep in mind that they are not
necessarily aligned with human knowledge, and might require inspections to
achieve this. As NCB features a concept revision via human feedback, it is
also necessary to consider that these revisions could have negative effects. A
user with malicious intents could modify the memory and thus make the concept
space incorrect. The fact that the learned representation of NCB is explicitly
inspectable can, however, prove to be helpful in limiting such malicious
interventions.

##  Appendix A Details on Neural Concept Binder

###  A.1 Selection Function

In the default setting, NCB selects that encoding from the retrieval corpus
with the minimal distance to infer a corresponding concept representation. We
further explore a top-$k$ approach for the selection function $s$ with $k>1$.
In this case, $s$ selects the values $v_{l}$, for the $k\in\mathbb{N}$ closest
encodings in the retrieval corpus and the resulting $c_{i}^{j}$ is obtained
via majority vote over these values. Additionally, via this selection approach
the probability of $c_{i}^{j}$ based on the occurrence distribution over the
top-$k$ values $v_{l}$ can be estimated. We provide ablations regarding this
in our evaluations in Sec. F.1.

Algorithm 1 Training NCB: Given a set of images, $X$, a block-slot encoder,
$g_{\theta}$, an unsupervised clustering model $h_{\phi}$.

1:$\hat{\theta}\leftarrow\texttt{fit}(g_{\theta},X)$ $\triangleright$
CommentStep 1: Optimize the block-slot encoder

2:$Z\leftarrow g_{\hat{\theta}}(X)$ $\triangleright$ CommentStep 2.1: Gather
block-slots from optimized $g$

3:$\bar{Z}\leftarrow\texttt{select\\_object\\_slots}($Z$)$ $\triangleright$
CommentStep 2.2: Filter out non-object slots

4:for $j\in\\{1,\cdots,N_{B}\\}$ do

5: $\hat{\phi}^{j}\leftarrow\texttt{fit}(h,\bar{Z}^{j})$ $\triangleright$
CommentStep 2.3: Obtain clustering of $\bar{Z}^{j}$

6: $R^{j}\leftarrow\texttt{distill}(\hat{\phi}^{j},\bar{Z}^{j})$
$\triangleright$ CommentStep 2.4: Extract clustering representation into
$\mathcal{R}^{j}$

###  A.2 Details on Training

The first step (_cf._ L.1 in Alg. 1) optimizes the encoder $g$ to provide
object-factorised block-slot encodings. It is optimized for unsupervised image
reconstruction based on the decoder model,
$g^{\prime}_{\theta^{\prime}}:z\rightarrow\tilde{x}\in\mathbb{R}^{D}$ (_cf._
Fig. 2) and a mean squared error loss:
$L=L_{\mathrm{MSE}}(x,g^{\prime}(g(x)))$. In practice, additional losses have
been shown to be beneficial for further improving the obtained block-slot
encodings [56, 55].

The goal of NCB’s second training step is to obtain the retrieval corpus,
$\mathcal{R}$. This procedure is based on obtaining an optimal clustering of
block encodings via an unsupervised clustering model $h$ and distilling the
resulting information from $h$ into explicit representations in the retrieval
corpus. This step is divided into several substeps (_cf._ L.2-6 in Alg. 1). It
starts with gathering a set of block-slot encodings $Z=g_{\hat{\theta}}(X)$.
As $Z$ can include slots which do not encode objects but, _e.g._ , the
background, we first select the "object-slot" encodings from $Z$. This step
results in $\bar{Z}\subseteq Z$ and consists of a heuristic selection based on
the corresponding slot attention masks (described in the following section).

For each block $j$ we next perform the following steps: (i) a clustering
model, $h_{\phi^{j}}$ (_cf._ Fig. 2), is fit to find a set of clusters within
$\bar{Z}^{j}$ thereby identifying $N_{C}\in\mathbb{N}$ meaningful clusters.
The learning of this optimal clustering is based on an unsupervised criteria,
_e.g._ , density based scores [44]. Ideally, this leads to that objects that
share similar block encodings are clustered together in the corresponding
latent block space, whereas objects that possess very different block
encodings are associated with distant clusters. This resulting clustering is
stored in $h$’s internal representation which we denote as $\phi^{j}$ (_e.g._
, the merge tree in a hierarchical clustering method [8, 9, 45]. Importantly,
$h_{\phi^{j}}$ is optimized individually for each block. (ii) In the distill
step representative block encodings of each cluster, $\texttt{enc}^{j}$, are
extracted from $h$’s internal representation, $\phi^{j}$. Hereby, every
$\texttt{enc}^{j}$ can represent either an averaged prototype or instance-
based exemplar encoding of a cluster. This is performed for every identified
cluster, $v\in\\{1,\cdots,N_{C}\\}$ and is based on $\bar{Z}^{j}$ and
$\phi^{j}$. As a result, the tuples $(\texttt{enc}^{j},v)$ are explicitly
stored in the retrieval corpus $\mathcal{R}^{j}$. The final retrieval corpus
consists of the set of individual corpora for each block,
$R=[R^{1},\cdots,R^{N_{B}}]$.

We note that in practice, it is further possible to finetune the block-slot
encoder, $g$, through supervision from the hard binder, _e.g._ , once initial
categories have been identified and can be achieved via a standard supervised
approach. Ultimately, this allows for dynamically finetuning NCB’s concept
representations.

Heuristic object-slot selection. In the following we describe the process of
identifying the slot which contains an object. This is based on heuristically
selecting slot ids based on their corresponding slot attention values.
Importantly, this approach can select object-slot ids without additional
supervision, _e.g._ , via (GT) object segmentation masks.

In principle, our object-slot selection approach finds the slots which contain
slot attention values above a predefined threshold, $\delta\in(0,1]$. However,
selecting such a threshold can be cumbersome in practice. In our evaluations
we therefore select only a single slot per image, _i.e._ , that slot which
contains the maximum slot attention value over all slots. Essentially, this
sets the maximum number of selected slots per image to $1$ and in images that
contain one objects represent no loss of object relevant information. In
preliminary evaluations we observed that the consensus between object-slot
selection based on GT object segmentation masks (matching object segmentation
masks with slot attention masks) and our maximum-based selection heuristic is
$99.45\%$ over 2000 single object images.

###  A.3 Instantiating Neural Concept Binder

We instantiate NCB’s soft binder via the SysBinder approach of Singh et al.
[56] which has been shown to provide valuable, object-factor disentangled
representations. Thus, the soft binder was trained as in the original setup
and with the published hyperparameters. Furthermore we instantiate the
clustering model, $h$, via the powerful HDBSCAN method [8, 9, 45] (based on
the popular HDBSCAN
library333https://hdbscan.readthedocs.io/en/latest/index.html). Hereby, $h$’s
internal representation, $\phi$, consists of the learned hierarchical merge
tree. In practice we found it beneficial to perform a grid search over $h$’s
hyperparameters based on the unsupervised density-based cluster validity score
[44]. The searched parameters are the minimal cluster size (the minimum number
of samples in a group for that group to be considered a cluster) and minimal
sample number (the number of samples in a neighborhood for a point to be
considered as a core point) each over the values
$[5,10,15,20,25,30,50,80,100]$. Moreover, we utilize the excess of mass
algorithm and allow for single clusters. We performed the training of the
retrieval corpus, _i.e._ , fitting $h$, on a dataset of images containing
single objects for simplifying the subsequent concept inspection mechanisms of
our evaluations. However, this can easily be extended to multiple object
images by utilising the soft binder’s slot attention masks to identify
relevant objects in an image. Finally, we instantiate the retrieval corpus as
a set of dictionaries and, unless stated otherwise, we utilise a retrieval
corpus which contains one prototype and a set of exemplar encodings per
concept. Furthermore, $s_{\mathcal{R}}$ represents the
$\operatorname*{argmin}$ selection function and we utilize the euclidean
distance as $d(\cdot,\cdot)$. It is important to note that $h$ does not make
any assumptions about the number of clusters, $N_{C}$. Thus, although $h$ fits
a clustering to best fit the block-slot encodings of a block, it can
potentially provide an overparameterized clustering, _e.g._ by representing
one underlying factor such as “gray” with several clusters. This highlights
the importance of task-alignment, _e.g._ , for symbolic downstream tasks, and
concept inspection for general concept alignment. We refer to our code for
more details444Code available here., where we will publish trained model
checkpoints and corresponding parameter logs upon acceptance.

###  A.4 Computational Resources

The resources used for training NCB were: CPU: AMD EPYC 7742 64- Core
Processor, RAM: 2064 GB, GPU: NVIDIA A100-SXM4-40GB GPU with 40 GB of RAM.
Hereby, training the SysBinder model [56] is the computational bottleneck of
NCB where we utilised two GPUs per SysBinder run. Training for 500 epochs took
$\approx$108 GPU hours. The fitting of $h$ (including the grid search over
hyperparameters) was performed on the CPU and finished within a few hours.

##  Appendix B Details on CLEVR-Sudoku

Figure 7: Examples of Sudoku CLEVR for different K values.

CLEVR-Sudoku provides Sudokus based on the datasets CLEVR and CLEVR-Easy.
Classic Sudokus have a 9x9 grid which is filled with digits from 1 to 9. In
CLEVR-Sudoku these digits are replaced by images of objects. Hereby, a digit
corresponds to a specific attribute combination, _e.g._ , "yellow" and
"sphere". Consequently, digits of the Sudoku are replaced by images of objects
with these attribute combinations. These images each contain one object. To
indicate, which attributes correspond to which digit, candidate examples of
the digits are provided. The number of these examples is a flexible parameter,
in our evaluations we used $N\in\\{1,3,5,10\\}$. Further, the number of images
provided in the Sudoku grid is flexible as well. In our main evaluations we
only considered CLEVR-Sudokus with $K=30$, meaning that 51 of the 81 Sudoku
cells are filled and 30 are left to complete. For additional investigation we
considered values for $K\in\\{10,50\\}$ as well. Examples of those Sudokus for
Sudoku CLEVR are shown in Fig. 7. The dataset has a number of 1000 samples for
Sudoku CLEVR-Easy and Sudoku CLEVR respectively for each value of $K$. Each
sample has a different puzzle and a distinct set of images, no image is used
twice for one puzzle555The code for generating the dataset is available in our
code repository, the already generated data files will be made public upon
acceptance..

##  Appendix C Datasets

CLEVR. Briefly, a CLEVR [24] image contains multiple 3D geometric objects
placed in an illuminated background scene. Hereby, the objects can possess one
of three forms, one of 8 colors, one of two sizes, one of two materials and a
random position within the scene.

CLEVR-Easy. CLEVR-Easy [56] images are similar to CLEVR images, except that in
CLEVR-Easy the size and material is fixed over all objects, _i.e._ , all
objects are large and metallic.

CLEVR-Hans3. The CLEVR-Hans3 [59] represents a classification dataset that
contains images with CLEVR objects where the image class is determined based
on the attribute combination of several objects (_e.g._ , an image belongs to
class 1 if it contains a large, gray cube and a large cylinder). Furthermore,
we utilize a confounded and non-confounded version of CLEVR-Hans3. In the
confounded case (_i.e._ , the original dataset) the train and validation set
contains spurious correlations among object attributes (_e.g._ , all large
cubes are gray in class 1) that are not present in the test set (_e.g._ ,
large cubes of class 1 take any color). In our evaluations investigating only
neural-based classification we utilize the original validation split as the
held-out test split and select a subset from the original training split as
validation set. Thus, the non-confounded version corresponds to a standard
classification setup in which the data distribution is identical over all
three data splits. Lastly we provide evaluations on a single object version of
CLEVR-Hans3 (class 1: a large, gray cube; class 2: a small metal cube; class
3: a large, blue sphere; _cf._ Tab. 3) and the original, multi-object version.

##  Appendix D Baseline Models

We note upfront, that all SysBind configurations below were trained for as
many epochs as NCB, followed by an additional finetuning for 2 epochs on the
same dataset that was used to distill NCB’s retrieval corpus.

SysBind (cont.). This denotes the original SysBinder configuration which was
trained as in [56] and provides continuous block-slot encodings. We refer to
the original work for hyperparameter details.

SysBind. This denotes a SysBinder configuration that was trained as in [56].
However, at inference time we perform discretisation via an argmin operation
over the attention values to each block’s prototype codebook.

SysBind (hard). This denotes a configuration in which the SysBinder model was
trained via a codebook attention softmax temperature of $1e-4$, resulting in a
learned discrete representation.

SysBind (step). SysBinder (step) is trained by step-wise decreasing this
temperature his denotes a configuration in which the SysBinder model was
trained via a step-wise decreasing codebook attention softmax temperature
(with a decrease by a factor of $0.5$ every $50$ epochs, starting from $1.$).

Supervised Concept Learner. This corresponds to a slot attention encoder [36]
that was trained for set prediction (_i.e._ , in a supervised fashion) to
predict the object-properties for every object in a CLEVR image. We refer to
Locatello et al. [36] and Stammer et al. [59] for details.

##  Appendix E Details on Experimental Setup

###  E.1 Classifying object-properties from concept encodings

For our evaluations in the context of (Q1) we utilise a decision tree as
classification model that is trained on a set of concept encodings to predict
corresponding object properties, _e.g._ , sphere, cube or cylinder.
Importantly, we train a separate classifier for each property category, _e.g._
, the categories shape, color, material and size in the case of CLEVR, and
average accuracies over these. The classifiers parameters correspond to the
default parameters of the sklearn library666https://scikit-
learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html.

###  E.2 CLEVR-Sudoku evaluations

For our CLEVR-Sudoku evaluations we use a solver that combines a symbolic
classifier with a constraint propagation based algorithm. To solve CLEVR-
Sudokus, it is at first required to detect the underlying mapping from the
object attribute combinations to the digits via the provided candidate
examples. For this, we require a symbolic classifier to learn this mapping,
which in the case of our evaluations is achieved via a decision tree
classifier. For each evaluated model the concept encodings of the candidate
example images of a CLEVR-Sudoku are retrieved and provided as input to the
classifier. Hereby, the corresponding digits are the labels to be predicted.
With the predictions of the trained classifier the concept encodings of the
images in the Sudoku grid are classified to get a symbolic representation of
the Sudoku, _i.e._ , map the images in the cells to their corresponding
digits. Based on this numerical representation of the puzzle, we use an
algorithm from [46] that uses a combination of constraint propagation [6] and
search. The algorithm keeps track of all possible values for each cell. Within
each step, the Sudoku constraints are used to eliminate all invalid digits
from the possibilities. Then the search of the algorithm select a digit for a
non-filled cell. Based on this digit, the possibilities are updated for all
other cells. When there is a constraint violation, the search-tree is
traversed backwards and other possible digits for non-filled cells are
explored. This process is repeated until the Sudoku is solved (in case the
initial state inferred from the objects was correct) or until there is no
possible solution left (meaning that the initial state was incorrectly
inferred from the objects). The implementation of the algorithm is based on
the code from777https://github.com/ScriptRaccoon/sudoku-solver-
python/tree/main. Finally, to avoid errors due to random seeding of the
classifier, for each puzzle we fit 10 independent classifiers (each with
different seeds) to predict the corresponding mapping. For the results in our
evaluations we average the performance over these 10 classifier seeds.

Lastly, the evaluations in the context of (Q2) are based on the trained (NCB)
models of (Q1).

###  E.3 Obtaining Revisory Feedback

We note that the evaluations in the context of (Q3) are based on the trained
NCB models of (Q1).

Revisory feedback for downstream Sudoku task.

To revise its discrete concepts, NCB offers the possibility to delete or merge
clusters in the blocks. In the case of merging, the prototypes and exemplars
of the clusters to be merged get aggregated so that they all map to the same
concept symbol. For deletion there are several processing cases, depending on
how many categories are in the block and how many are supposed to be deleted:

  * •

Case 1: if all clusters from a block should be deleted (or if there is only
one concept in the block, which should be deleted), we map all samples to the
same concept. This results in the block containing no information (we keep the
block to avoid issues with the dimensions of the concept representation).

  * •

Case 2: all clusters but one are to be deleted. In this case we still want to
distinguish between the presumably "informative" cluster and the uninformative
other clusters. Therefore we map all the blocks to be deleted to one cluster
id instead of deleting them completely.

  * •

Case 3: at least two clusters should not be deleted. In this case, we remove
the encodings of these clusters completely. The cluster id for these clusters
then no longer exist in the retrieval corpus.

Feedback via GPT-4. We systematically prompt GPT-4 [47] for receiving revisory
feedback. We provide example prompts in 1. First, we ask GPT-4 to name
relevant object properties for a set of example images, _e.g._ , "shapes:
[cube, cylinder], color: [red, blue]". Based on these provided property lists
we ask GPT-4 to provide a descriptive list of each exemplar object’s image for
each concept of each block, _e.g._ , "{Exemplar1: [cube, red], Exemplar2:
[cube, blue], … }". Based on these descriptions we identify whether all
exemplar objects of one concept share a common subproperty, _e.g._ , "cube".
If there is no common subproperty, the concept should be removed from the
retrieval corpus. In a second step we evaluate whether all exemplar objects
from two separate concepts share a common subproperty. In this case we decide
to merge the concepts based on GPT-4’s analysis. We finally integrate GPT-4’s
feedback into NCB’s retrieval corpus via the procedures described above.

Feedback via simulated humans. To simulate feedback by a human user, we
utilise a decision tree (DT) classifier to classify attributes of objects
based on NCB’s discrete concepts (similar to Q1). For this, we transform the
concept-slot encodings into multi-hot encodings. We then extract the
importance of the concepts from the trained DT classifier. Based on this we
select "unimportant" concepts to be deleted based on the procedures describe
above. Note that in this setting we do not query for feedback considering the
merging of concepts.

###  E.4 Neural Classification

We note that the evaluations in the context of (Q4) are based on the trained
NCB models of (Q1).

Neural classifier. In the context of the classification evaluations (Q4) we
utilize the setup of Stammer et al. [59]. Specifically, a set transformer [31]
is trained to classify images from the CLEVR-Hans3 dataset given encodings
that are, in turn, obtained from either NCB or a supervised trained slot
attention encoder [36] (SA). In the case of utilizing NCB’s encodings we
transform the concept-slot encodings into multi-hot encodings to match those
of the SA-based setup. We refer to Stammer et al. [59] and our code for
additional details concerning this setup.

Obtaining explanations from the neural classifier. We provide the explanations
in Tab. 3 for the single object version of Fig. 14. To obtain these
explanations for the neural classifier we utilize the approach of Stammer et
al. [59] which is based on the integrated gradients explanation method [61].
This estimates the importance value of each input element (in this case input
concept encodings) for a classifiers final decision. We remove negative
importance values and normalise the importance values as in [59]. We then sum
over the importance values corresponding to images of a class, normalise the
values per block and binarize these aggregated and normalised importance
values via the threshold of $0.25$ (_i.e._ , importance values above $0.25$
are set to $1$, otherwise $0$). This provides us with a binary vector
indicating which concepts are considered important per block. We illustrate
these investigations via explanations from one model.

Explanatory interactive learning (XIL). Explanatory interactive learning (\+
XIL on NN) is used to mitigate the confounder in the CLEVR-Hans dataset.
Hereby, (simulated) human feedback on the explanation of the neural classifier
is used to retrain the classifier via the loss based approach of Stammer et
al. [59]. The feedback annotations mark which of NCB’s concepts should not be
used for the NN’s classification decision. This is integrated into the NN by
training the model to provide (integrated gradients-based) explanations that
do not focus on these concepts. We refer to Stammer et al. [59] for details.
The second form of interactive learning (+XIL on NCB concepts) is directly
applied on the NCB’s concept representation. Specifically, concepts from NCB
that encode information concerning the irrelevant, confounding factors are
simply set to zero, corresponding to not being inferred for the object in the
image. _E.g._ , if the NCB infers concepts concerning the color “gray” to be
present in an object and the underlying confounder is the color “gray” the
corresponding concept activations of the NCB’s prediction are set to zero,
_i.e._ , no gray. Then the neural classifier is retrained on the new concept
representations. Next to a better performance, the advantage of this approach
is that it does not require the more costly loss-based XIL training loop. We
illustrate these investigations via interactions on one model.

Table 4: Classifying attributes from concept representations. Best results are
in bold.

N Train |  SysBinder $[$cont.$]$ | SysBinder | SysBinder (hard) | SysBinder (step) | NCB (P) | NCB (P+E) | NCB (P+E, topk)
---|---|---|---|---|---|---|---
— CLEVR-Easy —
N=2000 | $99.83\mbox{\scriptsize$\pm 0.24$}$ | $92.49\mbox{\scriptsize$\pm 5.45$}$ | $22.92\mbox{\scriptsize$\pm 0.00$}$ | $95.76\mbox{\scriptsize$\pm 4.92$}$ | $98.76\mbox{\scriptsize$\pm 1.05$}$ | $\mathbf{99.02}\mbox{\scriptsize$\pm 1.00$}$ | $98.93\mbox{\scriptsize$\pm 1.10$}$
N=200 | $99.20\mbox{\scriptsize$\pm 0.41$}$ | $87.90\mbox{\scriptsize$\pm 8.05$}$ | $22.92\mbox{\scriptsize$\pm 0.00$}$ | $92.42\mbox{\scriptsize$\pm 7.32$}$ | $97.11\mbox{\scriptsize$\pm 2.16$}$ | $\mathbf{98.50}\mbox{\scriptsize$\pm 1.80$}$ | $98.42\mbox{\scriptsize$\pm 1.91$}$
N=50 | $91.13\mbox{\scriptsize$\pm 4.21$}$ | $78.41\mbox{\scriptsize$\pm 8.69$}$ | $22.92\mbox{\scriptsize$\pm 0.00$}$ | $70.64\mbox{\scriptsize$\pm 11.89$}$ | $94.31\mbox{\scriptsize$\pm 4.47$}$ | $\mathbf{95.87}\mbox{\scriptsize$\pm 2.93$}$ | $95.72\mbox{\scriptsize$\pm 3.04$}$
N=20 | $64.88\mbox{\scriptsize$\pm 10.89$}$ | $62.61\mbox{\scriptsize$\pm 7.18$}$ | $22.92\mbox{\scriptsize$\pm 0.00$}$ | $54.61\mbox{\scriptsize$\pm 9.57$}$ | $90.50\mbox{\scriptsize$\pm 7.09$}$ | $\mathbf{94.22}\mbox{\scriptsize$\pm 4.11$}$ | $94.15\mbox{\scriptsize$\pm 4.14$}$
— CLEVR —
N=2000 | $98.86\mbox{\scriptsize$\pm 1.15$}$ | $86.22\mbox{\scriptsize$\pm 10.40$}$ | $36.46\mbox{\scriptsize$\pm 0.00$}$ | $88.90\mbox{\scriptsize$\pm 14.81$}$ | $96.77\mbox{\scriptsize$\pm 2.63$}$ | $\mathbf{97.26}\mbox{\scriptsize$\pm 2.67$}$ | $97.17\mbox{\scriptsize$\pm 2.68$}$
N=200 | $97.61\mbox{\scriptsize$\pm 2.58$}$ | $81.13\mbox{\scriptsize$\pm 12.39$}$ | $36.46\mbox{\scriptsize$\pm 0.00$}$ | $83.17\mbox{\scriptsize$\pm 17.05$}$ | $96.41\mbox{\scriptsize$\pm 2.64$}$ | $\mathbf{96.80}\mbox{\scriptsize$\pm 3.01$}$ | $96.80\mbox{\scriptsize$\pm 3.04$}$
N=50 | $93.25\mbox{\scriptsize$\pm 4.62$}$ | $61.67\mbox{\scriptsize$\pm 8.51$}$ | $36.46\mbox{\scriptsize$\pm 0.00$}$ | $68.81\mbox{\scriptsize$\pm 17.74$}$ | $94.29\mbox{\scriptsize$\pm 4.78$}$ | $\mathbf{94.67}\mbox{\scriptsize$\pm 4.65$}$ | $94.10\mbox{\scriptsize$\pm 5.25$}$
N=20 | $79.11\mbox{\scriptsize$\pm 8.75$}$ | $49.79\mbox{\scriptsize$\pm 6.73$}$ | $36.46\mbox{\scriptsize$\pm 0.00$}$ | $58.58\mbox{\scriptsize$\pm 16.09$}$ | $87.55\mbox{\scriptsize$\pm 5.35$}$ | $\mathbf{88.57}\mbox{\scriptsize$\pm 4.68$}$ | $88.42\mbox{\scriptsize$\pm 4.63$}$

Table 5: Ablation: Classifying attributes from concept representations with
sub-optimal NCB components. The left column serves as a reference and
represents the configurations used in the main evaluations, _i.e._ , where the
soft binder was trained for 600 epochs and the clustering model represented
the HDBSCAN approach that was optimized via a grid-search over its
corresponding hyperparameters.

N Train |  NCB |  NCB (50 epochs) |  NCB (100 epochs) |  NCB (w/o grid search) |  NCB (kmeans)
---|---|---|---|---|---
— CLEVR —
N=2000 | $97.26\mbox{\scriptsize$\pm 2.67$}$ | $95.19\mbox{\scriptsize$\pm 1.2$}$ | $94.91\mbox{\scriptsize$\pm 3.45$}$ | $97.69\mbox{\scriptsize$\pm 2.95$}$ | $97.26\mbox{\scriptsize$\pm 2.80$}$
N=200 | $96.80\mbox{\scriptsize$\pm 3.01$}$ | $93.69\mbox{\scriptsize$\pm 1.08$}$ | $93.83\mbox{\scriptsize$\pm 2.90$}$ | $96.80\mbox{\scriptsize$\pm 3.09$}$ | $96.01\mbox{\scriptsize$\pm 3.51$}$
N=50 | $94.67\mbox{\scriptsize$\pm 4.65$}$ | $89.10\mbox{\scriptsize$\pm 4.29$}$ | $89.67\mbox{\scriptsize$\pm 6.95$}$ | $94.46\mbox{\scriptsize$\pm 5.65$}$ | $87.65\mbox{\scriptsize$\pm 8.78$}$
N=20 | $88.57\mbox{\scriptsize$\pm 4.68$}$ | $83.46\mbox{\scriptsize$\pm 6.08$}$ | $88.48\mbox{\scriptsize$\pm 2.36$}$ | $90.51\mbox{\scriptsize$\pm 4.40$}$ | $73.52\mbox{\scriptsize$\pm 10.92$}$

##  Appendix F Additional Quantitative Results

###  F.1 Encoding Expressivity

In our evaluations in Tab. 2 it appears that training for discrete encodings
via SysBinder (hard) leads to no learning effect of the model altogether. In
contrast training step-wise via SysBinder (step) provides better results, even
slightly above the encodings of SysBinder (_i.e._ , training for continuous
representations and then discretising via $\operatorname*{argmin}$). Lastly,
we observe that NCB’s encodings lead to much lower performance variance
compared to all baselines. Particularly SysBinder (step)’s high variances,
hint towards issues with local optima.

We further provide ablations in the context of (Q1) on different model choices
of NCB in Tab. 4. Specifically, we investigate the effect of a top-$k$
selection function as well as the influence of using only prototype encodings
in the retrieval corpus (NCB (P)) versus using prototype and exemplar
encodings (NCB (P+E)). Unless noted otherwise, the NCB configurations in Tab.
4 utilize the $\operatorname*{argmin}$ selection function. We note that when
using prototypes, the average encoding of all elements in a cluster is formed,
resulting in one prototype encoding per cluster in $R^{j}$. In the second
variant, we extend the prototypes with exemplars for each cluster. Exemplars
are representative encodings for this cluster added to the corpus, resulting
in a larger corpus, which potentially provides an improved structure of the
encoding space. Indeed, we observe that NCB provides best performances via the
$\operatorname*{argmin}$ selection function and utilising both prototype and
exemplar encodings. This was the setting used in all evaluations of the main
paper.

###  F.2 Ablation Analysis of Suboptimal NCB Components

Lastly, in the context of (Q1) we further refer to ablations in Tab. 5 on the
specific implementation choices of the NCB instantiation of our evaluations.
We hereby investigate the effect of sub-optimal soft and hard binder
components on a classifier’s ability to identify object attributes from NCB’s
concept encodings. Specifically, we investigate (i) the effect when the soft
binder, _i.e._ , SysBinder encoder, was trained for fewer epochs, resulting in
less disentangled continuous representations, and (ii) when the HDBSCAN model
of the hard binder was not optimized via a parameter grid search or replaced
with a more rudimentary clustering model, _i.e._ , a k-means clustering
approach [38].

In the leftmost column of Tab. 5, we provide the performances of the NCB
configuration of our main evaluations as a reference. As a reminder, hereby,
NCB’s soft binder was finetuned for 600 epochs, and its hard binder component
contains a clustering model based on the HDBSCAN approach that was furthermore
optimized via a grid search over its corresponding hyperparameters. Focusing
on the next two columns right of the baseline, we observe that when the soft
binder component is trained for fewer epochs than the baseline NCB we indeed
observe a decrease in classification performance. Notably, however, we still
observe higher performances in comparison to the discrete SysBinder
configurations (_cf._ Sec. F.1), but also when compared to SysBinder’s
continuous configuration (for $N=20$). Focusing next on the rightmost column
of Tab. 5 where NCB’s clustering model was replaced with the more rudimentary
k-means clustering approach, we observe a strong decrease in classifier
performance. This is particularly true in the small data regime ($N=50$ and
$N=20$). Surprisingly, focusing on the second to the rightmost column, we
observe that when we select the default hyperparameter values of the HDBSCAN
package (rather than performing a grid-search over these), the classifier
reaches slightly improved performances than via the baseline NCB configuration
(particularly for $N=20$). Thus, in this particular case, the default values
seem practical. However, this cannot be guaranteed in all future cases, and we
still recommend performing a form of grid search if no prior knowledge can be
provided upfront on an optimal parameter set. We postulate that the specific
density-based cluster validity score used for selecting the optimal cluster
parameters has been sub-optimal and leave investigating other, more optimal
selection criteria for future work.

Overall, our ablation investigations indeed indicate that we obtain less
expressive concept encodings via NCB with less powerful sub-components.
However, we also observe a certain amount of robustness of our NCB
instantiation towards sub-optimal components.

###  F.3 Analysis of Learned Concept Space

We here provide a brief analysis of NCB’s learned concept space. These
evaluations were performed on the models that were trained in the context of
(Q1). Specifically, in Fig. 8, we provide the number of obtained concepts over
all blocks (averaged over the 3 initialization seeds) both for CLEVR-Easy and
CLEVR. We observe a much larger number of concepts overall for the CLEVR
dataset but also a much larger variance in the number of concepts. This is
largely due to that in CLEVR-Easy $N_{B}=8$ whereas in CLEVR $N_{B}=16$. Thus,
the models are able to learn a more overparameterized concept space in the
case of CLEVR. Further, in Fig. 9, we present the distribution of the number
of concepts per block over all 3 NCB runs, both for CLEVR-Easy and for CLEVR.
We observe that while most blocks contain maximally 20 concepts for CLEVR-Easy
and 50 for CLEVR, there are several block outliers which contain a much
greater set of concepts.

Figure 8: Average number of concepts in NCB’s retreival corpus.

These represent cases in which the initial block-slot encoding space was
uninformative to begin with and, therefore, difficult to find some form of
useful clustering via $h$. Where some of these blocks only contained
irrelevant information in general, some blocks encoded positional information,
which represents a continuous variable to begin with and is thus unlikely to
be well represented via a clustering.

Figure 9: The distribution of number of obtained concepts per block both for
CLEVR-Easy and CLEVR. These values are computed over all seeds.

###  F.4 CLEVR-Sudoku Evaluations

Figure 10: Error ratios (%) of the digit classification in CLEVR-Sudoku based
on different symbolic concept encodings.

In our evaluations on (Q2) we observe that, interestingly, for Sudoku CLEVR
the supervised object classifier shows better results than for CLEVR-Easy.
This seems counter-intuitive, however, in CLEVR-Easy-Sudoku digit labels are
mapped to combinations of attributes that only stem from two categories, shape
and color (in contrast to four categories in CLEVR-Sudoku) thus making it more
likely to obtain recurring attributes over several digits (_e.g._ , digits 3,
4 and 5 of Fig. 4 all depict green objects). Thus, if an error occurs in the
digit classification due to errors concerning one attribute the effect of this
error will have a larger effect. Moreover in the case of CLEVR-Easy, we
observe that in comparison to the supervised model, whose property
misprediction errors can lead to large issues in the downstream module, NCB’s
unsupervised and somewhat overparameterised concept space appears to dampen
this issue, thus leading to a higher number of solved puzzles, _e.g._ , for 3,
5 or 10 examples.

In Fig. 10 we report the errors in predicting the underlying digits of the
CLEVR-Sudokus. We observe that the errors of SysBinder (unsupervised) are
drastically higher than the errors of the other methods. These high
classification errors further explain this method’s low performances, _i.e._ ,
did not allow to solve any Sudoku. It can further be seen that for one example
per digit the digit classification errors are much higher. This is reasonable
as hereby the difficulty for the classifier is also higher. However, with an
increasing number of examples the classifier’s errors decrease. The relations
between the errors in the digit prediction and the overall performance in
CLEVR-Sudoku are similar which is sensible since the error is decisive for the
number of solved puzzles.

We further evaluate the influence of the number of missing images per Sudoku.
For this we consider Sudokus with $K\in\\{10,30,50\\}$. The results on these
variations with 5 candidate example images are reported in Fig. 11. We see
that the more empty cells there are in a Sudoku’s initial state (higher $K$),
the more Sudokus are solved. This is due to the lower probability of
misclassifying an image inside the Sudoku cells, as there are less images to
classify. This pattern is observable for all of the different concept
encodings we compared.

Figure 11: Solved Sudokus (%) of Sudoku CLEVR-Easy and Sudoku CLEVR with
different values for K (empty cells).

###  F.5 Revision Statistics

Figure 12: Average number of cluster deletions over all blocks via GPT-4 and
simulated human user revision.

We provide statistics of the number of resulting removal requests per agent in
Fig. 12. For the revision of CLEVR-Easy concepts we can see that GPT-4 detects
only a few concepts to delete while via simulated human revision more concepts
get deleted. In our initial evaluations (_cf._ Fig. 4) we had observed that
human revision leads to substantial improvements while GPT-4’s revision even
reduces performances slightly. For CLEVR-Sudoku in Fig. 12, we specifically
observe that the overall number of deletions via GPT-4 is significantly
higher. Interestingly, GPT-4 detects on average more blocks to delete here but
also has a higher variance over the 3 different NCB runs. We hypothesize that
this very “conservative” revision leads to the removal of concepts that
actually contain valuable concept information, thus leading to less expressive
concept encodings overall. Ultimately, this is due to mistakes in GPT-4’s
analysis of provided images (_cf._ Sec. E.3).

###  F.6 Dynamically Discretising Continuous Factors via Symbolic Revision

In our second set of evaluations in the context of (Q3) we investigate the
third form of symbolic revision as introduced in Sec. 3.3: adding concept
information to the hard binder’s retrieval corpus. Hereby, we focus on the
task of learning a novel concept that had only been stored implicitly in the
soft binder’s representations, but not explicitly in the hard binder’s
representations. Specifically, we focus on positional concepts of CLEVR
objects where the underlying GT position is represented via continuous values.
Overall, it is debatable whether one, in principle, should or even can
represent such a continuous underlying feature via a discrete concept
representation.

Figure 13: Test accuracy ($\%$) for classifying objects as placed left or
right in a scene.

In this set of evaluations we investigate a setting in which it is necessary
to identify coarse categorisations of an object’s position, _e.g._ , whether
the object is placed in the left or right half of an image. We hereby simulate
a human stakeholder that, having identified the block $j$ that generally
encodes position information, revises the corresponding concept encodings.
This revision is performed in two ways: (i) by iterating over all of the
block’s concepts and merging concepts into left and right concepts or (ii) by
replacing all information in $\mathcal{R}$ with encodings from a selected set
of positive example images for the two relevant positions. Fig. 13 presents
the results of training a classifier to predict the attributes “left” and
“right” from NCB’s encodings (we here focus only on one seeded run for
illustrations) with different types of revision. We observe that both allow to
easily retrieve relevant information from NCB’s newly revised concept space.
These results illustrate the important ability to easily adapt the hard
binder’s concept representations by dynamically re-reading out the information
of the soft binder’s representations in a use-case based manner. The results
further illustrate the effect of adding prior knowledge to NCB’s concept
representations, thereby potentially reducing the amount of inspection effort
required on the stakeholder’s side, _e.g._ , in comparison to the merge
revision.

###  F.7 Classifying CLEVR-Hans3

In our final evaluations (Q1) we highlight the advantage of NCB’s concept
encodings when combined with subsymbolic (_i.e._ , neural) modules for making
their decisions transparent.

Figure 14: Test accuracy ($\%$) for classifying CLEVR-Hans3 images with a
neural classifier that is provided concept representations of NCB and of a
supervised trained slot attention encoder. We differentiate here between class
rules based on one object and multiple objects.

Specifically, while a discrete concept representation is technically not
required for neural modules, it has a key advantage: a discrete and
inspectable representation allows for transparent downstream computations. We
highlight this property in the context of image classification on variations
of the benchmark CLEVR-Hans3 dataset [59]. For these evaluations we revert to
training a set transformer [31] (denoted as NN in the following) for
classifying images when provided the unsupervised concept encodings of NCB as
image representations. We denote this configuration as NCB + NN and compare it
to a configuration in which the set transformer is provided concept encodings
from a supervised slot attention encoder, denoted as SA + NN. In Fig. 14 we
obseve that NCB’s concepts perform on par with those learned supervisedly,
each reaching held-out test accuracies higher than $95\%$.

###  F.8 Confounding Evaluations

For the confounding mitigation evaluations in the context of (Q4) we train the
NCB + NN configuration on the confounded version of CLEVR-Hans3, where we
hereby focus on the single object class rules similar to those in Fig. 14. In
this case all large cubes of class one images posses the color gray at
training time, but arbitrary colors at test time. We observed accuracies of
NCB + NN on the confounded validation set of $99.22\%$ against the non-
confounded test set $79.29\%$. This very high validation accuracy versus a
significantly reduced test accuracy indicates that the classifier is strongly
influenced by the datasets underlying confounding factor.

##  Appendix G Qualitative Results

Fig. 15 and Fig. 16 represent qualitative inspection results of NCB’s learned
concepts. We specifically present implicit inspection via exemplars of
concepts from two blocks from NCB when trained on CLEVR-Easy. One can observe
that block 2 (Fig. 15) appears to encode shape concepts, however contains one
ambiguous concept. We further observe that Fig. 16 appears to encode color
concepts, whereby it contains one ambiguous concept (concept 8) and two
concepts that appear to both encode the color purple (concept 9 and 10) which
could potentially be merged.

Figure 15: Concepts of Block 2 for NCB with CLEVR-Easy. We here provide
implicit inspection examples (_i.e._ , via exemplars of each concept). We
observe that block 2 appears to encode shape information (concept 1-3) and
contains one ambiguous concept (concept 4). Figure 16: Concepts of Block 8 for
NCB with CLEVR-Easy. We here provide implicit inspection examples (_i.e._ ,
via exemplars of each concept). We observe that block 8 appears to be encoding
color information, contains one ambiguous concept (concept 8) and two concepts
that appear to both encode the color purple (concept 9 and 10).

##  Appendix H Numerical Results

In our evaluations we presented the results on CLEVR-Sudoku in the form of bar
plots. We refer to Tab. 6, Tab. 7 and Tab. 8 for the numerical values for the
different variations of the dataset.

Table 6: Percentage of solved CLEVR-Sudokus for different number of example images. Sudoku CLEVR-Easy | 1 Example | 3 Examples | 5 Examples | 10 Examples
---|---|---|---|---
GT Concepts | $100.0\pm 0.00$ | $100.00\pm 0.00$ | $100.00\pm 0.0$ | $100.00\pm 0.00$
SA (supervised) | $35.22\pm 5.63$ | $40.07\pm 3.76$ | $42.07\pm 3.14$ | $44.36\pm 2.54$
SysBinder (unsupervised) | $0.00\pm 0.00$ | $0.00\pm 0.00$ | $0.00\pm 0.00$ | $0.00\pm 0.00$
NCB (unsupervised) | $6.13\pm 4.42$ | $47.30\pm 33.06$ | $54.95\pm 31.86$ | $63.21\pm 27.45$
NCB revised (GPT-4) | $34.61\pm 43.46$ | $48.25\pm 34.80$ | $54.31\pm 30.85$ | $61.83\pm 26.10$
NCB revised (human) | $54.41\pm 37.40$ | $64.00\pm 28.53$ | $67.07\pm 24.83$ | $70.10\pm 21.52$
NCB revised (human)* | $50.81\pm 45.38$ | $62.00\pm 34.78$ | $66.40\pm 30.38$ | $70.43\pm 26.35$
Sudoku CLEVR |  |  |  |
GT Concepts | $100.0\pm 0.00$ | $100.0\pm 0.0$ | $100.0\pm 0.00$ | $100.0\pm 0.00$
SA (supervised) | $54.9\pm 5.99$ | $69.99\pm 2.76$ | $73.92\pm 1.69$ | $77.71\pm 0.38$
SysBinder (unsupervised) | $0.00\pm 0.00$ | $0.00\pm 0.00$ | $0.00\pm 0.00$ | $0.00\pm 0.00$
NCB (unsupervised) | $0.01\pm 0.00$ | $12.36\pm 8.46$ | $26.62\pm 18.47$ | $38.24\pm 26.97$
NCB revised (GPT-4) | $1.11\pm 1.19$ | $16.18\pm 11.93$ | $21.76\pm 15.84$ | $27.75\pm 20.14$
NCB revised (human) | $3.23\pm 4.55$ | $36.19\pm 32.64$ | $44.8\pm 35.58$ | $48.97\pm 37.13$
NCB revised (human)* | $4.84\pm 4.82$ | $54.0\pm 25.43$ | $66.69\pm 21.46$ | $72.68\pm 19.53$
Table 7: Error ratios on digit classification of CLEVR-Sudokus for different number of example images. Sudoku CLEVR-Easy | 1 Example | 3 Examples | 5 Examples | 10 Examples
---|---|---|---|---
GT Concepts | $0.00\pm 0.00$ | $0.00\pm 0.00$ | $0.0\pm 0.00$ | $0.0\pm 0.00$
SA (supervised) | $7.23\pm 0.89$ | $5.25\pm 0.16$ | $4.89\pm 0.09$ | $4.55\pm 0.09$
SysBinder (unsupervised) | $88.69\pm 0.05$ | $88.12\pm 0.18$ | $87.66\pm 0.31$ | $87.30\pm 0.40$
NCB (unsupervised) | $23.57\pm 1.02$ | $3.55\pm 2.47$ | $2.16\pm 1.74$ | $1.35\pm 1.13$
NCB revised (GPT-4) | $13.21\pm 9.89$ | $3.21\pm 2.56$ | $2.1\pm 1.64$ | $1.36\pm 1.06$
NCB revised (human) | $4.94\pm 5.68$ | $1.45\pm 1.49$ | $1.12\pm 1.05$ | $0.95\pm 0.83$
NCB revised (human)* | $6.55\pm 6.37$ | $1.79\pm 1.73$ | $1.31\pm 1.24$ | $1.06\pm 1.00$
Sudoku CLEVR |  |  |  |
GT Concepts | $0.00\pm 0.00$ | $0.00\pm 0.00$ | $0.00\pm 0.00$ | $0.00\pm 0.00$
SA (supervised) | $3.78\pm 0.79$ | $1.85\pm 0.18$ | $1.52\pm 0.09$ | $1.30\pm 0.08$
SysBinder (unsupervised) | $88.81\pm 0.03$ | $88.67\pm 0.03$ | $88.68\pm 0.1$ | $88.71\pm 0.11$
NCB (unsupervised) | $48.54\pm 5.22$ | $11.61\pm 3.79$ | $6.30\pm 4.52$ | $4.49\pm 4.37$
NCB revised (GPT-4) | $34.10\pm 6.51$ | $9.85\pm 4.93$ | $8.11\pm 4.85$ | $7.16\pm 4.79$
NCB revised (human) | $42.32\pm 14.56$ | $7.55\pm 5.84$ | $4.80\pm 4.94$ | $3.91\pm 4.38$
NCB revised (human)* | $39.85\pm 17.31$ | $3.69\pm 2.57$ | $1.34\pm 0.97$ | $0.84\pm 0.66$
Table 8: Percentage of solved CLEVR-Sudokus for different values of K with 5 example images. Sudoku CLEVR-Easy | K=10 | K=30 | K=50
---|---|---|---
GT Concepts | $100.0\pm 0.00$ | $100.0\pm 0.00$ | $100.0\pm 0.00$
SA (supervised) | $39.02\pm 3.25$ | $42.07\pm 3.14$ | $47.89\pm 3.37$
SysBinder (unsupervised) | $0.00\pm 0.00$ | $0.00\pm 0.00$ | $0.00\pm 0.00$
NCB (unsupervised) | $49.64\pm 35.07$ | $54.95\pm 31.86$ | $64.91\pm 25.50$
NCB revised (GPT-4) | $48.62\pm 33.92$ | $54.31\pm 30.85$ | $63.62\pm 25.52$
NCB revised (human) | $60.23\pm 28.77$ | $67.07\pm 24.83$ | $75.05\pm 20.51$
NCB revised (human)* | $60.50\pm 35.24$ | $66.40\pm 30.38$ | $73.39\pm 24.96$
Sudoku CLEVR |  |  |
GT Concepts | $100.00\pm 0.00$ | $100.00\pm 0.00$ | $100.00\pm 0.00$
SA (supervised) | $68.96\pm 0.65$ | $73.92\pm 1.69$ | $78.46\pm 1.72$
SysBinder (unsupervised) | $0.00\pm 0.00$ | $0.00\pm 0.00$ | $0.00\pm 0.00$
NCB (unsupervised) | $21.18\pm 14.85$ | $26.62\pm 18.47$ | $34.79\pm 21.84$
NCB revised (GPT-4) | $17.76\pm 12.86$ | $21.76\pm 15.84$ | $29.12\pm 20.49$
NCB revised (human) | $40.30\pm 34.34$ | $44.80\pm 35.58$ | $51.55\pm 35.05$
NCB revised (human)* | $60.10\pm 24.36$ | $66.69\pm 21.46$ | $74.54\pm 16.02$

Listing 1: Prompts for GPT-4.

⬇

—————————-

Property List Prompt:

You are provided six images. An image contains subimages.

Each subimage depicts one object. Each object represents

a reflective geometric solid that is placed in a neutral

gray background scene with a light source. Furthermore,

each object has multiple properties,

e.g., color, shape, size, material.

Each property can be subdivided into several sub-properties,

e.g., brown is a sub-property of the property color.

Please provide a list of obect properties and

subproperties that are depicted in all images. Ignore

the background and the object’s luminance and

reflectivity. Use the following answer template:

{

property: [sub-property, sub-property, …]

property: [sub-property, sub-property, …]

…

}

—————————-

Description Prompt:

You are provided an image. The image contains at most 25 subimages.

Each subimage depicts one object. Each object represents a reflective

geometric solid that is placed in a neutral gray background scene with

a light source. Furthermore, each object has multiple properties,

e.g., color. Each property can be subdivided into several sub-properties,

e.g., green is a sub-property of the property color. The possible

properties and sub-properties are the following:

INSERT_PREVIOUSLY_OBTAINED_PROPERTY_LIST

Focusing only on these properties, please perform the following tasks.

First, for every object in the image please list the sub-properties

from the given lists that the object depicts. Only name the sub-properties

that are given. Please use the following format:

{

Object1: [sub-property, …],

Object2: [sub-property, …],

…

}

—————————-