# CoCoG: Controllable Visual Stimuli Generation based on Human Concept
Representations

Chen Wei†,1,2 Jiachen Zou†,1 Dietmar Heinke2&Quanying Liu∗,1
1Southern University of Science and Technology, Shenzhen, China
2University of Birmingham, Birmingham, United Kingdom
{weic3<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

A central question for cognitive science is to understand how humans process
visual objects, i.e, to uncover human low-dimensional concept representation
space from high-dimensional visual stimuli. Generating visual stimuli with
controlling concepts is the key. However, there are currently no generative
models in AI to solve this problem. Here, we present the Concept based
Controllable Generation (CoCoG) framework. CoCoG consists of two components, a
simple yet efficient AI agent for extracting interpretable concept and
predicting human decision-making in visual similarity judgment tasks, and a
conditional generation model for generating visual stimuli given the concepts.
We quantify the performance of CoCoG from two aspects, the human behavior
prediction accuracy and the controllable generation ability. The experiments
with CoCoG indicate that 1) the reliable concept embeddings in CoCoG allows to
predict human behavior with 64.07% accuracy in the THINGS-similarity dataset;
2) CoCoG can generate diverse objects through the control of concepts; 3)
CoCoG can manipulate human similarity judgment behavior by intervening key
concepts. CoCoG offers visual objects with controlling concepts to advance our
understanding of causality in human cognition. The code of CoCoG is available
at https://github.com/ncclab-sustech/CoCoG.

††footnotetext: $\dagger$ Equal contribution.††footnotetext: ∗ Corresponding
author.††footnotetext: This paper has been accepted by IJCAI2024.

##  1 Introduction

Humans receive abundant visual stimulation from the natural world. Unlike
computer vision models that aim at object recognition tasks, humans aim to
survive in complex nature, which requires understanding abstract concepts of
these visual objects, such as functionality, toxicity, and danger. To explore
such concept representation in humans, scientists have proposed a series of
visual stimuli-based decision-making tasks Murphy and Medin (1985); Medin et
al. (1993); Hebart et al. (2020); Roads and Love (2023), such as the
similarity judgment task, in which visual stimuli of specific concepts are
presented to the participants and then their decision-making behaviors are
recorded. However, understanding human concept representation through these
cognitive tasks has two main difficulties. On the one hand, training an AI
agent to uncover interpretable concept representations by predicting human
decision-making behavior requires substantial human decision-making data under
massive visual objects. On the other hand, to understand the causal
relationship between concept representation and behavior, it is necessary to
manipulate the concepts, preserving all other low-level features, to generate
visual objects. This is unexplored territory for AI. It poses a new technical
challenge for image generation, namely controllable visual object generation
based on concept representation.

Figure 1: Motivations of our work.

Controllable generation models have made great progress in AI community, for
example, conditional generation models based on GAN Goodfellow et al. (2020);
Tao et al. (2022) and diffusion Song et al. (2020); Ho et al. (2020); Ho and
Salimans (2022). These conditional generation models have been applied in many
fields, including text-to-image/video generation Rombach et al. (2022); Ramesh
et al. , image inverse problem Kawar et al. (2022); Chung et al. (2022); Meng
et al. (2021) and biomedical imagingSong et al. (2021); Özbey et al. (2023).
The conditions for controllable image generation can come from multiple
aspects, such as text, sketches, edge maps, segmentation maps, depth maps
Rombach et al. (2022); Ramesh et al. ; Meng et al. (2021); Zhang et al.
(2023); Yu et al. (2023); Bansal et al. (2023). However, these conditions do
not include human subjective feelings nor human feedback, resulting in
generated images misaligned with human needs. For instance, generated images
by recommendation system may not meet human preference. To align the generated
images with human needs, some pioneer works brought human feedback (e.g., the
human visual preference score obtained offline Wu et al. (2023); Kirstain et
al. (2023), the human-in-the-loop visual comparison decision von Rütte et al.
(2023); Fan et al. (2023); Tang et al. (2023)) into conditional generative
models. Nevertheless, these works have not consider prior knowledge of
cognitive science, such as the factors that have the greatest impact on human
decision-making, that is, concepts, as control variables for image generation.

A large number of human research suggest that similarity judgment tasks are an
effective experimental paradigm for revealing human concept representations
Roads and Love (2023); Hebart et al. (2020). In these tasks, human subjects
are asked to compare different visual stimuli and make decisions based on
their similarity. AI models, such as Peterson et al. (2018); Marjieh et al.
(2022b, a); Muttenthaler et al. (2022a); Jha et al. (2023); Fu et al. (2023),
are proposed to predict the subjects’ decisions, under an assumption that
human perception of visual objects can be encoded into a low-dimensional
mental representation, namely concept embedding. The distance between visual
objects in this concept space reflects the distance of visual objects in the
human’s mind. Human decision-making behavior can be explained by different
dimensions of concept representation. Thus, aligning AI models with humans in
terms of concept representation would naturally align their outputs as well.
Also, compared with existing controllable image generation methods, a
controllable generation model based on concept representation would control
human decision-making behavior more effectively.

In this study, we propose a Concept based Controllable Generation (CoCoG)
framework. CoCoG utilize concept embeddings as conditions for the image
generation model, bridging cognitive science and AI. CoCoG comprises two
parts: a concept encoder for learning concept embedding via predicting human
behaviors and a concept decoder which employs a conditional diffusion model
for mapping concept embedding to visual stimuli through a two-stage generation
strategy.

CoCoG has three main contributions:

  * •

CoCoG’s concept encoder can predict human visual similarity decision-making
behaviors with higher accuracy than the state-of-the-art (SOTA) model (i.e.,
VICE Muttenthaler et al. (2022b)), uncovering a reliable, interpretable
concept space of humans (Figure 3).

  * •

CoCoG’s concept decoder can generate visual stimuli by controlling the concept
embedding. The generated visual stimuli have diversity and high consistency
with the target concept embeddings (Figure 4&5). The generation can be guided
with text prompts (Figure 6&7).

  * •

CoCoG can manipulate the human similarity decision-making behavior in a highly
controllable way by controlling the concepts of generated visual stimuli
(Figure 8).

Figure 2: The framework of CoCoG. (a) The concept encoder for learning concept
embeddings using a similarity judgment behavior dataset. Visual objects are
processed through the CLIP image encoder to obtain CLIP image embeddings, and
then passed through a learnable concept projector to obtain concept
embeddings. Then, we can predict similarity judgment behaviors by compute
similarity with others. (b) The stage I of the concept decoder, the prior
diffusion for determining the concept embedding based on our desired judgment
behavior (e.g., here modifying the concept “colorful”). Then, we train a
diffusion model conditioned on the concept embedding to generate the
corresponding CLIP embedding. (c) Stage II of the concept decoder, the CLIP
guided generation. It uses the CLIP embedding as a condition to guide the pre-
trained image diffusion generation model to generate VAE latent, which are
then processed through the VAE decoder to produce the generated visual object.

##  2 Method

The CoCoG method comprises two parts: a concept encoder and a two-stage
concept decoder, as shown in Figure 2.

###  2.1 Concept encoder for embedding low-dimensional concepts

The first step in CoCcoG is to train a concept encoder to learn the concept
embeddings of visual objects (Figure 2a). Given a dataset of visual objects
$X$, for each visual object $x$ in the dataset, we first input it into the
CLIP image encoder $f$ to extract the CLIP embedding $h\in\mathcal{R}^{D}$.
Then, the CLIP embedding is input into a learnable concept projector $g$,
thereby generating the concept embedding of the visual object
$c\in\mathcal{R}^{d}$:

CLIP embedding: $\displaystyle h=f(x),$ (1) concept embedding: $\displaystyle
c=g(h),$

where the each dimension in concept embedding $c$ represents aninterpretable
concept, and the corresponding activation of this dimension indicates the
activation strength of the concept in the visual object. In short, we have
$c=g(f(x))$.

To train this model, we used the triplet odd-one-out similarity judgment task
in the THINGS dataset Hebart et al. (2023). In this task, participants are
asked to view three visual objects and consider the similarity between each
pair of visual objects to determine the similar pair and select the remaining
one as the odd-one-out.

Similar to previous works Zheng et al. (2018); Hebart et al. (2020);
Muttenthaler et al. (2022b), we use the dot product similarity as the
similarity measurement function between concept embeddings (i.e.,
$S_{ij}=<c_{i},c_{j}>$) and use cross-entropy based on pairwise similarities
to predict human decisions. Therefore, for the concept embeddings
$c_{i},c_{j},c_{k}$ of three visual objects $x_{i},x_{j},x_{k}$, we have

$p(y)=CrossEntropy(S_{jk},S_{ik},S_{ij}),$ (2)

where $p(y)$ is the probability distribution of the triplet visual stimuli
$(x_{i},x_{j},x_{k})$ being the odd-one-out, with the highest probability
choice being the model’s predicted behavioral outcome. By comparing the
model’s predicted behavioral outcomes with the recorded behavioral outcomes
from human participants, we calculate the loss and perform backpropagation to
train the concept projector $g$. In the specific process, we added an $L_{1}$
regularization to $c$ to ensure the sparseness of low-dimensional concept
embedding. We verified different training strategies (see Appendix).

###  2.2 Two-stage concept decoder for controllable visual stimuli generation

After training the concept encoder, we can obtain the data triplet
$(x_{i},h_{i},c_{i})$ for each image in the dataset. Based on these data, we
then train a concept decoder to generate visual objects by controlling human
conceptual representations.

In our model, the concept embedding $c$ completely determines the distribution
of the human decision $y$, and the $h$ completely determines the distribution
of $c$. Therefore, the joint distribution of the human decisions $p(x,h,c,y)$
can be formulated as:

$p(x,h,c,y)=p(y)p(c|y)p(h|c)p(x|h).$ (3)

Next, we present each step of $p(c|y),p(h|c),p(x|h)$ respectively. $p(c|y)$
means finding the appropriate concept embedding $c$ given the behavioral
outcome $y$. This step is determined according to specific control objectives,
which we will discuss in chapter 4.2. For $p(c|y)$ and $p(x|h)$, we decompose
the concept decoder into two stages, the Prior Diffusion and the CLIP guided
generation, respectively, which execute the processes of generating CLIP
embedding $h$ from the concept embedding $c$ and generating visual object $x$
from the CLIP embedding $h$.

#### Stage I - Prior diffusion

Inspired by DALL-E 2 Ramesh et al. , we train a diffusion model conditioned on
the concept embedding $c$ to learn the distribution of CLIP embeddings
$p(h|c)$ (Figure 2b). The CLIP embedding $h$ obtained in this stage serves as
the prior for the next stage. We construct a lightweight U-Net:
$\epsilon_{prior}(h_{t},t,c)$, where $h_{t}$ represents the noisy CLIP
embedding in the diffusion time step $t$. We extract the CLIP embeddings and
the concept embeddings from ImageNet and use the obtained training pairs
$(h_{i},c_{i})$ to train the Prior Diffusion model. We employ the classifier-
free guidance method to train this conditional generative diffusion model. The
specific formulas can be found in the Appendix.

#### Stage II - CLIP guidance generation

After obtaining the CLIP embedding $h$ in Stage I, we model a generator
$p(x|h)$ to sample the visual object $x$ conditional to $h$ (Figure 2c). In
this study, we use the pre-trained SDXL and IP-Adapter modelsPodell et al.
(2023); Sauer et al. (2023); Ye et al. (2023). Speicfically, SDXL serves as
the backbone of the image diffusion generation model. Through the dual cross-
attention modules of the IP-Adapter, we input the CLIP embedding $h$ as a
condition, thereby guiding the denoising process of the U-Net. Similar to
Stage I, we have the model $\epsilon_{SD}(z_{t},t,h)$, where $z_{t}$ are the
noisy latents of SDXL’s VAE. Details can be found in the Appendix. Since we
freeze the pre-trained models without any modification, we can simply use the
existing functionalities of the pre-trained models and combine them with
concept embedding guidance.

Figure 3: The performance of the concept encoder in predicting and explaining
human behavior. (a) Our model’s prediction accuracy for similarity judgment
behavior is 64.07%, exceeding the previous SOTA model VICE’s 63.27% (blue
dashed line), with only slightly lower than the noise ceiling (gray dashed
line)The Pearson correlation coefficient between the similarity of visual
objects predicted by our model and by VICE is 0.94; (b) Example visual objects
and their concept embeddings, with dashed lines representing the 90th
percentile of activated concepts; (c) Example visual objects with significant
activation on the concept Home tools and Baked food, respectively.

#### CLIP embedding as an intermediate variable

In both processes from visual objects to concept embeddings and from concept
embeddings back to visual objects, we use CLIP embeddings as an intermediate
variable. This is for two purposes: 1) For the concept encoder, CLIP
embeddings are sufficiently low-dimensional and contain key information of
images. Previous study has shown that using CLIP embeddings along with simple
linear probing can well predict human behavior in similarity judgment tasks
Muttenthaler et al. (2022a). 2) For the concept decoder, using CLIP embeddings
helps us better utilize existing pre-trained conditional generative models.
Existing models can already use CLIP embeddings as a conditional input for the
generative model, allowing us to simply adopt a two-stage generation strategy.
We only need to train a Prior Diffusion model, which significantly reduces the
computational cost of training and inference.

##  3 Model Validation

###  3.1 Concept encoder can predict and explain human behaviors

We first validated the concept encoder from two aspects: the prediction
accuracy of human behaviors (Figure 3a) and the interpretability of the
learned lwo-dimensional concepts (Figure 3b&c). We used the THINGS Odd-one-out
dataset as the similarity judgment behavior dataset to train our concept
encoding model. We tested various configurations of the concept encoding model
(which will be detailed in the Appendix). Figure 3a shows the results of the
optimal model configuration. We used 42-dimensional concept embeddings for
comparison with previous SOTA model (VICE) and our experiments have found that
more than 42 dimensions only bring small improvements. In terms of behavioral
prediction, our best-performing model achieved an accuracy of 64.07% on the
THINGS Odd-one-out dataset, surpassing the previous best model VICE’s accuracy
of 63.27%. We also compared the similarity predictions of visual objects
between our model and VICE, and the Pearson correlation coefficient of their
predictions reached 0.94. This indicates that our model can accurately predict
human similarity judgment behaviors.

One major merit of our concept embedding is its good interpretability. Figure
3b shows the low-dimensional concept embeddings for the two example visual
objects. It is obvious that the concept embeddings encoded by CoCoG well
describe the visual objects, and the activation of concepts in a visual object
exhibits a favorable property of sparsity, which is in line with human
intuition (see Appendix). We used the CLIP-Dissect model to automatically
choose words from the lexicon to describe the concepts in each of the 42
dimensions. Note that the relationship between these words and the concepts is
not absolute and is only for examples. Figure 3 shows the visual objects with
the highest activation in the first two conceptual dimensions (i.e., home
tools and baked food). Importantly, we find that the visual objects that
significantly activate these dimensions are highly consistent with these two
concepts. This indicates that our model can effectively encode the conceptual
embeddings of visual objects, and the encoded concepts have good
interpretability.

###  3.2 Concept decoder can generate visual objects consistent with concept
embedding

Figure 4: The visual objects generated by controlling the concept embeddings.

In this section, we validate the generative effectiveness of the concept
decoder. Specific training parameters are shown in the Appendix. Figure 4
shows visual objects generated under the guidance of concept embeddings. These
visual objects generated from the same concept embedding are well-aligned with
the concept embedding and have good diversity, demonstrating that our
diffusion model can conditionally generate visual objects consistent with the
concept embeddings.

We quantify the similarity between the concept embeddings of the generated
visual objects and the target concept embeddings according to Eq.2. The
similarity between generated visual objects and target concept embeddings is
significantly higher than that between two random visual objects (Figure 5a).
Additionally, by adjusting the guidance scale, we can control the guiding
strength of the target concept embeddings, thereby controlling the similarity
and diversity of the generated visual objects (specific metrics calculation
can be found in the Appendix). As the guidance scale increases, the similarity
between the visual objects and the target concept embeddings increases while
the diversity decreases (Figure 5b).

Figure 5: Measurements of the performance of generated visual objects. (a) The
similarity between random visual objects and the target concept embeddings
(blue), and the similarity between generated visual objects and the target
concept embeddings (orange); (b) The similarity between visual objects and
target concept embeddings as the guidance scale changes (blue), and the
diversity of visual objects as the guidance scale changes (orange).

##  4 CoCoG for studying counterfactual explanations of human behaviors

According to the role of concept embedding, manipulating the concept embedding
can directly influence human similarity judgment behavior. Conversely, does
the change in visual objects that do not affect the concept embedding have no
impact on human behavior? CoCoG is an excellent tool to explore this
counterfactual question.

###  4.1 Flexible controlling of generated objects with text prompts

Figure 6: Visual objects generated with the same concept embedding combined
with different text prompts. The first image in the first row is generated
without using any text prompt, while other images are generated by using
different animal species as text prompts. The images in the second row are
generated by using different artistic styles as text prompts.

As shown in 6, we used the same concept embedding combined with different text
prompts to generate visual objects. Regardless of the changes in the category
and style of the visual objects, the images consistently retained the
characteristics of the concept embedding. Note that we fixed the random seed
to highlight the differences in text prompts. In theory, these completely
different visual objects would lead to similar judgment behaviors in
similarity judgment experiments.

Figure 7: Visual objects generated with different concept embeddings combined
with the same text prompt. The images in the upper and lower halves correspond
to each other. The images in the upper half were used to extract concept
embeddings, which were then combined with the text prompt ‘a teddy bear’ to
generate the new images in the lower half. Figure 8: Manipulating similarity
judgment behavior by key concepts intervention. (upper) The visual objects
generated by using “many colors” and “plant” as key concepts. (bottom) The
visual objects generated by using “ground animals”, “cotton clothing”, and
“baby toys” as key concepts.

Further, we show visual objects generated using the same text prompt combined
with different concept embeddings (Figure 7). The images in the upper half
were used to extract concept embeddings, which were then combined with text
prompts to generate the new images in the lower half. The generated images
reflect the property ”teddy bear” after adding the text prompt, with
preserving the concept embeddings. Therefore, contrary to the concept-editing
experiment, modifying text prompts do not lead to changes in judgment behavior
because the concept embedding is preserved.

###  4.2 Manipulating the similarity judgment decisions by intervening the key
concepts

In this experiment, we designed a simple scenario to show how CoCoG can
manipulate similarity judgment behavior. Suppose participants need to perform
a Two alternative forced choice experiment, i.e., they need to choose from
Reference 1 & 2 which is more similar to the Query. In this scenario, we can
directly increase or decrease certain concepts to make the Query more similar
to one of the references. Considering that we use dot product similarity, we
can simply choose the concept that is significant in the Reference but not in
the Query as the key concept. The results in Figure 8, show that the
preference of the Query for Reference 1 & 2 is manipulated by modifying two
and three concepts, with the concepts shifting to two different directions. It
is evident that the generated visual objects by CoCoG can change smoothly but
significantly with the manipulation of concepts.

This method provides us with an effective tool for analyzing the causal
mechanisms of concepts in similarity judgment tasks. For example, in the
experiment, we can directly manipulate the concepts of interest and observe
subsequent behavior. Researchers can actively choose to explore the impact of
certain concepts on behavior and analyze the causal relationship between
changes in concepts and judgment behavior. By focusing on informative concepts
and precisely controlling the activation values of concepts, this method is
expected to improve the efficiency of data collection and significantly reduce
the number of experimental trials needed (rather than using millions of trials
Hebart et al. (2023)).

##  5 Related Works

###  5.1 Concept Embeddings

Numerous computational models have been developed to model embeddings encoding
in similarity judgment tasks Roads and Love (2023); Hebart et al. (2020).
These methods aim to study concept embeddings through similarity judgment
tasks, either by optimizing concept embeddings for each object Zheng et al.
(2018); Roads and Love (2021); Hebart et al. (2020); Muttenthaler et al.
(2022b) or by using DNNs to predict human similarity judgment behavior
Peterson et al. (2018); Marjieh et al. (2022b, a); Muttenthaler et al.
(2022a); Jha et al. (2023); Fu et al. (2023). Previous methods, assuming
sparsity, continuity, and positivity of concept embeddings, learn low-
dimensional concept embeddings for each object through probabilistic models.
These embeddings have good interpretability but cannot generalize to new
objects. On the other hand, DNN-based methods construct concept embeddings
from neural activations by reducing the dimensionality of high-dimensional DNN
latent representations Jha et al. (2023), aligning DNN-based similarity
judgments with human judgments Muttenthaler et al. (2023), or using multimodal
inputs to improve similarity judgment predictions Marjieh et al. (2022b, a).
This approach can utilize state-of-the-art DNN models to form embeddings that
naturally generalize to new objects. To enhance behavioral experiments, Roads
et al. utilized active learning coupled with a trial selection strategy for
efficient concept embedding inference Roads and Love (2021). Similarly,
DreamSim leveraging Stable Diffusion, generates synthetic data within
specified categories to create a Perceptual dataset, aimed at studying Human
perceptual judgments Fu et al. (2023).

###  5.2 Conditional Diffusion Models

Recently, conditional generative models based on diffusion models have seen
significant development. The classifier-free guidance method has demonstrated
strong controllable generative capabilities through supervised learning Ho and
Salimans (2022). These methods dominate the tasks of text-to-image generation.
From the widely used Stable Diffusion Rombach et al. (2022); Podell et al.
(2023); Sauer et al. (2023) to subsequent methods like ControlNet Zhang et al.
(2023) and IP-Adapter Ye et al. (2023), they have added more controllable
conditions to conditional generation, such as edge maps, segmentation maps,
depth maps Rombach et al. (2022); Ramesh et al. ; Meng et al. (2021); Zhang et
al. (2023); Yu et al. (2023); Bansal et al. (2023). Additionally, conditional
generative models based on human feedback and preferences have also shown
potential. Works like FABRIC and DPOK use real-time similarity tasks to guide
models to generate images meeting human needs, proving that human feedback can
provide more nuanced control over conditional generative models von Rütte et
al. (2023); Fan et al. (2023); Tang et al. (2023). Methods like Pick-a-pic and
Human Preference Score generate images aligned with human aesthetics Kirstain
et al. (2023); Wu et al. (2023). These advancements demonstrate that, in
comparison to standard text-to-image frameworks, conditional generative models
possess significant potential for more closely meeting human needs and
preferences.

##  6 Discussion

We proposed the CoCoG model, capable of predicting human visual similarity
judgment behavior and learning human conceptual embeddings for visual objects.
It can also efficiently and controllably generate visual objects in line with
human cognition (Fig 3), manipulating human similarity judgment behavior and
studying causal mechanisms in human cognition (Fig 8).

#### Contributions to AI

Our approach bridges generative models and human visual cognition. Through the
concept encoding model, we align DNNs with human visual concept
representations, simulating human processing and responses to visual objects
more precisely, with potential to enhance AI’s visual understanding
capabilities; through controllable diffusion generation based on concept
embeddings, we make conditional generative models more closely linked to human
cognition and can manipulate human behavior through generated stimuli,
promising to improve control and safety in AI-human interactions.

#### Contributions to cognitive science

Our approach significantly expands research on human visual cognition in
cognitive science. With the concept encoding model, we achieved interpretable
concept encoding for visual objects (Figure 3b); with controllable diffusion
generation based on concept embeddings, we can generate a rich variety of
natural stimuli to control human similarity judgment behaviors (Figure Roads
and Love (2021); Fu et al. (2023)). By combining advanced AI models with
cognitive science research methods, we greatly enhance the efficiency and
breadth of visual cognition research. Additionally, this method may reveal
causal mechanisms in human visual cognition, offering new perspectives for
understanding human cognitive processes.

#### Future directions

In the future, we will extend the paradigm of human visual cognition research
to the study of AI representational spaces, which would help align AI with
humans and provide new insights for understanding AI’s cognition. Also, we
recommend to bring optimal experimental design Rainforth et al. (2023); Roads
and Love (2021) into human experiments. It will largely improve the efficiency
of human behavioral data collection, facilitate model learning, optimizing
experimental paradigms with broader applications in cognitive science and AI.

## Ethical Statement

The human behavioral data in this study is from THINGS public dataset. No
animal or human experiments are involved.

## Acknowledgments

This work is supported by the National Key R&D Program of China
(2021YFF1200804), Shenzhen Science and Technology Innovation Committee
(20200925155957004, KCXFZ20201221173400001, SGDX2020110309280100), Guangdong
Provincial Climbing Plan (pdjh2024c21310).

## Appendix

##  Appendix A Concept encoder

###  A.1 Objective functions

Our concept encoder is designed to learn human concept representations of
visual objects by fitting human similarity judgment behavior. In line with
prior studies Muttenthaler et al. (2022a), we employ two strategies to train
the concept encoder.

#### Fitting human behavior

Given an image similarity $S$ and a triplet $\\{c_{i},c_{j},c_{k}\\}$ with
$c=g_{\theta}(f(x))$ and $S_{ij}=\langle c_{i},c_{j}\rangle$, we use the
softmax of object similarity to model the probability that
$\\{a,b\\}\in\\{i,j,k\\}$ is the most similar pair:

$p(\\{a,b\\}|\\{i,j,k\\}):=\frac{\exp(S_{ab})}{\exp(S_{ij})+\exp(S_{ik})+\exp(S_{jk})}$
(4)

Our model aims to maximize the log-likelihood of the odd-one-out judgments.
Diverging from previous work, we introduce an $l_{1}$ regularization term in
the concept $c$ to maintain the sparsity of concept embeddings:

$\underset{\theta}{\text{arg min}}\ -\frac{1}{n}\sum_{s=1}^{n}\log
p(\\{a,b\\}|\\{i,j,k\\})+\lambda\|c\|$ (5)

#### Fitting VICE embeddings

To facilitate comparisons with current methodologies, our model is also
capable of fitting existing concept embeddings. As an example, we use VICE
Muttenthaler et al. (2022b) embeddings:

$\underset{\theta}{\text{arg
min}}\sum_{i=1}^{m}\|\bar{c}_{m}-g_{\theta}(f(x_{m}))\|+\lambda\|c_{m}\|$ (6)

where $\bar{c}_{m}$ represents the VICE dimensions for image $m$, and
$c_{m}=g_{\theta}(f(x_{m}))$ denotes the concept embeddings predicted by our
model for image $m$.

###  A.2 Model comparison

In Table 1, we compare various network architectures in terms of their ability
to predict behavioral outcomes. As shown, we have evaluated different
combinations of linear projection (LinProj) and multi-layer perceptron (MLP)
with embedding-based (Emb) and behavior-based (Beh) objective functions. The
results are benchmarked against a baseline (Chance) and an upper bound (noise
ceiling) to provide context for their performance.

From these comparisons, we can observe that the MLP models, especially when
combined with behavior-based objectives, tend to offer superior performance,
as indicated by their higher accuracy in the test set. This suggests that the
MLP architecture, particularly when aligned with behavior-oriented training,
is more effective in capturing the nuances of human conceptual
representations. But for comparison with VICE, we use linear projection
mbedding-based objective functions for subsequent concept decoder.

Table 1: Comparison of different networks architectures for behavioral prediction. The accuracy in the test set for each model is listed. CoCoG variants of the concept encoder are compared to a baseline (Chance) and an upper bound (noise ceiling). Emb and Beh mean different objective functions.  | Behavioral prediction
---|---
Chance* |  33.33%
LinProj + Emb | 61.90%
MLP + Emb | 63.02%
LinProj + Beh | 62.53%
MLP + Beh | 64.07%
noise ceiling* | 67.22%

###  A.3 Additional results for model validation

Figure 9: Model generalization performance within a dataset, across datasets,
and across tasks.

#### Generalization across objects, datasets, and tasks

Figure 9 demonstrates our model’s performance within and across datasets, as
well as its adaptability to different task paradigms. In the context of the
THINGS dataset Hebart et al. (2023), our model exhibits strong generalization
capabilities, as evidenced by its accuracy in predicting novel odd-one-out
decisions in both the validation set (Novel Triplets) and a subset of objects
(Novel Objects), achieving accuracies of 64.07% and 59.55%, respectively. The
object similarity computed by our model’s embeddings correlates highly with
the VICE embeddings, showcasing Pearson correlation coefficients of 0.94 and
0.92 for the Novel Triplets and Novel Objects experiments, respectively. These
results underscore the robustness of our model in handling within-dataset
variations.

When focusing on the cross-dataset scenario with the Food Dataset Carrington
et al. (2024), our model continues to maintain high performance standards.
Here, it achieves a prediction accuracy of 72.41% on the validation set after
training on the respective train set. The object similarity calculated by our
model in this novel food dataset exhibits a significant Pearson correlation
with VICE embeddings, indicated by a coefficient of 0.81. This outcome
demonstrates the model’s capacity to adapt and perform effectively across
different content domains.

Additionally, we explored the cross-task generalization ability of our model
through the Multi-arrangement task paradigm King et al. (2019). When trained
on this new task paradigm, the object similarity calculated by our model
showed a notable correlation with the experimental data, indicated by a
Pearson coefficient of 0.62. The alignment of model-predicted and
experimentally measured representation distance matrices, particularly
concerning object classification, further validates our model’s efficacy.

Figure 10: Concept prediction by concept encoder. (a) Predicted accuracies for
each concept of VICE; (b) images with high activation in selected concepts.

#### Concept encoder performance

Figure 10 showcases the proficiency of our concept encoder in extracting and
representing various concepts. Our analysis reveals a significant negative
correlation between the importance of a concept and its predictability,
quantified by Pearson’s r. Specifically, the correlation coefficient for this
relationship stands at -0.75 (p = 1.3e-8), suggesting that concepts deemed
more important tend to be more predictable, which aligns with our initial
expectations. For this evaluation, we utilized concept embeddings from
previous research as a benchmark for comparison.

Complementing this quantitative analysis, the visual representations provided
in the figure illustrate our model’s effectiveness in activating multiple
conceptual dimensions. These example images clearly demonstrate the model’s
nuanced capability to recognize and encode a diverse range of concepts,
thereby reaffirming its applicability and effectiveness in complex conceptual
understanding tasks.

Figure 11: Model for concept description selection.

###  A.4 Description of concepts

Inspired by the CLIP-Dissect model Oikarinen and Weng (2022), we adopt a
mutual information maximization method to select descriptions for each concept
dimension. As depicted in Figure 11, we compiled a descriptive word dataset
$T$ containing 23,153 words. For a given concept $C_{k}$, we calculate the
mutual information between the descriptions and the maximum activation visual
objects $B_{k}$ of the concept dimension, selecting words that yield the
highest mutual information. We employ the SoftWPMI equation, as utilized by
CLIP-Dissect:

$\text{sim}(t_{m},q_{k};P)=\log\mathbb{E}[p(t_{m}|B_{k})]-\lambda\log
p(t_{m})$ (7)

Index | Language Description
---|---
1 | home tools, metal tools, hand tools, metallic tools, tools
2 | baked food, prepared food, food, made dish, entrees
3 | plant, green plants, green plants and herbs, plants, common in outdoor
4 | ground animals, small animals, land animals, mammal, animal type
5 | household furniture, furniture, main component of room, affordability, living furniture
6 | cotton clothing, clothing, clothes, things to wear, unisex
7 | decorative, gaudiness, decoration, intricacy, jewelry
8 | outdoor objects, building materials, ground, foundation, protruding
9 | pointy tools, hand tools, sharp tools, metal tools, metallic tools
10 | wood, wood-colored, made of wood, furniture, building materials
Table 2: Five language descriptions found by the model that best match the
first ten concepts.

##  Appendix B Concept decoder

In this section, we provide a concise overview of the conditional diffusion
model framework used in our concept decoder, following the presentation of
continuous-time diffusion models in Song et al. (2020); Karras et al. (2022).

#### Diffusion models

Diffusion Models (DMs) engage in a generative process by transforming high-
variance Gaussian noise into structured data representations. This
transformation is achieved by gradually reducing noise levels across a
sequence of steps. Specifically, we begin with a high-variance Gaussian noise
$x_{M}\sim\mathcal{N}(0,\sigma_{\text{max}}^{2})$ and systematically denoise
it through a series of steps to obtain $x_{t}\sim p(x_{t};t)$, where
$\sigma_{t}<\sigma_{t+1}$ and $\sigma_{M}=\sigma_{\text{max}}$. For a well-
calibrated DM, and with $\sigma_{0}=0$, the final $x_{0}$ aligns with the
original data distribution.

#### Sampling process

The sampling in DMs is implemented by numerically simulating a Probability
Flow ordinary differential equation (ODE) or a stochastic differential
equation (SDE). The ODE is represented as:

$dx=-\dot{\sigma}(t)\sigma(t)\nabla_{x}\log p(x;t)dt,$ (8)

where $\nabla_{x}\log p(x;t)$ is the score function, and $\sigma(t)$ is a pre-
defined schedule with its time derivative $\dot{\sigma}(t)$. The SDE variant
includes a Langevin diffusion component and is expressed as:

$\begin{split}dx=&-\dot{\sigma}(t)\sigma(t)\nabla_{x}\log p(x;t)dt\\\
&-\beta(t)\sigma^{2}(t)\nabla_{x}\log p(x;t)dt\\\
&+\sqrt{2\beta(t)}\sigma(t)d\omega_{t},\end{split}$ (9)

where $d\omega_{t}$ is the standard Wiener process.

#### Training of DMs

The core of DM training is to learn a model $s_{\theta}(x;t)$ for the score
function. This is typically achieved through denoising score matching (DSM),
where $\epsilon_{\theta}$ is a learnable denoiser. The training process can be
formulated as:

$\mathbb{E}_{(x_{0},c)\sim p_{\text{data}}(x_{0},c),(n_{t},t)\sim
p(n_{t},t)}\left[\|\epsilon_{\theta}(x_{0}+n_{t};t,c)-x_{0}\|_{2}^{2}\right],$
(10)

where $n_{t}$ is Gaussian noise with variance $\sigma_{t}^{2}$, and $c$
represents a condition.

###  B.1 Stage I - prior diffusion

The training of the prior diffusion stage is a critical step in our concept
decoder, employing the classifier-free guidance technique in conjunction with
data pairs of CLIP embeddings and concept embeddings $(h_{i},c_{i})$. Drawing
on principles from advanced generative models, our prior diffusion process is
conditioned on the concept embedding $c$ to effectively learn the distribution
of CLIP embeddings $p(h|c)$. The CLIP embedding $h$ obtained in this stage
serves as the prior for the subsequent stage. Our model architecture includes
a lightweight U-Net, denoted as $\epsilon_{prior}(h_{t},t,c)$, where $h_{t}$
represents the noisy CLIP embedding at the diffusion time step $t$. Training
pairs consisting of CLIP and concept embeddings are extracted from the
ImageNet dataset, which have over 1 million images. These pairs are then used
to train the Prior Diffusion model. The model is proficiently trained using
the classifier-free guidance method, which ensures a balance between fidelity
to the conditioning signal and the generative diversity of the outputs.

#### Classifier-free guidance technique

The Classifier-Free Guidance method plays a pivotal role in directing the
iterative sampling of a Diffusion Model (DM) in response to a given condition,
specifically a concept $c$. This technique operates by harmonizing the outputs
from both a conditional model and an unconditional model. The formulation of
the combined model, $\epsilon^{w}_{prior}(h;t,c)$, is given as:

$\epsilon^{w}_{prior}(h;t,c)=(1+w)\epsilon_{prior}(h;t,c)-w\epsilon_{prior}(h;t),$
(11)

where $w\geq 0$ indicates the guidance scale. This approach enables
simultaneous training of the unconditional model alongside the conditional
model within a single network architecture. It is accomplished by
intermittently replacing the concept $c$ with a null vector, as per Equation
(3), at a fixed proportion of the time, like 10%. The chief application of
this method is to refine the quality of the samples generated by DMs,
balancing it against the diversity of the outputs.

###  B.2 Stage II - CLIP guidance generation

In Stage II of our concept decoder, the CLIP embedding $h$ obtained from the
prior diffusion training serves as the basis for generating visual objects $x$
conditioned on $h$. For this purpose, we employ the integration of pre-trained
models, namely SDXL and IP-Adapter Podell et al. (2023); Ye et al. (2023), to
achieve efficient and high-quality image generation.

The backbone of our image generation process is the SDXL model, known for its
robustness in text-to-image diffusion processes. By incorporating the IP-
Adapter, which introduces dual cross-attention modules, the CLIP embedding $h$
is effectively used as a conditional input, thereby guiding the denoising
process in the U-Net architecture. The combined model for this process is
denoted as $\epsilon_{SD}(z_{t},t,h)$, where $z_{t}$ represents the noisy
latents from SDXL’s Variational Autoencoder (VAE).

#### Advantages of pre-Trained model integration

Utilizing these pre-trained models allows us to leverage their inherent
capabilities without the need for additional modifications. This approach not
only simplifies the implementation process but also ensures the retention of
the quality attributes inherent in these models. The integration of concept
embedding guidance with these pre-trained models facilitates the generation of
visual objects that are closely aligned with the concept information encoded
in $h$.

#### SDXL-turbo for enhanced speed

To further enhance the efficiency of our model, we also explore the use of
SDXL-Turbo Sauer et al. (2023), a distilled version of SDXL, optimized for
real-time synthesis. This model is particularly advantageous in scenarios
where rapid generation of high-fidelity images is required.

#### IP-Adapter

The IP-Adapter, with its relatively lightweight architecture, has demonstrated
its effectiveness in adding image prompt capability to pre-trained text-to-
image models. Its ability to work in tandem with text prompts for multimodal
image generation broadens the scope of applications for our concept decoder.

###  B.3 Performance Metrics

#### Similarity

We define similarity using the dot product as mentioned in the Objective
Functions section. Specifically, the similarity $S_{ij}$ between the target
concept embedding and the concept embedding extracted from the generated image
is calculated as:

$S_{ij}=\langle c_{i},c_{j}\rangle$ (12)

where $c_{i}$ and $c_{j}$ represent the respective concept embeddings.

#### Diversity

Inspired by works in the literature von Rütte et al. (2023); Corso et al.
(2023), we describe image diversity for a single concept embedding using In-
batch CLIP similarity. We use this approach to assess the variety of images
generated from the same concept embedding. In-batch CLIP similarity quantifies
how different the generated images are from one another within the same batch,
reflecting the diversity of outputs for a given concept embedding. The In-
batch Image Diversity $d$ for a batch of images $x_{1},\ldots,x_{n}$ is
defined as follows:

$d(x_{1},\ldots,x_{n})=1-\frac{2}{n(n-1)}\sum_{i=2}^{n}\sum_{j=1}^{i-1}\text{CLIP}(x_{i},x_{j})$
(13)

where $\frac{n(n-1)}{2}$ is the number of elements in the upper triangular
cosine similarity matrix.

## References

  * Bansal et al. [2023] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Universal guidance for diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 843–852, 2023.
  * Carrington et al. [2024] Madeline Carrington, Alexander G Liu, Caroline Candy, Alex Martin, and Jason A Avery.  Naturalistic food categories are driven by subjective estimates rather than objective measures of food qualities.  Food Quality and Preference, 113:105073, 2024.
  * Chung et al. [2022] Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye.  Improving diffusion models for inverse problems using manifold constraints.  Advances in Neural Information Processing Systems, 35:25683–25696, 2022.
  * Corso et al. [2023] Gabriele Corso, Yilun Xu, Valentin De Bortoli, Regina Barzilay, and Tommi Jaakkola.  Particle guidance: non-iid diverse sampling with diffusion models.  In NeurIPS 2023 Workshop on Deep Learning and Inverse Problems, 2023.
  * Fan et al. [2023] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee.  Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models.  arXiv preprint arXiv:2305.16381, 2023.
  * Fu et al. [2023] Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola.  Dreamsim: Learning new dimensions of human visual similarity using synthetic data.  arXiv preprint arXiv:2306.09344, 2023.
  * Goodfellow et al. [2020] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 63(11):139–144, 2020.
  * Hebart et al. [2020] Martin N Hebart, Charles Y Zheng, Francisco Pereira, and Chris I Baker.  Revealing the multidimensional mental representations of natural objects underlying human similarity judgements.  Nature human behaviour, 4(11):1173–1185, 2020.
  * Hebart et al. [2023] Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kidder, Anna Corriveau, Maryam Vaziri-Pashkam, and Chris I Baker.  Things-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior.  Elife, 12:e82580, 2023.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  arXiv preprint arXiv:2207.12598, 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in neural information processing systems, 33:6840–6851, 2020.
  * Jha et al. [2023] Aditi Jha, Joshua C Peterson, and Thomas L Griffiths.  Extracting low-dimensional psychological representations from convolutional neural networks.  Cognitive science, 47(1):e13226, 2023.
  * Karras et al. [2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  Advances in Neural Information Processing Systems, 35:26565–26577, 2022.
  * Kawar et al. [2022] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song.  Denoising diffusion restoration models.  Advances in Neural Information Processing Systems, 35:23593–23606, 2022.
  * King et al. [2019] Marcie L King, Iris IA Groen, Adam Steel, Dwight J Kravitz, and Chris I Baker.  Similarity judgments and cortical visual responses reflect different properties of object and scene categories in naturalistic images.  NeuroImage, 197:368–382, 2019.
  * Kirstain et al. [2023] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy.  Pick-a-pic: An open dataset of user preferences for text-to-image generation.  arXiv preprint arXiv:2305.01569, 2023.
  * Marjieh et al. [2022a] Raja Marjieh, Ilia Sucholutsky, Theodore R Sumers, Nori Jacoby, and Thomas L Griffiths.  Predicting human similarity judgments using large language models.  arXiv preprint arXiv:2202.04728, 2022.
  * Marjieh et al. [2022b] Raja Marjieh, Pol Van Rijn, Ilia Sucholutsky, Theodore Sumers, Harin Lee, Thomas L Griffiths, and Nori Jacoby.  Words are all you need? language as an approximation for human similarity judgments.  In The Eleventh International Conference on Learning Representations, 2022.
  * Medin et al. [1993] Douglas L Medin, Robert L Goldstone, and Dedre Gentner.  Respects for similarity.  Psychological review, 100(2):254, 1993.
  * Meng et al. [2021] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  In International Conference on Learning Representations, 2021.
  * Murphy and Medin [1985] Gregory L Murphy and Douglas L Medin.  The role of theories in conceptual coherence.  Psychological review, 92(3):289, 1985.
  * Muttenthaler et al. [2022a] Lukas Muttenthaler, Jonas Dippel, Lorenz Linhardt, Robert A Vandermeulen, and Simon Kornblith.  Human alignment of neural network representations.  In The Eleventh International Conference on Learning Representations, 2022.
  * Muttenthaler et al. [2022b] Lukas Muttenthaler, Charles Y Zheng, Patrick McClure, Robert A Vandermeulen, Martin N Hebart, and Francisco Pereira.  Vice: Variational interpretable concept embeddings.  Advances in Neural Information Processing Systems, 35:33661–33675, 2022.
  * Muttenthaler et al. [2023] Lukas Muttenthaler, Lorenz Linhardt, Jonas Dippel, Robert A Vandermeulen, Katherine Hermann, Andrew K Lampinen, and Simon Kornblith.  Improving neural network representations using human similarity judgments.  arXiv preprint arXiv:2306.04507, 2023.
  * Oikarinen and Weng [2022] Tuomas Oikarinen and Tsui-Wei Weng.  Clip-dissect: Automatic description of neuron representations in deep vision networks.  In The Eleventh International Conference on Learning Representations, 2022.
  * Özbey et al. [2023] Muzaffer Özbey, Onat Dalmaz, Salman UH Dar, Hasan A Bedel, Şaban Özturk, Alper Güngör, and Tolga Çukur.  Unsupervised medical image translation with adversarial diffusion models.  IEEE Transactions on Medical Imaging, 2023.
  * Peterson et al. [2018] Joshua C Peterson, Joshua T Abbott, and Thomas L Griffiths.  Evaluating (and improving) the correspondence between deep neural networks and human representations.  Cognitive science, 42(8):2648–2669, 2018.
  * Podell et al. [2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: Improving latent diffusion models for high-resolution image synthesis.  arXiv preprint arXiv:2307.01952, 2023.
  * Rainforth et al. [2023] Tom Rainforth, Adam Foster, Desi R Ivanova, and Freddie Bickford Smith.  Modern bayesian experimental design.  arXiv preprint arXiv:2302.14545, 2023.
  * [30] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.
  * Roads and Love [2021] Brett D Roads and Bradley C Love.  Enriching imagenet with human similarity judgments and psychological embeddings.  In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 3547–3557, 2021.
  * Roads and Love [2023] Brett D Roads and Bradley C Love.  Modeling similarity and psychological space.  Annual Review of Psychology, 75, 2023.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * Sauer et al. [2023] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach.  Adversarial diffusion distillation.  arXiv preprint arXiv:2311.17042, 2023.
  * Song et al. [2020] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In International Conference on Learning Representations, 2020.
  * Song et al. [2021] Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon.  Solving inverse problems in medical imaging with score-based generative models.  In International Conference on Learning Representations, 2021.
  * Tang et al. [2023] Zhiwei Tang, Dmitry Rybin, and Tsung-Hui Chang.  Zeroth-order optimization meets human feedback: Provable learning via ranking oracles.  arXiv preprint arXiv:2303.03751, 2023.
  * Tao et al. [2022] Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Changsheng Xu.  Df-gan: A simple and effective baseline for text-to-image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16515–16525, 2022.
  * von Rütte et al. [2023] Dimitri von Rütte, Elisabetta Fedele, Jonathan Thomm, and Lukas Wolf.  Fabric: Personalizing diffusion models with iterative feedback.  arXiv preprint arXiv:2307.10159, 2023.
  * Wu et al. [2023] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li.  Human preference score: Better aligning text-to-image models with human preference.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2096–2105, 2023.
  * Ye et al. [2023] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang.  Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.  arXiv preprint arXiv:2308.06721, 2023.
  * Yu et al. [2023] Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang.  Freedom: Training-free energy-guided conditional diffusion model.  arXiv preprint arXiv:2303.09833, 2023.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023.
  * Zheng et al. [2018] Charles Y Zheng, Francisco Pereira, Chris I Baker, and Martin N Hebart.  Revealing interpretable object representations from human behavior.  In International Conference on Learning Representations, 2018.


subject to variance from random initialization
($\pi_{0}:\texttt{\text{random\\_uniform}(0,1)}$, relative to the error in the
inexact critic ($\tau$), and over different levels of policy approximation
($k$). Results correspond to example (ii) from Sec. G.1.  Fig. 20:  Compares
the statistics of the policy optimization dynamics of PI, PMD and PMD(+mom)
subject to variance from random initialization
($\pi_{0}:\texttt{\text{random\\_uniform}(0,1)}$, relative to the error in the
inexact critic ($\tau$), and over different levels of policy approximation
($k$). Results correspond to example (iii) from Sec. G.1.  Fig. 21:  Compares
the statistics of the policy optimization dynamics of PI, PMD and PMD(+mom)
subject to variance from random initialization
($\pi_{0}:\texttt{\text{random\\_uniform}(0,1)}$, relative to the error in the
inexact critic ($\tau$), and over different levels of policy approximation
($k$). Results correspond to example (iv) from Sec. G.1.


captionUnknown document class

HySpecNet

    Hyperspectral Net
HySpecNet-11k

    Hyperspectral Net 11k
EnMAP

    Environmental Mapping and Analysis Program
L2A

    Level 2A
GPU

    graphics processing unit
RGB

    red, green and blue
HSI

    hyperspectral image
convolutional neural network

    CNN
PSNR

    peak signal-to-noise ratio
MSE

    mean squared error
SSIM

    structural similarity index measure
CR

    compression ratio
$\mathrm{bpppc}$

    bits per pixel per channel
$\mathrm{dB}$

    decibels
$\mathrm{MP}$

    megapixel
GSD

    ground sample distance
CAE

    convolutional autoencoder
LeakyReLU

    leaky rectified linear unit
PReLU

    parametric rectified linear unit
HardTanh

    hard hyperbolic tangent
1D-CAE

    1D-Convolutional Autoencoder
1D-CAE-Adv

    Advanced 1D-Convolutional Autoencoder
1D-CAE-Ext

    Extended 1D-Convolutional Autoencoder
SSCNet

    Spectral Signals Compressor Network
3D-CAE

    3D Convolutional Auto-Encoder

# HYSPECNET-11K: A LARGE-SCALE HYPERSPECTRAL DATASET FOR BENCHMARKING
LEARNING-BASED HYPERSPECTRAL IMAGE COMPRESSION METHODS

###### Abstract

The development of learning-based hyperspectral image compression methods has
recently attracted great attention in remote sensing. Such methods require a
high number of hyperspectral images to be used during training to optimize all
parameters and reach a high compression performance. However, existing
hyperspectral datasets are not sufficient to train and evaluate learning-based
compression methods, which hinders the research in this field. To address this
problem, in this paper we present HySpecNet-11k that is a large-scale
hyperspectral benchmark dataset made up of $11,483$ nonoverlapping image
patches. Each patch is a portion of 128x128 $\mathrm{pixels}$ with
$224\text{\,}\mathrm{spectral\ bands}$ and a ground sample distance of
$30\text{\,}\mathrm{m}$. We exploit HySpecNet-11k to benchmark the current
state of the art in learning-based hyperspectral image compression by
focussing our attention on various 1D, 2D and 3D convolutional autoencoder
architectures. Nevertheless, HySpecNet-11k can be used for any unsupervised
learning task in the framework of hyperspectral image analysis. The dataset,
our code and the pre-trained weights are publicly available at
https://hyspecnet.rsim.berlin.

Index Terms—  EnMAP, hyperspectral dataset, image compression, deep learning,
remote sensing.

##  1 Introduction

Advancements in hyperspectral imaging technologies have led to a significant
increase in the volume of hyperspectral data archives [1]. Dense spectral
information provided by hyperspectral images leads to a very high capability
for the identification and discrimination of the materials in a given scene.
However, to reduce the storage required for the huge amounts of hyperspectral
data, it is needed to compress the images before storing them. Accordingly,
one emerging research topic is associated to the efficient and effective
compression of hyperspectral images [2].

Many HSI compression algorithms are presented in the literature. Generally,
they can be divided into two categories:  i) traditional methods, e.g. [3, 4,
5]; and  ii) learning-based methods, e.g. [6, 7, 8, 9, 10].  The most popular
traditional algorithms are defined based on transform coding in combination
with a quantization step and entropy coding. In contrast, learning-based
methods mostly rely on convolutional autoencoders to reduce the dimensionality
of the latent space. Recent studies show that learning-based HSI compression
methods can preserve the reconstruction quality at lower rates compared to
traditional compression approaches.

Learning-based compression methods generally require a large number of
unlabeled images to optimize their model parameters during training. There are
only few hyperspectral benchmark datasets publicly available in remote sensing
(see Table 1). To the best of our knowledge, most of the existing datasets
only contain a single HSI, which is divided into patches for the training and
evaluation processes. Thus, they are not sufficient to train learning-based
compression methods to reach a high generalization ability as the models may
overfit dramatically, when using such training data from spatially joint
areas. The lack of a large hyperspectral dataset is an important bottleneck,
affecting the research and development in the field of learning-based HSI
compression. Hence, a large-scale benchmark archive consisting of a high
number of HSIs acquired in spatially disjoint geographical areas is needed.

Table 1: A summary of publicly available hyperspectral benchmark datasets and their characteristics. Dataset | Acquisition | Sensor | GSD | Spectral Range | #Bands | Dataset Size
---|---|---|---|---|---|---
Indian Pines | 1992 | AVIRIS | $20.0\text{\,}\mathrm{m}$ | $4002500\text{\,}\mathrm{nm}$ | $224$ |  00$0.021,025\text{\,}\mathrm{MP}$
Kennedy Space Center (KSC) | 1996 | AVIRIS | $18.0\text{\,}\mathrm{m}$ | $4002500\text{\,}\mathrm{nm}$ | $224$ |  00$0.314,368\text{\,}\mathrm{MP}$
Salinas Scene | 1998 | AVIRIS |  0$3.7\text{\,}\mathrm{m}$ | $4202450\text{\,}\mathrm{nm}$ | $224$ |  00$0.111,104\text{\,}\mathrm{MP}$
Pavia Center | 2001 | ROSIS |  0$1.3\text{\,}\mathrm{m}$ |  $430$ – 0$860\text{\,}\mathrm{nm}$ | $102$ |  00$1.201,216\text{\,}\mathrm{MP}$
Pavia University | 2001 | ROSIS |  0$1.3\text{\,}\mathrm{m}$ |  $430$ – 0$860\text{\,}\mathrm{nm}$ | $103$ |  00$0.2074\text{\,}\mathrm{MP}$
Botswana | 2001 | Hyperion | $30.0\text{\,}\mathrm{m}$ | $4002500\text{\,}\mathrm{nm}$ | $242$ |  00$0.377,856\text{\,}\mathrm{MP}$
Cooke City | 2008 | HyMap |  0$3.0\text{\,}\mathrm{m}$ | $4502480\text{\,}\mathrm{nm}$ | $126$ |  00$0.224\text{\,}\mathrm{MP}$
ShanDongFeiCheng (SDFC) | 2021 | HAHS |  0$0.5\text{\,}\mathrm{m}$ | $4001000\text{\,}\mathrm{nm}$ |  0$63$ |  00$0.721,801\text{\,}\mathrm{MP}$
HySpecNet-11k (ours) | $2022$ | EnMAP | $30.0\text{\,}\mathrm{m}$ | $4202450\text{\,}\mathrm{nm}$ | $224$ | $188.137,472\text{\,}\mathrm{MP}$

To address this problem, in this paper we introduce a new hyperspectral
benchmark dataset (denoted as HySpecNet-11k) and exploit it to benchmark the
current state of the art in learning-based HSI compression.

##  2 The HySpecNet-11k Dataset

HySpecNet-11k is made up of $11,483$ image patches acquired by the
Environmental Mapping and Analysis Program (EnMAP) satellite [11]. Each image
patch in HySpecNet-11k consists of 128x128 $\mathrm{pixels}$ and
$224\text{\,}\mathrm{bands}$ with a ground sample distance (GSD) of
$30\text{\,}\mathrm{m}$ (see Table 1).

To construct HySpecNet-11k, a total of $250$ EnMAP tiles acquired during the
routine operation phase between 2 November 2022 and 9 November 2022 were
considered. It is worth nothing that the considered tiles are associated with
less than $10\text{\,}\mathrm{\char 37\relax}$ cloud and snow cover. The tiles
were radiometrically, geometrically and atmospherically corrected (L2A water &
land product). Then, the tiles were divided into nonoverlapping image patches.
Therefore, cropped patches at the borders of the tiles were eliminated. Thus,
we were able to generate more than $45$ patches per tile resulting in an
overall number of $11,483$ patches for the full dataset. Due to the
L2A-processed data, the number of bands is reduced from $224$ to $202$ by
removing bands [$127141\text{\,}$] and [$161167\text{\,}$] that are affected
by strong water vapor absorption.

We provide predefined splits to make the results of the considered methods
reproducible. Therefore, we randomly divided the dataset into:  i) a training
set that includes $70\text{\,}\mathrm{\char 37\relax}$ of the patches,  ii) a
validation set that includes $20\text{\,}\mathrm{\char 37\relax}$ of the
patches, and  iii) a test set that includes $10\text{\,}\mathrm{\char
37\relax}$ of the patches.  Depending on the way that we used for splitting
the dataset, we define two different splits:  i) an easy split, where patches
from the same tile can be present in different sets (patchwise splitting); and
ii) a hard split, where all patches from one tile must belong to the same set
(tilewise splitting).

To get an overview of the dataset, Figure 1 illustrates representative
HySpecNet-11k images patches. It is worth noting that compression methods
generally do not require labeled training images and therefore our dataset
does not contain image annotations.

Fig. 1: True color representations of example images from our proposed
HySpecNet-11k dataset. Red, green and blue channels are extracted from EnMAP
bands $43$, $28$ and $10$ at wavelengths $634.919\text{\,}\mathrm{nm}$,
$550.525\text{\,}\mathrm{nm}$ and $463.584\text{\,}\mathrm{nm}$, respectively.

##  3 Learning-based Hyperspectral Image Compression

In order to provide an initial benchmarking of the proposed HySpecNet-11k
dataset in the framework of learning-based HSI compression, we train and
evaluate the following state of the art baseline methods:  i) 1D-Convolutional
Autoencoder(1D-CAE) [6];  ii) Advanced 1D-Convolutional Autoencoder(1D-CAE-
Adv) [7];  iii) Extended 1D-Convolutional Autoencoder(1D-CAE-Ext) [8];  iv)
Spectral Signals Compressor Network(SSCNet) [9]; and  v) 3D Convolutional
Auto-Encoder(3D-CAE) [10].  All methods are based on convolutional operations
paired with down- and upsamplings in the encoder and decoder, respectively.
They differ from each other with respect to the approaches considered for
spatial and spectral compression.

An overview of the considered methods is given in the following. For a
detailed description we refer the reader to the respective papers.

The 1D-Convolutional Autoencoder (1D-CAE) [6] applies pixelwise compression
without utilizing spatial content. The encoder consists of multiple 1D
convolutions in combination with two 1D max pooling layers that fix the
compression ratio (CR) to a factor of $4$. leakyrelu are present as activation
functions after each convolution. The decoder mirrors the encoder, however it
uses two upsampling layers to reverse the respective downsamplings from the
encoder. A sigmoid activation function scales the reconstructed pixel values
to the range from $01\text{\,}$ in the last decoder layer. Furthermore,
padding and unpadding are needed in encoder and decoder, respectively. The
Advanced 1D-Convolutional Autoencoder (1D-CAE-Adv) [7] adapts the decoder by
substituting the combinations of 1D convolutional layer and upsampling layer
with 1D transposed convolutions to make the upsampling operation trainable and
more adaptive. On the encoder side, the 1D max poolings are dropped for 1D
average poolings. Moreover, the Extended 1D-Convolutional Autoencoder (1D-CAE-
Ext) [8] adds a 1D batch normalization [12] between each (transposed)
convolutional and activation layer. The sigmoid activation in the last decoder
layer is replaced by a hard hyperbolic tangent (HardTanh) and 1D average
poolings are substituted by 1D max poolings again. The repetition of down- and
upsampling blocks enables the construction of models with different CRs.

The Spectral Signals Compressor Network (SSCNet) [9] encoder uses 2D
convolutions with parametric rectified linear units as activation after each
convolutional layer. Three 2D max pooling layers are added for a fixed spatial
compression by a factor of $64$. The final CR is set via the number of latent
channels in the bottleneck layer. Simultaneously, the ratio between input and
latent channels decides the spectral compression factor. 2D transposed
convolutions are used to reconstruct the HSIs on the decoder side. After each
transposed convolution there is a PReLU activation except for the last layer
where a sigmoid activation is used for scaling the outputs into the correct
range from $01\text{\,}$. For all convolutional layers a kernel size of 3x3
$\mathrm{pixels}$ ensures the integration of nearby spatial content into the
compression of the currently considered pixel.

The 3D-CAE [10] is built by three strided 3D convolutional layers, each of
which is followed by a 3D batch normalization [12] and a leaky rectified
linear unit (LeakyReLU). Residual blocks are added to increase the depth of
the network. Upsampling layers are furthermore present on the decoder side for
reconstruction. Padding operations are required as in the 1D-CAE. The 3D
kernels in the convolutional layers are able to integrate the local spatial
and spectral neighborhood jointly during the compression. Spatial and spectral
CRs are fixed to a factor of $64$ and $4$, respectively and the overall CR can
be set via the number of latent channels in the bottleneck of the network.

##  4 Experimental Results

Our code is implemented in PyTorch based on the CompressAI [13] framework. We
applied gradient clipping and a global min-max normalization to scale the
input data in the range between $01\text{\,}$ that is usually a requirement
for learning-based compression methods. As optimizer, we used Adam [14] with a
learning rate of $1\text{\times}{10}^{-4}$ for the 1D-CAE [6], 1D-CAE-Adv [7]
and 1D-CAE-Ext [8]. For SSCNet [9] and 3D-CAE [10] the learning rate was set
to $1\text{\times}{10}^{-5}$. We trained the networks using mean squared error
(MSE) as loss function until convergence on the validation set that took
$500\text{\,}\mathrm{epochs}$, $2000\text{\,}\mathrm{epochs}$ and
$1000\text{\,}\mathrm{epochs}$ for the 1D-CAEs, SSCNet and 3D-CAE,
respectively. Training runs were carried out on a single NVIDIA A100 SXM4 80
GB GPU and took between $110\text{\,}\mathrm{days}$ each, depending on the
method. In general, SSCNet requires fewer GPU hours than 1D-CAE and 3D-CAE.
For the 1D-CAE another factor is the CR. The higher the CR, the deeper the
network and thus the longer the training time. Therefore, the modified
versions of the 1D-CAE with increased CRs were trained for only
$250\text{\,}\mathrm{epochs}$ due to runtime limitations.

Rate-distortion performance of the baseline methods on the HySpecNet-11k test
set is shown in Figure 2. The results have been obtained by using the
HySpecNet-11k easy split as introduced in Section 2. Overall, the 1D-CAE
produces the highest quality reconstructions with a peak signal-to-noise ratio
(PSNR) of $54.847,359,908,951,65\text{\,}\mathrm{dB}$ at a relatively high
fixed rate of $8.079,207,921\text{\,}\mathrm{bpppc}$. Despite the added
trainable upsampling operations, the 1D-CAE-Adv reaches a
$0.662,468,817\text{\,}\mathrm{dB}$ lower PSNR than the 1D-CAE at the same
rate and the 1D-CAE-Ext only achieves a PSNR of
$43.078,505,211,406,28\text{\,}\mathrm{dB}$, while increasing training time
significantly due to batch normalization. As a result, further experiments
were only performed for the 1D-CAE. SSCNet is able to operate at a lower rate
of $2.534,653,465,346,536,5\text{\,}\mathrm{bpppc}$ due to spatial
dimensionality reduction, which on the other hand introduces blurry
reconstructions that reduce the PSNR to
$43.643,470,896,614,92\text{\,}\mathrm{dB}$. The 3D-CAE only achieves
$39.538,442,916,340,3\text{\,}\mathrm{dB}$ PSNR at
$2.019,801,980,198,015,4\text{\,}\mathrm{bpppc}$ while being particularly
unstable on the validation set during training. We would like to note that we
have observed similar behaviour on the HySpecNet-11k hard split, which shows
the high generalization ability of the considered methods due to our proposed
large-scale dataset.

$0$$1$$2$$3$$4$$5$$6$$7$$8$$9$$40$$45$$50$$55$$60$$65$Rate
[$\mathrm{bpppc}$]PSNR [$\mathrm{dB}$]1D-CAE [6]SSCNet [9]1D-CAE-Adv [7]3D-CAE
[10]1D-CAE-Ext [8] Fig. 2: Rate-distortion performance of learning-based
hyperspectral image compression methods on the test set of our proposed
HySpecNet-11k dataset (easy split). Rate is visualized in bits per pixel per
channel ($\mathrm{bpppc}$) and distortion is given as peak signal-to-noise
ratio (PSNR) in decibels ($\mathrm{dB}$).

To achieve a better comparison along several CRs, we modified the baseline
models by repeating the downsampling blocks in the 1D-CAE and varying the
number of latent channels for SSCNet and 3D-CAE. As seen in Figure 2, the
1D-CAE, which only compresses the spectral content, is superior in all cases
compared to both other methods that apply spatial downsampling. Even when
increasing the number of latent channels in the bottleneck, SSCNet and 3D-CAE
are not able to compensate the loss of information introduced by the spatial
compression. Thus, in contrast to RGB imagery, HSI compression methods should
focus implicitly on the spectral domain while maintaining the spatial
dimensions when applied on hyperspectral data with a low spatial resolution.

##  5 Conclusion

In this paper, we have introduced HySpecNet-11k that is a large-scale
hyperspectral benchmark dataset (which consist of $11,483$ unlabeled image
patches) for learning-based hyperspectral image compression problems. To the
best of our knowledge, HySpecNet-11k is the first publicly available benchmark
dataset that includes images acquired by the EnMAP satellite. It is worth
nothing that the use of HySpecNet-11k is not limited to image compression
problems and can be exploited for any unsupervised learning task. We believe
that HySpecNet-11k will make a significant advancement in the field of
unsupervised learning from hyperspectral data by overcoming the current
limitations of existing hyperspectral image datasets. With the continuous
release of EnMAP tiles we plan to enrich the dataset and develop further
extended versions of HySpecNet as a future work.

##  6 Acknowledgements

This work is supported by the European Research Council (ERC) through the
ERC-2017-STG BigEarth Project under Grant 759764.

## References

  * [1] D. Landgrebe, “Hyperspectral image data analysis,” _IEEE Signal Processing Magazine_ , vol. 19, no. 1, pp. 17–28, 2002.
  * [2] Y. Dua, V. Kumar, and R. S. Singh, “Comprehensive review of hyperspectral image compression algorithms,” _Optical Engineering_ , vol. 59, no. 9, pp. 090 902–090 902, 2020.
  * [3] S. Lim, K. Sohn, and C. Lee, “Compression for hyperspectral images using three dimensional wavelet transform,” _IEEE International Geoscience and Remote Sensing Symposium_ , pp. 109–111, 2001.
  * [4] B. Penna, T. Tillo, E. Magli, and G. Olmo, “A new low complexity klt for lossy hyperspectral data compression,” _IEEE International Symposium on Geoscience and Remote Sensing_ , pp. 3525–3528, 2006.
  * [5] Q. Du and J. E. Fowler, “Hyperspectral image compression using jpeg2000 and principal component analysis,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 4, no. 2, pp. 201–205, 2007.
  * [6] J. Kuester, W. Gross, and W. Middelmann, “1d-convolutional autoencoder based hyperspectral data compression,” _International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences_ , vol. 43, pp. 15–21, 2021.
  * [7] J. Kuester, W. Gross, S. Schreiner, M. Heizmann, and W. Middelmann, “Transferability of convolutional autoencoder model for lossy compression to unknown hyperspectral prisma data,” _IEEE Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing_ , pp. 1–5, 2022.
  * [8] J. Kuester, J. Anastasiadis, W. Middelmann, and M. Heizmann, “Investigating the influence of hyperspectral data compression on spectral unmixing,” _SPIE Image and Signal Processing for Remote Sensing_ , vol. 12267, pp. 125–135, 2022.
  * [9] R. La Grassa, C. Re, G. Cremonese, and I. Gallo, “Hyperspectral data compression using fully convolutional autoencoder,” _Remote Sensing_ , vol. 14, no. 10, p. 2472, 2022.
  * [10] Y. Chong, L. Chen, and S. Pan, “End-to-end joint spectral–spatial compression and reconstruction of hyperspectral images using a 3d convolutional autoencoder,” _Journal of Electronic Imaging_ , vol. 30, no. 4, p. 041403, 2021.
  * [11] L. Guanter, H. Kaufmann, K. Segl, S. Foerster, C. Rogass, S. Chabrillat, T. Kuester, A. Hollstein, G. Rossner, C. Chlebek _et al._ , “The enmap spaceborne imaging spectroscopy mission for earth observation,” _Remote Sensing_ , vol. 7, no. 7, pp. 8830–8857, 2015.
  * [12] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” _International Conference on Machine Learning_ , pp. 448–456, 2015.
  * [13] J. Bégaint, F. Racapé, S. Feltman, and A. Pushparaja, “Compressai: A pytorch library and evaluation platform for end-to-end compression research,” _arXiv preprint arXiv:2011.03029_ , 2020.
  * [14] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.

  *[HSI]: hyperspectral image
  *[HSIs]: hyperspectral image
  *[GSD]: ground sample distance
  *[EnMAP]: Environmental Mapping and Analysis Program
  *[L2A]: Level 2A
  *[1D-CAE]: 1D-Convolutional Autoencoder
  *[1D-CAE-Adv]: Advanced 1D-Convolutional Autoencoder
  *[1D-CAE-Ext]: Extended 1D-Convolutional Autoencoder
  *[SSCNet]: Spectral Signals Compressor Network
  *[3D-CAE]: 3D Convolutional Auto-Encoder
  *[CR]: compression ratio
  *[HardTanh]: hard hyperbolic tangent
  *[CRs]: compression ratio
  *[PReLU]: parametric rectified linear unit
  *[LeakyReLU]: leaky rectified linear unit
  *[MSE]: mean squared error
  *[1D-CAEs]: 1D-Convolutional Autoencoder
  *[GPU]: graphics processing unit
  *[PSNR]: peak signal-to-noise ratio
  *[$\mathrm{bpppc}$]: bits per pixel per channel
  *[$\mathrm{dB}$]: decibels
  *[RGB]: red, green and blue