# LDMol: Text-Conditioned Molecule Diffusion Model Leveraging Chemically
Informative Latent Space

Jinho Chang
Graduate School of AI
KAIST
Daejeon, South Korea
<EMAIL_ADDRESS>
&Jong Chul Ye
Graduate School of AI
KAIST
Daejeon, South Korea
<EMAIL_ADDRESS>

###### Abstract

With the emergence of diffusion models as the frontline of generative models,
many researchers have proposed molecule generation techniques using
conditional diffusion models. However, due to the fundamental nature of a
molecule, which carries highly entangled correlations within a small number of
atoms and bonds, it becomes difficult for a model to connect raw data with the
conditions when the conditions become more complex as natural language. To
address this, here we present a novel latent diffusion model dubbed LDMol,
which enables a natural text-conditioned molecule generation. Specifically,
LDMol is composed of three building blocks: a molecule encoder that produces a
chemically informative feature space, a natural language-conditioned latent
diffusion model using a Diffusion Transformer (DiT), and an autoregressive
decoder for molecule re. In particular, recognizing that multiple SMILES
notations can represent the same molecule, we employ a contrastive learning
strategy to extract the chemical informative feature space. LDMol not only
beats the existing baselines on the text-to-molecule generation benchmark but
is also capable of zero-shot inference with unseen scenarios. Furthermore, we
show that LDMol can be applied to downstream tasks such as molecule-to-text
retrieval and text-driven molecule editing, demonstrating its versatility as a
diffusion model.

##  1 Introduction

Designing compounds with the desired characteristics is the essence of solving
many chemical tasks. Inspired by the rapid development of generative models in
the last decades, _de novo_ molecule generation via deep learning models has
been extensively studied. Diverse models have been proposed for generating
molecules that agree with a given condition on various data modalities,
including string representations [44], molecular graphs [28], and point clouds
[20]. The attributes controlled by these models evolved from simple chemical
properties [33, 12] to complex preferences such as biological activity [48]
and multi-objective conditioning [26, 5]. More recently, as AI models’ natural
language comprehension ability has rapidly increased, molecule generation aims
to be controlled by natural language [29, 49].

Meanwhile, diffusion models [46, 19] have emerged as a frontline of generative
models over the past few years. Through a simple and stable training objective
of predicting noise from noisy data [19], diffusion models have achieved
highly realistic and controllable data generation [8, 22, 18]. Furthermore,
leveraging that the score function of the data distribution is learned in
their training [47], state-of-the-art image diffusion models enabled various
applications on the image domain [43, 23, 7]. Inspired by the success of
diffusion models, several papers suggested diffusion-based molecule generative
models on various molecule domains including a molecular graph [31], strings
like Simplified Molecular-Input Line-Entry System (SMILES) [13], and point
cloud [20].

Figure 1: Different strategies of data domain selection for molecule
generative models. (a) The model directly learns the raw representation of the
molecule such as Simplified Molecular-Input Line-Entry System (SMILES) tokens,
point cloud coordinates, etc. (b) An autoencoder can be employed to let the
generative model learn the latent distribution, but these latent spaces are
generally hard to learn without any regularization on the autoencoder. (c) A
regularized, chemically pre-trained encoder can provide a latent space
learnable by external generative models.

However, most molecule diffusion models presented so far have used a few,
relatively simple conditions to control. Also, the diffusion model was mostly
trained on a raw data representation (Figure 1-(a)). In any molecule modality,
the molecule’s vast and complex correlations are encapsulated between
relatively small dimensions compared to common data domains like images. As a
result, the output of the generative model trained on raw molecule data often
failed to faithfully follow the given conditions or showed poor data quality
(_e.g._ , low validity) as the condition became more sophisticated like
natural language. Furthermore, raw molecule modality has inevitable
discreteness (_e.g._ , atom and bond type, connectivity, SMILES tokens) which
makes it hard to connect the deeply studied concept of continuous diffusion
models to molecule generation. Naively converting discrete values into real
numbers wouldn’t provide rich information within those components, and still
retain certain restrictions on the data.

These issues lead us to the idea of molecule generation via latent diffusion
models, which have two major advantages: Building a diffusion model on a
latent space allows the model to move out of the regulated raw data domain and
work on in a minimally restricted continuous domain. Furthermore, by having
the encoder pre-construct a chemically useful and interpretable feature space,
the diffusion model can more easily connect the molecule data with the highly
complicated condition of natural texts. In fact, we show that a naive
autoencoder cannot provide a feature space that the diffusion model can learn
(Figure 1-(b)), and making the encoder provide a chemically informative
molecule feature (Figure 1-(c)) is essential to the performance of the latent
diffusion model (see Section 4.3).

Therefore, we proposed a Latent Diffusion Molecule generative model (LDMol)
that enables a text-conditioned molecule generation, trained on the latent
space of the separately pre-trained molecule encoder. To construct a feature
space that is chemically meaningful and easily learnable, we suggest a novel
contrastive learning strategy of training the encoder to encode a structural
similarity of the molecule. The experimental results show that LDMol can
generate molecules that are valid and also meet the input text condition.
Additionally, utilizing the modeled score function, LDMol has the potential to
be applied to different multi-modal downstream tasks such as molecule-to-text
retrieval, text-guided molecule editing, etc. We summarize the contribution of
this work as follows:

  * •

We propose a latent diffusion model LDMol for text-conditioned molecule
generation to generate more valid molecules that are better aligned to the
text condition. This approach demonstrates the potential of generative models
for chemical entities in a latent space.

  * •

We report the importance of preparing a chemically informative latent space
for the molecule latent diffusion model, and suggest a novel contrastive
learning method to train an encoder that captures the molecular structural
similarity.

  * •

LDMol outperforms the text-to-molecule generation baselines, and its modeled
conditional distribution allows LDMol to retain the advanced attributes of
diffusion models including various applications like molecule-to-text
retrieval and text-driven molecule editing.

##  2 Background

Diffusion generative models. Diffusion models first define a forward process
that perturbs the original data, and generates the data from the known prior
distribution by the learned reverse process of the pre-defined forward
process. For example, DDPM [19] fixed their forward process by gradually
adding Gaussian noise to the data, which can be formalized as follows:

$\displaystyle q(x_{t}|x_{t-1})$
$\displaystyle=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}I)$ (1)

where $\beta_{t},t=1,\dots,T$ is a noise schedule. This definition of forward
process allows us to sample $x_{t}$ directly from $q(x_{t}|x_{0})$ as follows,
where $\alpha_{t}=1-\beta_{t}$ and
$\overline{\alpha}_{t}=\prod_{i=1}^{t}\alpha_{i}$:

$\displaystyle x_{t}$
$\displaystyle=\sqrt{\overline{\alpha}_{t}}x_{0}+\sqrt{1-\overline{\alpha}_{t}}\epsilon\text{,
where }\epsilon\sim\mathcal{N}(0,I)$ (2)

The model $\epsilon_{\theta}$ learns the reverse process $p(x_{t-1}|x_{t})$ by
approximating $q(x_{t-1}|x_{t})$ with a Gaussian distribution
$p_{\theta}(x_{t-1}|x_{t})=\mathcal{N}(x_{t-1};\mu_{\theta}(x_{t},t),\sigma_{t}^{2}I)$
where

$\displaystyle\mu_{\theta}(x_{t},t)$
$\displaystyle=\frac{1}{\sqrt{\alpha_{t}}}\left(x_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\overline{\alpha}_{t}}}\epsilon_{\theta}(x_{t},t)\right)$
(3)

which can be trained by minimizing the difference between $\epsilon$ and
$\epsilon_{\theta}(x_{t},t)$:

$\displaystyle\theta^{*}$
$\displaystyle=\text{arg}\min_{\theta}\mathbb{E}_{x_{0},t,\epsilon}||\epsilon-\epsilon_{\theta}(x_{t},t)||_{2}^{2}$
(4)

Once $\theta$ is trained, novel data can be generated with the learned reverse
process $p_{\theta}$: starting from the random noise
$x_{T}\sim\mathcal{N}(0,{I})$, the output can be gradually denoised according
to the modeled distribution of $p_{\theta}(x_{t-1}|x_{t})$.

Various real-world data generation tasks require to generate data $x_{0}$ with
a given condition $c$. To build diffusion models that can generate data from
the conditional data distribution $q(x_{0}|c)$, the model that predicts the
injected noise should be also conditioned by $c$.

$\displaystyle\theta^{*}$
$\displaystyle=\text{arg}\min_{\theta}\mathbb{E}_{x_{0},y,t,\epsilon}||\epsilon-\epsilon_{\theta}(x_{t},t,c)||_{2}^{2}$
(5)

Although the reverse diffusion process with ancestral sampling of $x_{0}\sim
q(x_{0}|y)$ can be identical to unconditional diffusion models, [18] proposed
classifier-free guidance to improve the quality of the conditional diffusion
models’ output. Specifically, using a scalar guidance scale $\omega$,
classifier-free guidance sampling utilizes a linearly combined noise of
conditional and unconditional model output,

$\displaystyle\hat{\epsilon_{\theta}}(x_{t},t,c)=\epsilon_{\theta}(x_{t},t,\emptyset)+\omega\\{\epsilon_{\theta}(x_{t},t,c)-\epsilon_{\theta}(x_{t},t,\emptyset)\\}$
(6)

where $\emptyset$ denotes the special condition input for unconditional noise.

As the diffusion model’s target data domain becomes more complex and high-
dimensional, diffusion models began to perform their diffusion process on the
latent data space instead of the raw data [51, 39]. Specifically, using a pre-
trained pair of an encoder $\mathcal{E}$ and its corresponding decoder
$\mathcal{D}$, datapoint $x$ is encoded into $z:=\mathcal{E}(x)$ that is used
to train the diffusion model. Generated latent $z^{\prime}$ from the pre-
trained diffusion model is transformed back to
$x^{\prime}:=\mathcal{D}(z^{\prime})$ on the original data domain.

Conditional molecule generation. Compared to more common domains for
generative models like images or text, generative models for molecules are
more challenging since there are rigid constraints on the data modality
itself. String representations like SMILES has a strict syntax, which often
makes SMILES-based molecule generative models show low validity on their
output. A similar challenge also exists for the modalities of point clouds and
graphs to ensure the model output represents a valid molecule.

As a promising tool for many important chemical and engineering tasks like de
novo drug discovery and material design, conditional molecule generation has
been extensively studied with various models including recurrent neural
networks [44], graph neural networks [28], and variational autoencoders [14,
27]. With the advent of large and scalable pre-trained models with
transformers [53], the controllable conditions became more abundant and
complicated [1, 5], and reached a text-guided molecule generation [10, 49, 29]
which requires a deep comprehension ability for natural language.

Recent works attempted to import the success of the diffusion model into
molecule generation. Several graph-based and point cloud-based works have
built conditional diffusion models that could generate molecules with simple
physicochemical and biological properties [20, 31, 50]. The most related model
to our work would be TGM-DLM [13], which attempted a text-conditioned molecule
diffusion model trained on the sequence of tokenized SMILES indices.

##  3 Method

###  3.1 Chemically informative SMILES authoencoder

The primary goal of introducing autoencoders for image latent diffusion models
is to map raw images into a low-dimensional space which reduces the
computation cost [51, 39]. This is plausible because a high-resolution image
has an enormous dimension on a pixel domain, yet each pixel contains little
information.

In molecule generation, we chose the SMILES string as a base modality to
represent molecules, and built a SMILES encoder to map raw SMILES strings into
a latent vector. In this case, the role of our SMILES autoencoder has to be
different from the autoencoders for images. A molecule structure can be fully
expressed by a sequence of $L$ integers for SMILES tokens, where $L$ is the
maximum token length. However, each token carries a significant amount of
information, and hidden interactions between these tokens are much more
complicated than interactions between image pixels. Therefore, the SMILES
encoder should be more concerned with extracting chemical meaning into the
latent space rather than reducing dimensions, even if it results in a latent
space whose dimension is larger than the raw SMILES string. In fact, when a
SMILES autoencoder was trained with naive reconstruction loss only, we
observed that diffusion models failed to learn the latent space from this
autoencoder (as will be seen in Section 4.3). This phenomenon also occurs even
in the image latent to some extent, which is why autoencoders for image
diffusion models are usually trained with regularizations such as KL
divergence loss [17] or vector quantization [52].

Accordingly, here we propose three conditions that our SMILES autoencoder’s
latent space has to satisfy: the input should be reconstructible, embed
chemical information that diffusion models can learn, and have a small
dimension as possible. A number of molecule encoders have been presented that
can extract various useful chemical features [54, 29] including biochemical
activity or human-annotated descriptions [55, 30]. Nonetheless, these molecule
feature extractors didn’t aim to encode the entire information of the molecule
structure therefore the input cannot be restored from the model output.
Considering that an input molecule has to be reconstructed from the latent,
the most important characteristic that our SMILES latent should have is that
all SMILES representing the same molecule must be encoded into the same
vector.

Figure 2: Overview of the proposed molecule autoencoder and the latent
diffusion model. (a) SMILES encoder is trained with contrastive learning to
construct latent space that embeds a structural similarity. (b) After the
SMILES encoder is prepared, a compression linear layer and the autoregressive
decoder are trained to restore the encoder input. (c) Training and inference
process of the latent diffusion model conditioned by the output of the frozen
external text encoder.

Encoder design. In this respect, we trained our SMILES encoder with
contrastive learning similar to simCLR [6](Figure 2-(a)). Contrastive learning
aims to learn better representation by assimilating features containing
similar information (_i.e._ positive pair) and distancing semantically
unrelated features (_i.e._ negative pair). Taking advantage of the SMILES’s
characteristic that a single molecule can be represented with many different
SMILES strings, we defined two different SMILES from the same molecule as a
positive pair. In the process of matching the outputs of two SMILES in a
positive pair, the encoder learns the syntax and the information within SMILES
so that its output encapsulates the structural information of the input
molecule.

Specifically, a SMILES string is fed into the SMILES encoder with a special
"[SOS]" token which denotes the start of the sequence. For a batch of $N$
input SMILES $M=\\{m_{1},m_{2},\dots,m_{k},\dots,m_{N}\\}$, we prepare a
positive pair SMILES $m_{k}^{\prime}$ for each $m_{k}$ by SMILES augmentation
to construct
$M^{\prime}=\\{m_{1}^{\prime},m_{2}^{\prime},\dots,m_{k}^{\prime},\dots,m_{N}^{\prime}\\}$.
After $M$ and $M^{\prime}$ are passed through the SMILES encoder, we fed each
SMILES’ output vector corresponding to the [SOS] token into an additional
linear projection and normalization layer, denoting the output features as
$v_{k}$ and $v_{k}^{\prime}(k=1,2,\dots,N)$.

Assimilating $v_{k}$ and $v_{k}^{\prime}$ from the positive pairs and
distancing the others can be done by minimizing the following contrastive
loss, which is essentially cross-entropy loss where the logit is the cosine
similarity between given pairs.

$\displaystyle\mathcal{L}_{contrastive}(M,M^{\prime})=-\sum_{k=1}^{N}\log\frac{\exp(v_{k}\cdot
v_{k}^{\prime}/\tau)}{\sum_{i=1}^{N}\exp(v_{k}\cdot v_{i}^{\prime}/\tau)}\;$
(7)

Here, $\tau$ is a positive temperature parameter. To utilize a symmetric loss
against the input, we trained our encoder with the following loss function.

$\displaystyle\mathcal{L}_{enc}(M,M^{\prime})=\mathcal{L}_{contrastive}(M,M^{\prime})+\mathcal{L}_{contrastive}(M^{\prime},M)\;$
(8)

While our encoder was pre-trained to map structurally similar molecules to
similar vectors, we found that the encoder tried to map stereoisomers to
almost the same point in the latent space; indeed, in terms of structural
differences, stereoisomers are the most challenging molecules to distinguish.
To resolve this, if the training data has a stereoisomer, we considered it as
”hard-negative" samples and included it in the loss calculation batch with the
original data.

Compressing the latent space. The pre-trained SMILES encoder maps a molecule
into a vector of size $\\{L\times d_{enc}\\}$, where $d_{enc}$ is a feature
size for the encoder. To avoid the curse of dimensionality and construct a
more learnable feature space for diffusion models, we additionally employed a
linear compression layer (Figure 2-(b)) to reduce the dimension size from
$\\{L\times d_{enc}\\}$ to $\\{L\times d_{z}\\}$. The range of this linear
layer output $z$ is a target domain of our latent diffusion model.

Decoder design. We prepared the SMILES decoder to reconstruct the input SMILES
sequence from a given $z$. Following many major works that treated SMILES as a
variant of language data[44, 5, 41], we built an autoregressive transformer
[53] as our decoder (Figure 2-(b)) which is widely used to successfully
generate sequential data [4]. Specifically, starting from the [SOS] token, the
decoder predicts the next SMILES token using information from $z$ with cross-
attention layers. When $m=\\{t_{0},t_{1},\dots,t_{n}\\}$ is the token sequence
of the input SMILES, the decoder is trained to minimize the next-token
prediction loss described as Eq. (9):

$\displaystyle L_{dec}=-\sum^{n}_{i=1}\log p(t_{n}|t_{0:n-1})\;$ (9)

Here, the decoder and the compression layer are jointly trained while the
encoder’s parameter is frozen. After being fully trained, the decoder was able
to reconstruct roughly 98% of the SMILES encoder input.

###  3.2 Text-conditioned latent diffusion model

As shown in Figure 2-(c), our diffusion model learns the conditional
distribution of the SMILES latent $z$ whose dimension size is $\\{L\times
d_{z}\\}$. In the training phase, a molecule in the training data is mapped to
the latent $z$ and applied a forward noising process into $z_{t}$ with
randomly sampled diffusion timestep $t$ and injected noise $\epsilon$. A
diffusion model predicts the injected noise from $z_{t}$, conditioned by the
paired text description via a frozen external text encoder. In the inference
phase, the diffusion model iteratively generates a new latent sample
$z^{\prime}$ from a given text condition, which is then decoded to a molecule
via the SMILES decoder.

Since most contributions to diffusion models were made in the image domain,
most off-the-shell diffusion models have the architecture of convolution-based
Unet [40]. However, to step beyond the spatial inductive bias and make the
model applicable for more general data modality, diffusion models using
transformer architectures are proposed. One of the most successful approaches
among them is DiT [35], which is the model we’ve employed. Specifically, we
utilized a DiTbase model with two major modifications: first, DiT’s two-
dimensional attributes (_e.g._ , patch embedding) were modified accordingly to
handle one-dimensional latent. Second, cross-attention layers for external
conditions were added after the self-attention layers to receive high-
dimensional text conditions.

Text-based molecule generation requires a text encoder to process natural
language conditions. Existing text-based molecule generation models trained
their text encoder from scratch [36], or utilize a separate encoder model pre-
trained on scientific domain corpora [3]. In this work, we took the encoder
part of MolT5 [10] as our text encoder.

###  3.3 Implementation details

The pre-training of the SMILES encoder and the corresponding decoder was done
with 10,000,000 general molecules from PubChem [24]. To ensure enough batch
size for negative samples [15], we build a memory queue that stores $Q$ recent
input and use them for the encoder training. To be fed into the autoencoder,
an input SMILES must be tokenized as a sequence of tokens in a pre-defined
vocabulary. The tokenizer vocabulary consists of 300 tokens, which were
obtained from the pre-training data SMILES corpus using the BPE algorithm
[11]. We only used SMILES data that does not exceed a fixed maximum token
length $L$.

To train the text-conditioned latent diffusion model, we gathered three
existing datasets of text-molecule pair: PubchemSTM curated by [30], ChBEI-20
[9], and PCdes [55]. Only a train split for each dataset was used for the
training, and pairs that appear in the test set for the experiments in the
Results are additionally removed. We also used 100,000 molecules from PubChem
without any text descriptions, which helps the model learn the common
distribution of molecules. When these unlabeled data were fed into the
training model, we used a pre-defined null text which represents the absence
of text condition.

The latent diffusion model was trained with the training loss suggested in ADM
[8]. To take advantage of classifier-free guidance, we randomly replaced 3% of
the given text condition with a null text during the training. More detailed
training hyperparameters can be found in Appendix A. Unless specifically
mentioned, the sampling iteration in the inference stage used DDIM-based [45]
100 sampling steps with a classifier-free guidance scale of 5.0.

##  4 Experiments

###  4.1 Text-conditioned molecule generation

In this section, we evaluated the trained LDMol’s ability to generate
molecules that agree with the given natural language conditions. We designed
two scenarios: a benchmark study to see if the model can find the output that
satisfies very long and specific conditions, and zero-shot text-to-molecule
generation with unseen text conditions to see LDMol’s generalization ability
and the learned alignment between text and molecules.

Quantitative benchmark study. We generated molecules with LDMol using the text
captions in the ChEBI-20 test set and compared them with the ground truth.
Here, we assume that the given description is long and detailed so that the
ideal generative model can figure out the ground truth. On top of the training
with molecule-text pairs from various sources, we additionally fine-tuned the
model with the ChEBI-20 training set and 10,000 unlabeled molecules. The
metrics we’ve used are as follows: SMILES validity, BLEU score [34] and
Levenshtein distance between two SMILES, Tanimoto similarity [2] between two
SMILES with three different fingerprints (MACCS, RDK, Morgan), the exact match
ratio, and Frechet ChemNet Distance (FCD) [37].

We note that since our diffusion model sampling starts from randomly sampled
noise from a prior distribution, the output from deterministic DDIM sampling
still has stochasticity. However, the performance difference was marginal
between starting from random noise and starting from a fixed point (_e.g._ ,
zero vector). Since the given condition is more specific and detailed than
zero-shot case studies, we used the lower classifier-free guidance scale of
3.5.

Table 1: Benchmark results of text-to-molecule generation on ChEBI-20 test
set. The best performance for each metric was written in bold, and the second-
best performance was underlined. The results for baseline models are taken
from [13].

Model | Validity$\uparrow$ | BLEU$\uparrow$ | Levenshtein$\downarrow$ | MACCS FTS$\uparrow$ | RDK FTS$\uparrow$ | Morgan FTS$\uparrow$ | Match$\uparrow$ | FCD$\downarrow$
---|---|---|---|---|---|---|---|---
Transformer | 0.906 | 0.499 | 57.660 | 0.480 | 0.320 | 0.217 | 0.000 | 11.32
SciBERT [3] | 0.915 | 0.459 | 24.458 | 0.721 | 0.588 | 0.529 | 0.005 | -
T5base [38] | 0.660 | 0.765 | 24.950 | 0.731 | 0.605 | 0.545 | 0.069 | 2.48
MolT5base [10] | 0.772 | 0.769 | 24.458 | 0.721 | 0.588 | 0.529 | 0.081 | 2.18
GIT-Mol [29] | 0.928 | 0.756 | 26.315 | 0.738 | 0.582 | 0.519 | 0.051 | -
TGM-DLM [13] | 0.871 | 0.826 | 17.003 | 0.854 | 0.739 | 0.688 | 0.242 | 0.77
LDMol | 0.928 | 0.900 | 9.259 | 0.931 | 0.860 | 0.820 | 0.363 | 0.25

Table 1 contains the performance of LDMol and other baselines for text-to-
molecule generation on the ChEBI-20 test set. Including both transformer-based
autoregressive models and diffusion-based models, LDMol outperformed the
existing models in almost every metric. While one model showed similar
validity to ours, it had a poor agreement with the ground truth compared to
LDMol, with an exact match of 0.051. One thing to emphasize is that MolT5 uses
the same text encoder with LDMol, yet there’s a significant performance
difference between the two models. We believe this is because our constructed
latent space is much easier to align with the same textual information,
compared to the token sequence used by transformer-based models.

Figure 3: Examples of the generated molecules by LDMol with various text
conditions. Validity on 1,000 generated samples is included in the lower left
corner of each box.

Zero-shot case studies. To demonstrate the generalization ability of LDMol, we
analyzed the generated output with several hand-written text conditions. These
input prompts were not contained in the training data and were relatively
vague and high-level so that many different molecules could satisfy the
condition.

Figure 3 shows the samples of generated molecules from LDMol with several
input prompt examples. We found that LMDol can generate molecules with high
validity that follow the various levels of input conditions for specific
atoms(a), functional groups(b), molecular substructure(c), compound class(d),
and substance names(e). The validity was calculated by the number of valid
SMILES over the 1,000 generated samples, which was above 0.9 for all five
scenarios we’ve shown. Considering that these short, broad, and hand-written
text conditions are distinct from the text conditions in the training dataset,
we’ve concluded that our model is able to learn the general relation between
natural language and molecules.

###  4.2 Applications toward downstream tasks

Well-trained diffusion models learned the score function of a data
distribution, which enables high applicability to various downstream tasks.
The state-of-the-art image diffusion models have shown their versatility in
image editing [32, 16], classification [25], retrieval [21], inverse problems
like inpainting and deblurring [7], image personalization [42], etc. To
demonstrate LMDol’s potential versatility as a diffusion model, we applied the
pre-trained LDMol to the molecule-to-text retrieval and text-driven molecule
editing task.

Molecule-to-text retrieval. Our approach to molecule-to-text retrieval is
similar to the idea of using a pre-trained diffusion model as a classifier
[25]; LDMol takes each candidate text with a query molecule’s noised latent,
and retrieves the text that minimizes the difference between output noise
$\hat{\epsilon_{\theta}}$ and the injected noise $\epsilon$ (see Appendix B
for more details). Since this process has randomness due to the stochasticity
of $t$ and $\epsilon$, we repeated the same process $n$ times with resampled
$t$ and $\epsilon$ and used a mean loss to minimize the performance variance.

Table 2: 64-way accuracy of molecule-to-text retrieval task. The custom test
set and the baseline model results are taken from [49]. For LDMol, $n$ is a
number of iterations where $||\hat{\epsilon_{\theta}}-\epsilon||^{2}_{2}$ was
calculated. The best performance for each task is written in bold.

Model | PCdes test set | custom test set from Su _et al._
---|---|---
| sentence-level | paragraph-level | sentence-level | paragraph-level
SciBERT [3] | 50.38 | 82.57 | 1.38 | 1.38
KV-PLM [55] | 55.92 | 77.92 | 1.37 | 1.51
MoMu-S [49] | 58.64 | 80.62 | 39.50 | 45.66
MoMu-K [49] | 58.74 | 81.09 | 39.07 | 46.21
LDMol($n$=10) | 52.28 | 89.36 | 57.90 | 78.16
LDMol($n$=25) | 55.94 | 90.82 | 63.27 | 81.13

We measured a 64-way retrieval accuracy of LDMol using two different test
sets: PCdes test split and another text-molecule retrieval dataset separately
curated by [49]. Table 2 contains the text-to-molecule accuracy of LDMol with
other baseline models. Only one randomly selected sentence in each candidate
description was used for the retrieval in the “sentence-level" column, and the
entire descriptions were used for the “paragraph-level" column. LDMol achieved
a higher performance in three out of four scenarios compared to the previously
presented models and became more accurate as the number of function
evaluations increased. It maintained its performance on a relatively out-of-
distribution test set with minimal accuracy drop. The actual examples from the
retrieval result can be found in Appendix C.

Text-driven molecule editing. We applied a method of Delta Denoising Score
(DDS) [16], which was originally suggested for text-guided image editing, to
see whether LDMol can be used to optimize a source molecule to match a given
text. Using two texts that describe the source data $z_{src}$ and the desired
target, DDS presents how a text-conditioned diffusion model can modify
$z_{src}$ to a new data $z_{tgt}$ that follows the target text description
(more detailed procedures can be found in Appendix B).

Figure 4: Examples of text-guided image editing with LDMol. The difference
between the source text and the target text, and the corresponding region is
colored in purple.

We imported a method of DDS on LDMol’s molecule latent to edit a given
molecule to match the target text, where the case studies are shown in Figure
4., The editing results successfully modified the input molecule towards the
target text with minimal corruption of the unrelated region. We emphasize that
this kind of direct data optimization is possible because of the continuous
and non-restricted latent domain from the LDMol’s encoder.

###  4.3 Ablation studies

We’ve conducted ablation studies for each design choice of the proposed LDMol
to analyze and emphasize their role. We measured the SMILES validity and the
exact match ratio of text-to-molecule generation on the ChEBI-20 test set,
with three models pre-trained with the same number of DiT training iterations.

Table 3: Quantitative results of the ablation study.

models | SMILES Validity$\uparrow$ | Match$\uparrow$
---|---|---
LDMol | 0.922 | 0.303
w/o contrastive encoder pre-training | 0.000 | -
w/o compression layer | 0.110 | 0.003

The result of the ablation study is listed in Table 3. When the molecule
latent space was constructed by a naive autoencoder (_i.e._ the SMILES encoder
is not pre-trained with the proposed method), the diffusion model completely
failed to learn the data distribution to generate valid SMILES. A similar
problem occurred when we didn’t introduce a compression layer to reduce the
dimension of the latent space, meaning that the diffusion model could not
properly model the data distribution in the given latent space.

##  5 Conclusion

In this work, we presented a text-conditioned molecule diffusion model LDMol
that runs on a chemical latent space reflecting structural information. By
introducing the deeply studied paradigm of the latent diffusion model in
general domains into the molecule domain with minimal modifications, LDMol
retains many advanced attributes of diffusion models that enable various
applications.

Despite the noticeable performances of LDMol, it still has limitations that
can be improved; Since the text captions of our obtained paired data mostly
describe structural features, LDMol still struggles to follow some text
conditions such as biological properties. The current suggested text-guided
molecule editing performance is relatively sensitive to hyperparameters as the
imported method assumes broader and more substantial knowledge that emerged
from the data generation. Nonetheless, we expect that the LDMol’s performance
could be improved further with the emergence of richer text-molecule pair data
and more powerful text encoders, with possible future works for the
construction of a better latent space.

We believe that our approach could inspire tackling various chemical
generation tasks using latent space, not only text-conditioned but also many
more desired properties such as biochemical activity. Especially, we expect
LDMol could be a starting point to bridge achievements in the state-of-the-art
diffusion model into the chemical domain.

## References

  * Bagal et al. [2021] Viraj Bagal, Rishal Aggarwal, PK Vinod, and U Deva Priyakumar.  Molgpt: molecular generation using a transformer-decoder model.  _Journal of Chemical Information and Modeling_ , 62(9):2064–2076, 2021.
  * Bajusz et al. [2015] Dávid Bajusz, Anita Rácz, and Károly Héberger.  Why is tanimoto index an appropriate choice for fingerprint-based similarity calculations?  _Journal of cheminformatics_ , 7:1–13, 2015.
  * Beltagy et al. [2019] Iz Beltagy, Kyle Lo, and Arman Cohan.  Scibert: A pretrained language model for scientific text.  _Empirical Methods in Natural Language Processing_ , 2019.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _NIPS_ , 33:1877–1901, 2020.
  * Chang and Ye [2024] Jinho Chang and Jong Chul Ye.  Bidirectional generation of structure and properties through a single molecular foundation model.  _Nature Communications_ , 15(1):2323, 2024.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _ICML_ , pages 1597–1607. PMLR, 2020.
  * Chung et al. [2023] Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Marc L Klasky, and Jong Chul Ye.  Diffusion posterior sampling for general noisy inverse problems.  _ICLR_ , 2023.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _NIPS_ , 34:8780–8794, 2021.
  * Edwards et al. [2021] Carl Edwards, ChengXiang Zhai, and Heng Ji.  Text2mol: Cross-modal molecule retrieval with natural language queries.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 595–607, 2021.
  * Edwards et al. [2022] Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji.  Translation between molecules and natural language.  _Empirical Methods in Natural Language Processing_ , 2022.
  * Gage [1994] Philip Gage.  A new algorithm for data compression.  _C Users Journal_ , 12(2):23–38, 1994.
  * Gómez-Bombarelli et al. [2018] Rafael Gómez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik.  Automatic chemical design using a data-driven continuous representation of molecules.  _ACS central science_ , 4(2):268–276, 2018.
  * Gong et al. [2024] Haisong Gong, Qiang Liu, Shu Wu, and Liang Wang.  Text-guided molecule generation with diffusion language model.  _AAAI_ , 2024.
  * Gómez-Bombarelli et al. [2018] Rafael Gómez-Bombarelli, Jennifer N. Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D. Hirzel, Ryan P. Adams, and Alán Aspuru-Guzik.  Automatic chemical design using a data-driven continuous representation of molecules.  _ACS Central Science_ , 4:268–276, 01 2018.  doi: 10.1021/acscentsci.7b00572.
  * He et al. [2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _CVPR_ , pages 9729–9738, 2020.
  * Hertz et al. [2023] Amir Hertz, Kfir Aberman, and Daniel Cohen-Or.  Delta denoising score.  _ICCV_ , pages 2328–2337, 2023.
  * Higgins et al. [2017] Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner.  beta-vae: Learning basic visual concepts with a constrained variational framework.  _ICLR_ , 3, 2017.
  * Ho and Salimans [2021] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _NIPS_ , 2021.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _NIPS_ , 12 2020.
  * Hoogeboom et al. [2022] Emiel Hoogeboom, Vıctor Garcia Satorras, Clément Vignac, and Max Welling.  Equivariant diffusion for molecule generation in 3d.  In _ICML_ , pages 8867–8887. PMLR, 2022.
  * Jin et al. [2023] Peng Jin, Hao Li, Zesen Cheng, Kehan Li, Xiangyang Ji, Chang Liu, Li Yuan, and Jie Chen.  Diffusionret: Generative text-video retrieval with diffusion model.  In _ICCV_ , pages 2470–2481, 2023.
  * Karras et al. [2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  _NIPS_ , 35:26565–26577, 2022.
  * Kim and Ye [2021] Kwanyoung Kim and Jong Chul Ye.  Noise2score: tweedie’s approach to self-supervised image denoising without clean images.  _NIPS_ , 34:864–874, 2021.
  * Kim et al. [2023] Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin A Shoemaker, Paul A Thiessen, Bo Yu, et al.  Pubchem 2023 update.  _Nucleic acids research_ , 51(D1):D1373–D1380, 2023.
  * Li et al. [2023] Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak.  Your diffusion model is secretly a zero-shot classifier.  In _ICCV_ , pages 2206–2217, 2023.
  * Li et al. [2018] Yibo Li, Liangren Zhang, and Zhenming Liu.  Multi-objective de novo drug design with conditional graph generative model.  _Journal of cheminformatics_ , 10:1–24, 2018.
  * Lim et al. [2018] Jaechang Lim, Seongok Ryu, Jin Woo Kim, and Woo Youn Kim.  Molecular generative model based on conditional variational autoencoder for de novo molecular design.  _Journal of cheminformatics_ , 10(1):1–9, 2018.
  * Lim et al. [2020] Jaechang Lim, Sang-Yeon Hwang, Seokhyun Moon, Seungsu Kim, and Woo Youn Kim.  Scaffold-based molecular design with a graph generative model.  _Chem. Sci._ , 11:1153–1164, 2020.  doi: 10.1039/C9SC04503A.
  * Liu et al. [2024] Pengfei Liu, Yiming Ren, Jun Tao, and Zhixiang Ren.  Git-mol: A multi-modal large language model for molecular science with graph, image, and text.  _Computers in Biology and Medicine_ , 171:108073, 2024.
  * Liu et al. [2023] Shengchao Liu, Weili Nie, Chengpeng Wang, Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang, Chaowei Xiao, and Animashree Anandkumar.  Multi-modal molecule structure–text model for text-based retrieval and editing.  _Nature Machine Intelligence_ , 5(12):1447–1457, 2023.
  * Luo et al. [2023] Tianze Luo, Zhanfeng Mo, and Sinno Jialin Pan.  Fast graph generation via spectral diffusion.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * Meng et al. [2022] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  _ICLR_ , 2022.
  * Olivecrona et al. [2017] Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen.  Molecular de-novo design through deep reinforcement learning.  _Journal of Cheminformatics_ , 9, 09 2017.  doi: 10.1186/s13321-017-0235-x.
  * Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318, 2002.
  * Peebles and Xie [2023] William Peebles and Saining Xie.  Scalable diffusion models with transformers.  In _ICCV_ , pages 4195–4205, 2023.
  * Pei et al. [2023] Qizhi Pei, Wei Zhang, Jinhua Zhu, Kehan Wu, Kaiyuan Gao, Lijun Wu, Yingce Xia, and Rui Yan.  Biot5: Enriching cross-modal integration in biology with chemical knowledge and natural language associations.  _Empirical Methods in Natural Language Processing_ , 2023.
  * Preuer et al. [2018] Kristina Preuer, Philipp Renz, Thomas Unterthiner, Sepp Hochreiter, and Gunter Klambauer.  Fréchet chemnet distance: a metric for generative models for molecules in drug discovery.  _Journal of chemical information and modeling_ , 58(9):1736–1741, 2018.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of machine learning research_ , 21(140):1–67, 2020.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , pages 10684–10695, 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241, 2015.
  * Ross et al. [2022] Jerret Ross, Brian Belgodere, Vijil Chenthamarakshan, Inkit Padhi, Youssef Mroueh, and Payel Das.  Large-scale chemical language representations capture molecular structure and properties.  _Nature Machine Intelligence_ , 4(12):1256–1264, 2022.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _CVPR_ , pages 22500–22510, 2023.
  * Saharia et al. [2022] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  _IEEE transactions on pattern analysis and machine intelligence_ , 45(4):4713–4726, 2022.
  * Segler et al. [2017] Marwin H. S. Segler, Thierry Kogej, Christian Tyrchan, and Mark P. Waller.  Generating focused molecule libraries for drug discovery with recurrent neural networks.  _ACS Central Science_ , 4:120–131, 12 2017.  doi: 10.1021/acscentsci.7b00512.
  * Song et al. [2021a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _ICLR_ , 2021a.
  * Song and Ermon [2019] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  _NIPS_ , 32, 2019.
  * Song et al. [2021b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _ICLR_ , 2021b.
  * Staszak et al. [2022] Maciej Staszak, Katarzyna Staszak, Karolina Wieszczycka, Anna Bajek, Krzysztof Roszkowski, and Bartosz Tylkowski.  Machine learning in drug design: Use of artificial intelligence to explore the chemical structure–biological activity relationship.  _Wiley Interdisciplinary Reviews: Computational Molecular Science_ , 12(2):e1568, 2022.
  * Su et al. [2022] Bing Su, Dazhao Du, Zhao Yang, Yujie Zhou, Jiangmeng Li, Anyi Rao, Hao Sun, Zhiwu Lu, and Ji-Rong Wen.  A molecular multimodal foundation model associating molecule graphs with natural language.  _arXiv preprint arXiv:2209.05481_ , 2022.
  * Trippe et al. [2023] Brian L Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi Jaakkola.  Diffusion probabilistic modeling of protein backbones in 3d for the motif-scaffolding problem.  _ICLR_ , 2023.
  * Vahdat et al. [2021] Arash Vahdat, Karsten Kreis, and Jan Kautz.  Score-based generative modeling in latent space.  _NIPS_ , 34:11287–11302, 2021.
  * Van Den Oord et al. [2017] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  _NIPS_ , 30, 2017.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2019] Sheng Wang, Yuzhi Guo, Yuhong Wang, Hongmao Sun, and Junzhou Huang.  Smiles-bert: large scale unsupervised pre-training for molecular property prediction.  In _Proceedings of the 10th ACM international conference on bioinformatics, computational biology and health informatics_ , pages 429–436, 2019.
  * Zeng et al. [2022] Zheni Zeng, Yuan Yao, Zhiyuan Liu, and Maosong Sun.  A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals.  _Nature communications_ , 13(1):862, 2022.

##  Appendix A Model hyperparameters and training setup

Table 4: Our choice of the model hyperparameters and training setup.
hyperparameters
---
$L$ | 128
$d_{enc}$ | 1024
$d_{z}$ | 64
$\tau$ | 0.07
$Q$ | 16384
training setup
optimizer | autoencoder: AdamW, DiT: Adam
learning rate | autoencoder: cosine annealing(1e-4$\rightarrow$1e-5), DiT: 1e-4
batch size per gpu | encoder: 64, decoder: 128, DiT: 16
training resources |  encoder: NVIDIA A100(VRAM:40GB), 15 hours decoder: NVIDIA A100(VRAM:40GB), 15 hours DiT: 8 NVIDIA A100(VRAM:40GB), 96 hours

##  Appendix B More detailed overview for LDMol’s application for downstream
tasks

Figure 5: Overall pipeline for the applications of LDMol. (a) Molecule-to-text
retrieval. (b) Text-guided molecule editing. The SMILES autoencoder and the
text encoder are not drawn in this figure.

Figure 5-(a) shows the LDMol’s molecule-to-text retrieval process. A given
query molecule is converted to a latent $z$ then a forward noise process is
applied with a randomly sampled timestep $t$ and noise $\epsilon$. This
$z_{t}$ is fed to LDMol with each candidate text, and the candidate that
minimizes the loss $||\hat{\epsilon_{\theta}}-\epsilon||^{2}_{2}$ between
$\epsilon$ and the output noise $\hat{\epsilon_{\theta}}$ is retrieved. To
minimize the variance from the stochasticity of $t$ and $\epsilon$, the same
process can be repeated multiple times with resampled $t$ and $\epsilon$ to
use a mean loss.

Figure 5-(b) illustrates the DDS-based molecule editing with LDMol.
Specifically, DDS requires source data $z_{src}$, target data $z_{tgt}$ which
is initialized to $z_{src}$, and their corresponding source and target text
descriptions. We apply the forward noise process to $z_{src}$ and $z_{tgt}$
using the same randomly sampled $t$ and $\epsilon$ to get $z_{t,src}$ and
$z_{t,tgt}$. These are fed into the pre-trained LDMol with their corresponding
text, where we denote the output noise as $\hat{\epsilon}_{\theta,src}$ and
$\hat{\epsilon}_{\theta,tgt}$. Finally, $z_{tgt}$ is modified towards the
target text by optimizing it to the direction of
$(\hat{\epsilon}_{\theta,tgt}-\hat{\epsilon}_{\theta,src})$ with a learning
rate $\gamma$. Here, $\hat{\epsilon}_{\theta,tgt}$ and
$\hat{\epsilon}_{\theta,tgt}$ can be replaced with the classifier-free-
guidanced noises, using the output with the null text and the guidance scale
$\omega$. We repeated this optimization loop 200 times with $\gamma=0.4$ and
$\omega=2.0$.

##  Appendix C Molecule-to-text retrieval examples

Figure 6: The examples of molecule-to-text retrieval result on the PCdes test
set. The retrieval was done at the sentence level, and the top three retrieval
outputs for each query molecule are described. The corresponding description
from the data pair was correctly retrieved at first for both cases, and the
other retrieved candidates weakly show a correlation with the query molecule.


# AI-based Density Recognition

Simone Müller Leibniz Supercomputing Centre (LRZ<EMAIL_ADDRESS>Daniel
Kolb Leibniz Supercomputing Centre (LRZ<EMAIL_ADDRESS>

Matthias Müller German Aerospace Center (DLR<EMAIL_ADDRESS>Dieter
Kranzlmüller Ludwig-Maximilians-Universität (LMU<EMAIL_ADDRESS>
###### Abstract

Learning-based analysis of images is commonly used in the fields of mobility
and robotics for safe environmental motion and interaction. This requires not
only object recognition but also the assignment of certain properties to them.
With the help of this information, causally related actions can be adapted to
different circumstances. Such logical interactions can be optimized by
recognizing object-assigned properties. Density as a physical property offers
the possibility to recognize how heavy an object is, which material it is made
of, which forces are at work, and consequently which influence it has on its
environment. Our approach introduces an AI-based concept for assigning
physical properties to objects through the use of associated images. Based on
synthesized data, we derive specific patterns from 2D images using a neural
network to extract further information such as volume, material, or density.
Accordingly, we discuss the possibilities of property-based feature extraction
to improve causally related logics.

### Keywords

AI, Density Recognition, Computer Vision

##  1 Introduction

[b]  Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are not
made or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee.

Modern machines and robots use various sensors to capture and navigate their
surroundings. Particularly in road traffic, situations may appear
inconspicuous at first sight but require constant attention and quick
reactions. This can involve evaluating the potential risks in autonomous
driving scenarios when a car not only recognizes objects but can also estimate
the potential damage in the event of a collision and adapt its driving
behavior accordingly. Additional information could increase a system’s scope
of action and decision-making as well as the automatic assessment of real-life
scenes.

Whether it is a ball that rolls onto the road, a car that suddenly brakes, or
an item that falls off a moving vehicle. Such reactions are often based on
causal relationships that are logical for us humans but not for machines.
Despite their logic, machines lack the necessary background knowledge and
specific skills, such as the assessment of physical properties, to gain a
causal understanding.

Material recognition and the association of related properties can be helpful
in causal decision-making [26]. For example, an industrial robot can apply the
optimum force for gripping an inelastic object if it knows the approximate
material, mass, roughness, and size of this object. All this information
relates to the physical density and material property.

Machine learning offers solutions for material recognition [26]. Databases
such as Flickr [14] are able to recognize different materials, as shown in
Tab. 1.

|  |  |  |
---|---|---|---|---
Fabric | Foliage | Glass | Leather | Metal
|  |  |  |
Plastic | Water | Paper | Wood | Stone
Table 1: State-of-the-Art Material Database [14]. Illustration of ten example
material categories of Flickr database. The lighting conditions, compositions,
colors, textures, surface shapes, material subtypes, and object associations
were considered by an image diversity of 100 pictures, 50 close-ups, and 50
normal views in each category [1].

These patterns are recognized by visual features and stored in the model
through a learning process, using sophisticated AI algorithms for the
recognition of objects in 2D images, such as YOLOv3 [21], Faster Region-Based
Convolutional Neural Networks (R-CNN) [22], and Multi-Scale Convolutional
Neural Network (MSCNN) [3]. However, the previously trained 2D object
recognition is limited to specific object classes and visual interference
effects of images [17]. To collect ambient information and make accurate
decisions with high confidence, the AI usually needs to be trained extensively
with vast sets of data. Additionally, this material information has usually no
connection with the associated physical properties.

Based on the challenges of accurate processing and linking causal information,
we present an approach that enables the assignment of physical properties in
objects based on a 2D image by using machine learning and pattern recognition.
The object is extracted and scaled into triangles to estimate the volume. We
derive the associated materials from a database and ultimately calculate the
density as a physical quality.

AI-based recognition of density and volume provides a solid foundation for the
extraction of additional information from the environment. As an example,
object-related forces can be calculated based on equation-specific
coefficients, constants, and acquired sensor data including density. This
expands the information content in a visual scene. Especially in road traffic,
this can be an additional aid to improve the perception of autonomous
vehicles.

This paper describes a proof of concept for the implementation of AI-based
density recognition. Our work comprises the following contributions:

  * •

Neural-specific object and texture detection based on object classification

  * •

Concept of AI-based density recognition

  * •

Analysis of recognized object density and material composition

Our evaluation reveals the feasibility and transferability of AI-based density
recognition. For our empirical examination, we use synthetically generated
data from the Unreal Engine.

The paper is organized according to a fixed structure consisting of related
work, concept, methodology of AI-based object and texture detection,
evaluation, conclusion, as well as future work.

##  2 Modern recognition

This section presents recognition models and existing approaches for physical
property recognition. The basic idea involves material recognition, which
gives rise to entire databases such as Flickr [14] which assigns materials
based on visual appearance.

Liu et al. [14] describe that the visual appearance of a surface depends on
illumination conditions, geometric structure of surfaces at different spatial
scales, and reflectance properties. Thereby, the reflectance properties of the
surface are often characterized by features with a bidirectional reflectance
distribution function (BRDF) [18]. In this context, material recognition can
employ the recognition of colors and textures, micro-textures, outline shapes,
or reflectance-based features [14] by SIFT algorithms. This algorithm can
recognize contours based on corners and edges [13].

Standard k-means algorithms Eq. 1 are used to cluster instances of each
feature [14] in order to assign the image-specific materials $M$ to the
respective words.

$J=\sum^{k}_{\mathrm{i=1}}\sum_{\mathrm{x_{\mathrm{j}}\in
S_{\mathrm{i}}}}||x_{\mathrm{j}}-\mu_{\mathrm{i}}||^{2}$ (1)

$S_{\mathrm{i}}$ describes the cluster in Eq. 1, which is determined from data
points $x_{\mathrm{j}}$ and centroids $\mu_{\mathrm{i}}$ on the basis variance
minimization [6] and squared Euclidean distance
$||x_{\mathrm{j}}-\mu_{\mathrm{i}}||^{2}$. The random mean value $k$ is
determined in the visual data set $m_{\mathrm{i}}$, …, $m_{\mathrm{k}}$. Each
data object is assigned to the cluster with the lowest variance for all
$l=[1,...,k]$, shown in Eq. 2.

$S_{\mathrm{i}}=\\{x_{\mathrm{j}}:||x_{\mathrm{j}}-m_{\mathrm{i}}||^{2}\leq||x_{\mathrm{j}}-m_{\mathrm{l}}||^{2}\\}$
(2)

Machine learning can be used as a static method to learn continuously and
specifically from experiences. Thereby, the training-based data is divided
into different classes. The classification $S_{\mathrm{i}}$ permits the
mapping of input variables
$f_{\mathrm{i,x}}:\mathbb{R}^{n_{\mathrm{i}}}\rightarrow\mathbb{R}^{n_{\mathrm{i-1}}}$
to discrete output variables $O$. In this view, the regression
$g_{\mathrm{(x)}}$ refers to a distinction between more than two categories
[15]. Fig. 1 summarises the related components of neural networks.

Figure 1: Schematic Neural Network, adapted from [29]. The input signals
$f(i,x):$[$x_{\mathrm{1}}$, .. , $x_{\mathrm{n}}$] are weighted using the
weighting factor $w_{\mathrm{i}}$ before they reach the main part of the
neuron. A bias $b_{\mathrm{i}}$ is included as a threshold value which must
first be exceeded to generate the output signal $O$.

Maschine learning techniques are broadly divided into two processes: feature
extraction and classification. Feature extraction involves the identification
and correlation of patterns with large datasets suitable for modeling.
Thereby, a feature refers to a property derived from raw data input intending
to provide a suitable representation [10]. In both cases, numerous processing
nodes of neural networks are tightly interconnected and further layered in
organized nodes to perform complex calculations [24].

The overall quality of learning-based feature extraction depends on the task
area and associated data set. In this respect, the data must contain a high
density of information. To differentiate between classes, the algorithms try
to recognize existing patterns of colors, shapes, textures or pixel values
within this information [9].

Girshick et al. [8] describe an approach in which high-capacity convolutional
neural networks are applied to bottom-up region proposals for localization and
segmentation. They introduce a paradigm for training large CNNs when labeled
training data is scarce. They detail how pre-training the network with
supervision on an auxiliary task with abundant data (image classification) and
then fine-tuning the network on the target task where data is scarce
(recognition) can improve overall efficiency. This approach is similar to
R-CNN. In fast R-CNN, the CNN is first fed with the input image to generate a
convolutional feature map. Subsequently, the selective search is performed and
the region suggestions are warped into squares. Those region suggestions are
called Regions of Interest (RoI) and refer to a subset of the original image
[5]. By using RoI pooling layers, each region that has been proposed and which
may have different sizes, is reshaped into fixed size so that it can be fed
into a fully connected layer. On the output, a softmax layer is used to
predict the class of the proposed region and the values of the bounding box
[19]. This method produces results faster since it calculates the CNN features
only once per image and not two thousand times as with the R-CNN method.

Sean Bell et al. [2] suggest in a direct comparison between three different
CNN models (AlexNet, VGG-16, GoogleNet) that material recognition and
segmentation of everyday images which are based on Materials in Context
Database (MINC) is possible with a probability of 82.2 $\%$ (AlexNet) to 85.9
$\%$ (GoogleNet).

Shukla et al [25] evaluated the accuracy between CNN classifiers for material
recognition and deep learning classifiers. They found that CNN classifiers
have better and faster recognition accuracy since the existing probability
level allows the classifier to recognize materials with higher accuracy.

Various previous works [3, 16, 21, 22] refer to AI algorithms such as YOLOv3,
Faster R-CNN and MSCNN that use a class-verifying diversity of object
propositions for object recognition.

Modern Architectures such as YOLO have been continuously improved to perform
tasks in the areas of general and oriented recognition, instance segmentation,
pose, key points and classification [12]. A DarkNet-19 model architecture
(YOLOv2) was expanded to a more complex backbone model DarkNet-53 (YOLOv3) in
which features on three different scales can be recognized [4]. Although the
implementation of such new functions and targeted optimizations reduce latency
times, they often require computationally intensive operations that demand
considerable computing power.

Ren et al. [22] found a solution to an issue of R-CNN caused by selective
search. Selective search is a rigid algorithm that is unable to improve or
learn, which can lead to poor suggestions for candidate regions. They
developed the Faster-R-CNN algorithm, replacing selective search with a
separate network, the Region Proposal Network (RPN), to predict region
proposals. The RPN takes an image as input and outputs a series of rectangular
object proposals, each with a class prediction and a confidence value. The
network can be trained throughout by backpropagation, where the gradient of
the loss function is calculated taking into account the weights of the network
for a single input-output example [23].

Wu et al. [28] teach a computational vision system to understand physical
relationships with the help of unlabelled videos. They address specific
physical scenarios and distinguish between two groups of physical properties:
The first inherits the intrinsic physical properties of objects such as
volume, material, and mass. The other group is the descriptive physical
properties, which describe the scene and are determined by the first group.
These include, but are not limited to, the speed of the objects, the distance
they travel, or whether they fall into water. The presented model uses CNN to
learn the object properties exclusively from unlabelled data. This approach
provides serviceable results on a physical data set.

##  3 Density Recognition

This chapter describes the concept of density and material recognition in
order to calculate physical properties like masses. Fig. 2 illustrates the
fundamental pipeline of AI-based density recognition.

ImageObject DetectionMesh AssignmentTexture Detection$\sum M$$\sum
V$$V_{\mathrm{n}}$$V_{\mathrm{1}}$$b_{\mathrm{1}}$$b_{\mathrm{n}}$$M_{\mathrm{1}}$$M_{\mathrm{n}}$$\sum
m$$\sum\varrho$Density AssignmentMass Figure 2: Pipeline of AI-based density
recognition. The pipeline includes the detection of objects and their textures
as well as the assignment of density and meshes to calculate physical
quantities like object masses.

Building on identifying specific features from 2D images of previous work, we
combine object detection with the assignment and calculation of physical
properties. The image data is first analyzed by using a neural network.
Salient objects can be identified and classified texturally. Object areas are
identified as $b_{\mathrm{1}},...,b_{\mathrm{n}}$. Within the bounding boxes,
possible materials can be attributed based on Flickr Material Database (FMD)
and Materials in Context Database (MINC). In this process, these materials $M$
will be assigned a specific literary density $\rho$ where we consider further
pattern properties like image colors, shapes, textures and pixels. Since it is
necessary to determine the volume $V$ of the respective objects, the process
calculates their corresponding meshes. The physical properties like mass $M$
can be derived from the extensive information.

###  3.1 Object Detection

In order to detect objects, we use the convolutional network of YOLOv4 [12].
We extract features from the input images and calculate them into feature
maps. As part of the YOLO architecture, we use backbone, neck, and head
detectors, as shown in Fig. 3.

Figure 3: YOLO Detectors used in this Work. The backbone extracts important
features from the image at different scales. The neck concatenates the
semantic information from different layers of the backbone network and
transmits it as input to the head. The head applies the refined features for
predictive object recognition.

YOLOv4 contains a pre-trained convolutional neural network such as VGG16 or
CSPDarkNet53 as a backbone which is based on SPP-Modul (Spatial Pyramid
Pooling) and PAN (Path Aggregation Network). As part of the prediction, the
head processes aggregated features and predicts the bounding boxes,
objecthood, and classification values.

Our model is trained on a MS COCO dataset, which contains over 80 different
classes and 1.5 million object instances in 200 thousand labeled images.

$c_{\mathrm{x}}$$c_{\mathrm{y}}$$\sigma_{\mathrm{t_{\mathrm{y}}}}$$\sigma_{\mathrm{t_{\mathrm{x}}}}$$p_{\mathrm{w}}$$p_{\mathrm{h}}$$b_{\mathrm{w}}$$b_{\mathrm{h}}$$b_{\mathrm{x}}=\sigma_{\mathrm{t_{\mathrm{x}}}}+c_{\mathrm{x}}$$b_{\mathrm{y}}=\sigma_{\mathrm{t_{\mathrm{y}}}}+c_{\mathrm{y}}$$b_{\mathrm{w}}=p_{\mathrm{w}}\cdot
e^{t_{\mathrm{w}}}$$b_{\mathrm{h}}=p_{\mathrm{h}}\cdot e^{t_{\mathrm{h}}}$
Figure 4: Bounding Boxes with Dimension Priors and Location Prediction,
adapted from [20]. The center coordinates of the box can be calculated with
the predicted values $t_{\mathrm{x}},t_{\mathrm{y}}$ using a sigmoid function
and offset by the location of grid cell $c_{\mathrm{x}},c_{\mathrm{y}}$. The
width and height of the final box are adjusted to the previous width
$p_{\mathrm{w}}$ and height $p_{\mathrm{h}}$ and scaled by
$e^{t_{\mathrm{w}}}$ and $e^{t_{\mathrm{h}}}$.

Recognition first divides the image into a grid of cells like shwon in the
example in Fig. 4. The number of cells depends on the size of the image. For
example, With a size of $608\times 608$ pixels, the cell size is usually
$32\times 32$ pixels. Our data set is divided into $19\times 19$ cells.

Each object is assigned to exactly one cell, containing the object’s center
point. Objectless cells are filtered out according to their low probability of
all 80 classes. The use of Non-Max-Suppression, as shown in Fig. 5, eliminates
unwanted bounding boxes so that only the most probable bounding box remains
for each detected object.

Non-Max SuppressionBeforeAfter Figure 5: Effect of Non-Max Suppression (NMS).
The post-processing technique Non-maximum suppression reduces the number of
overlapping bounding boxes.

The bounding boxes localize the position of objects in order to recognize
possible textures and calculate the object-specific volumes. The area is
adjusted to $224\times 224$ pixels for the matching designation of the model
input.

###  3.2 Texture Detection

The object-specific bounding box detection enables the continuous application
of a material detection model to the image area of the box. Inside the box,
features such as color, SIFT, jet, micro-SIFT, micro-jet, curvature, edge-
slice and edge-ribbon are combined and quantized into visual words by Bayesian
framework [14].

Three different models are used, consisting of the MINC dataset. MINC
comprises 23 different classes, each with 2500 images. We measure the
confidence score for all three models across MINC dataset and Flickr Material
Database (FMD) for appropriate model selection (see also Tab. 2).

Classes | VGG16 | GoogleNet | AlexNet
---|---|---|---
Fabric | (78 | 78) | (69 | 78) | (45 | 64)
Foliage | (71 | 95) | (68 | 95) | (62 | 93)
Glass | (40 | 82) | (40 | 84) | (27 | 78)
Leather | (29 | 88) | (18 | 84) | (9 | 80)
Metal | (44 | 72) | (37 | 76) | (33 | 69)
Paper | (41 | 90) | (35 | 90) | (11 | 85)
Plastic | (78 | 75) | (84 | 78) | (74 | 68)
Stone | (78 | 89) | (62 | 87) | (52 | 85)
Water | (47 | 96) | (43 | 94) | (30 | 93)
Wood | (36 | 74) | (25 | 78) | (13 | 71)
Table 2: Recognition on (FMD | MINC). VGG16 slightly outperforms GoogleNet. AlexNet provides the lowest accuracy out of these three contrasted models.

FMD consists of ten classes with 100 images of each. Tab. 2 shows the
performance of the models for the FMD dataset. VGG16 provides optimum results
with an overall accuracy of 86 % at MINC and 52 % at FMD. The model’s size,
however, requires significant time for both training and recognition.
Alternatively, GoogleNet offers a good replacement model with an accuracy of
86 % for MINC and 47 % for FMD. The use of so-called inception modules allows
a shorter calculation time. This module replaces a sparse CNN with a normal
dense construction since most activations in a deep network are zero values or
redundant due to correlations. As a result, not all output channels are
connected to the input channels, hence the reduced computing time [27].

###  3.3 Mesh and Density Assignment

Through the accompanying object recognition, the next step is to assign a
suitable 3D mesh. The mesh must have certain properties for the correct
calculation of volume. For example, each triangle of the mesh must have corner
points stored in a clockwise direction. The mesh must be complete without open
areas and completely closed to prevent subsequent miscalculations. Eq. 3
describes the surface calculation of each signed triangle with
$V_{\mathrm{i}}\in V$ and $i\in[1,n]$:

$V_{\mathrm{i}}^{{}^{\prime}}=\dfrac{1}{6}(-x_{\mathrm{i,3}}y_{\mathrm{i,2}}z_{\mathrm{i,1}}+x_{\mathrm{i,2}}y_{\mathrm{i,3}}z_{\mathrm{i,1}}+$
(3)
$x_{\mathrm{i,2}}y_{\mathrm{i,3}}z_{\mathrm{i,1}}+x_{\mathrm{i,3}}y_{\mathrm{i,1}}z_{\mathrm{i,2}}-x_{\mathrm{i,1}}y_{\mathrm{i,3}}z_{\mathrm{i,2}}-$
$-x_{\mathrm{i,2}}y_{\mathrm{i,1}}z_{\mathrm{i,3}}+x_{\mathrm{i,1}}y_{\mathrm{i,2}}z_{\mathrm{i,3}})$

We use i as the index for the triangles. $x_{\mathrm{i,[1,2,3]}}$,
$y_{\mathrm{i,[1,2,3]}}$ and $z_{\mathrm{i,[1,2,3]}}$ are the coordinates of
the vertices of triangle i. Various shapes of objects are not considered in
our analysis. Instead, each class is assigned to a specific 3D mesh. Fig. 6
exemplifies the resulting 3D mesh calculated by Eq. 3.

Figure 6: Object Detection and Triangle Estimation. Triangles: 5.022,
Vertices: 4.159, UV Channels: 4 with approx. Size: 50 $\times$ 198 $\times$
114.

After recognition, the applicable model is assigned. The object’s mass is
attributed based on the result of texture recognition. We simplify our
analysis by assuming solid material for the respective model. A further
database is created for the classification, which specifies the density of
each recognizable material.

###  3.4 Physical Properties

Physical quantities such as forces, friction, pressures, temperatures, air
resistances, inertia moments, energies or material properties $n$ depend on
density $\varrho$. The fundamental calculation of density $(\varrho=m/V)$ and
the inclusion of further coefficients and constants offers possibilities of
inferring different values.

By recognizing the actual object and the possible material assignment, we can
deduce the volume and mass of the object. The physical property is determined
by iterating over each section since many objects are divided into sub-
objects. The bicycle in Fig. 6, for example, includes the individual wheels,
the handlebars, the saddle, and the frame. For each section, the signed volume
is now calculated for each triangle and added to the total volume. Surfaces
that point outwards contribute to the total volume. Surfaces that point
inwards subtract from it. This leaves only the volume on the inside. The
density and volume can then be multiplied to calculate the weight of the
object.

##  4 Experimental Setup

Our approach utilizes Unreal Engine 4.27 and its high-fidelity rendering
pipeline. The realistic rendering and lighting allow us to assume real test
simulations as shown in Tab. 3. The neuronal training is based on realistic
test images.

|  |  |
---|---|---|---
Phone | Chair 1 | Chair 2 | Chair 3
|  |  |
Chair 4 | Table 1 | Hydrant | Bicycle 1
|  |  |
Bicycle 2 | Dog | Scene 1 | Scene 2
Table 3: Sample of Items used in our Experiment. We used Unreal Engine 4.27
for its realistic rendering capabilities.

The accuracy of the physical information is directly linked to the image
quality. In our conceptual consideration, we use images with a size of
$608\times 608$ pixels and three assigned color channels.

Our computing hardware has an integrated Quad-Core Intel or AMD, 2.5 GHz, 8GB
RAM, external GPU1 Nvidia GeForce GTX 1050 Ti, and onboard GPU0 of Intel HD
Graphics 630. Cv2 is used for image processing and reading deep neural
networks, and NumPy for mathematical functions. Mean subtraction calculates
the average pixel intensity over all images of the used training set of all
three color channels and subtracts these values from the channels of the input
image. When using YOLOv4, channel swapping is also applied for the mean
subtraction. Here the image is swapped in RGB order. In order to obtain one
optimal bounding box for each object, non-max suppression is applied. To
predict the material, we iterate over each object and use the corresponding
image area as input to the texture model, similar to the object detection
model.

Material 25$\%$ Silicon 23$\%$ Polypropylene 20$\%$ Iron 14$\%$ Aluminium7$\%$
Copper 6$\%$ Lead  Density
1.1 $kg/dm^{3}$ $\cdot$ 25$\%$ 1.2 $kg/dm^{3}$ $\cdot$ 23$\%$ 7.8 $kg/dm^{3}$
$\cdot$ 20$\%$ 2.7 $kg/dm^{3}$ $\cdot$ 14$\%$ 9.0 $kg/dm^{3}$ $\cdot$ 7$\%$
11.3 $kg/dm^{3}$ $\cdot$ 6$\%$ Texture
94$\%$ Plastic, 4$\%$ Glass, 1$\%$ Metal Material80$\%$ Skin 20$\%$ Hair
Density 1.0 $kg/dm^{3}$ $\cdot$ 80$\%$ 1.3 $kg/dm^{3}$ $\cdot$ 20$\%$ Texture
63$\%$ Unknown, 16$\%$ Fabric, 7$\%$ Plastic, 3$\%$ Skin, 2 $\%$ Ceramic, 2
$\%$ Food PhoneCat Figure 7: Material and Density Composition of Recognized
Objects. The inner ingredients are frequently more complex and diverse than
the exterior texture suggests. The illustration describes the composition of
two example objects: A smartphone and cat. The density of occluded components
can be estimated from the average composition of each object.

##  5 Evaluation

Our evaluation employs density recognition of several diverse objects.

We used a Convolutional Neural Network to recognize the texture of the object
and MINC for texture recognition. The model trained and used in our approach
shows the highest accuracy when compared to other CNN architectures.

In order to evaluate the functionality of our approach, we explored and tested
select scenarios (see Tab. 3). We summarize our results in Tab. 4.

| Material |  Density $[kg/dm^{3}]$
---|---|---
| Type | (Literary | Measured)
Phone | Plastic | (4.0 | 1.2) $\pm$ 70 %
Chair 1 | Wood | (0.7 | 0.7) $\pm$ 4 %
Chair 2 | Plastic | (4.8 | 1.2 ) $\pm$ 75 %
Chair 3 | Metal | (7.9 | 8.0) $\pm$ 1 %
Chair 4 | Metal | (0.9 | 1.2) $\pm$ 30 %
Table 1 | Metal | (7.9 | 8.0) $\pm$ 1 %
Hydrant | Metal | (7.9 | 8.2) $\pm$ 14 %
Bicycle 1 | Metal | (2.9 | 8.0) $\pm$ 180 %
Bicycle 2 | Metal | (7.9 | 8.0) $\pm$ 2 %
Bench 1 | Metal | (7.9 | 8.0) $\pm$ 1 %
Bench 2 | Wood | (2.1 | 0.7) $\pm$ 66 %
Dog | Other | (1.1 | 1.0) $\pm$ 5.6 %
Person | Plastic | (1.1 | 1.0) $\pm$ 9 %
Backback | Fabric | (1.4 | 1.6) $\pm$ 16 %
Car | Plastic | (5.4 | 1.2) $\pm$ 77 %
Table 4: Detected Materials and Density as well as Percentage Error [%] from
the actual Physical Values of the Objects. The measurements for Phone, Chair
2, Bicycle 1, Bench 2, and Car deviated significantly from the actual values.
These deviations stemmed from incorrect detection of the material.

However, the FMD dataset only achieves an accuracy of 52 $\%$ across all
classes. One of the reasons is the insufficient data set, which consists of
only ten categories in the neural network and 23 in the MINC data set.
Increasing the data density would significantly increase the hit rate. With
more extensive training, our approach can also be transferred to other
environments. This requires the RGB image for analysis and the database with
the necessary 3D networks and recognition models for evaluation.

The actual size of objects within a scene has not been considered in previous
work. This would be useful for volume calculation and different scaling of
objects. Even within the categories, no distinctions are made between
different types of objects. Each object is only assigned a 3D mesh, which is
considered the average for that class. This means that object shapes are not
taken into account. Furthermore, for successful mapping and analysis of
physical properties, the use of error-free, detailed, and complete 3D models
is essential. However, depending on the orientation and movement of the
objects, the calculated volume may be inaccurate. Serial images could help
alleviate this error.

##  6 Limitations

While our implementation shows promising initial results, it solely serves to
illustrate the feasibility of our proposed concept. Although our
implementation used neural networks that were trained with real-world images,
we relied on synthetic datasets to assess the performance of the
implementation. Consequently, the generalizability and applicability of our
findings may be limited.

Additionally, our evaluation of the detected physical properties examined the
average density of each object. Therefore, our work might not guarantee that
our approach is readily applicable to other properties, such as plasticity or
thermal conductivity, as well as to more complex object compositions.

##  7 Conclusion and Future Work

In this paper, we presented a concept for the object-based recognition and
assignment of physical properties as density or material based on a 2D image.
Our work is motivated by the challenges of distinguishing objects and their
properties from each other. The distinction of mass or density enables new
interaction possibilities in which the causal relationships of an environment
can be linked to the properties of a given object. Our method recognizes
specific patterns from 2D images by neural networks in which we estimate the
volume by the number of object-recognized triangles. The density is ultimately
calculated from the object-specific assignment of a material recognition model
and the associated volume.

Despite the promising results of our approach, further work and improvements
are needed. An essential aspect relates to the data sets that are used. The
accuracy of object-based density recognition goes hand in hand with the
quality of the trained AI model. Our current field of application is limited
to synthetic test data. Future iterations and evaluations using real-world
data sets could help deliver further insights into AI-based density
recognition. To achieve serviceable recognition and acceptable results, the
appropriate data set needs to be selected for the application area. Therefore,
it is necessary to extend the data sets and the training model. The COCO
dataset covers numerous categories but neglects existing subcategories. By
selecting such data sets, the transferability of the model could be increased.

Our evaluation shows only a limited number of materials assigned to the
objects. In reality, the number and composition of materials may be different
and more diverse (see Fig. 7). In particular, the interior composition of a
recognized object may differ from the recognized surface texture.
Consequently, drawing on and combining a wide variety of databases could lead
to more precise and serviceable results. These assignments can be linked using
an ontological approach. Utilizing ontologies, information and their
relationship to each other can be stored in a machine-readable form or made
comprehensible. Additional graph databases can visualize the data nodes and
their relationship and make them interpretable. In principle, different
ontologies can be merged within one ontology. In this context, possible
databases on physical properties such as material, geometry and objects could
be linked ontologically and applied to the principle of AI-based density
recognition.

Supporting these classifications with suitable image segmentation, such as
with self-organized maps [17], could further increase the number of
distinguishable materials. The partial change of the segmentable areas could
be cut out or reduced to densely recognizable areas, which would also reduce
quality restrictions and latencies.

Estimating the object size within a scene proves to be a difficult task. This
process assumes the same size for all everyday objects in a scene. In the
future, it will be necessary to measure the object size and distance of
acquired 3D models for meter-level distinctions. The model transfer to a
spatial data set would be suitable for this purpose.

The use of image series or video material can also be helpful to support a
spatial data set [7]. In this context, the distance and perspective of objects
within a scene can be used to determine the speed and possible acceleration of
an object. Javadi et al. [11] describe a video-based vehicle speed system for
measuring speed based on a measured route. By determining the speed and
acceleration of an object, statements can be made about the forces released in
a collision. Video analysis can also be useful for other areas of physical
property recognition. The depiction of an object in several individual images
with different perspectives allows properties derived previously to be checked
and re-evaluated. This includes, for example, the volume or size of the
object.

##  8 Acknowledgements

We thank Philip Raschdorf for his support during concept development and data
collection. We also thank Thomas Odaker and Elisabeth Mayer, who supported
this work with helpful discussions and feedback.

## References

  * [1] Flickr material database (fmd). MIT (2024-05-06), https://people.csail.mit.edu/celiu/CVPR2010/FMD/index.html
  * [2] Bell, S., Upchurch, P., Snavely, N., Bala, K.: Material recognition in the wild with the materials in context database. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 3479–3487 (2015). https://doi.org/10.1109/CVPR.2015.7298970
  * [3] Cai, Z., Fan, Q., Feris, R., Vasconcelos, N.: A unified multi-scale deep convolutional neural network for fast object detection. In: Computer Vision - ECCV. pp. 354–370 (2016). https://doi.org/10.1007/978-3-319-46493-0 _22
  * [4] Daglioglu, M.A.: Object-detection mit you only look once (yolo) : Einführung in die objekterkennung mit yolo sowie die weiterentwicklung in den versionen v2-v4 (2021)
  * [5] Erdem, K.: Understanding region of interest (roi pooling) (2020), https://erdem.pl/2020/02/understanding-region-of-interest-ro-i-pooling, (visited on 2024-02-20)
  * [6] Ester, M., Sander, J.: Knowledge discovery in databases. Springer (2000). https://doi.org/10.1007/978-3-642-58331-5
  * [7] Evain, A., Khemmar, R., Orzalesi, M., Ahmedali, S.: Impact of calibration matrices on 3d monocular object detection: Filtering, dataset combination and integration of synthetic data. International Conferences in Central Europe on Computer Graphics, Visualization and Computer Vision (WSCG) (2024)
  * [8] Girshick, R.: Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision. pp. 1440–1448 (2015)
  * [9] Ippolito, P.P.: Feature extraction techniques (2019), https://towardsdatascience.com/feature-extraction-techniques-d619b56e31be, (visited on 2024-02-20)
  * [10] Janiesch, C., Zschech, P., Heinrich, K.: Machine learning and deep learning. Electronic Markets 31(3), 685–695 (2021). https://doi.org/10.1007/s12525-021-00475-2
  * [11] Javadi, S., Dahl, M., Pettersson, M.I.: Vehicle speed measurement model for video-based systems. Computers & electrical engineering 76, 238–248 (2019). https://doi.org/10.1016/j. compeleceng.2019.04.001
  * [12] Jocher, G., Chaurasia, A., Qiu, J.: Ultralytics yolov8 (2023), https://github.com/ultralytics/ultralytics, (visited on 2024-02-20)
  * [13] Krig, S.: Computer vision metrics: Survey, taxonomy and analysis of computer vision, visual neuroscience, and deep learning. Springer (2016)
  * [14] Liu, C., Sharan, L., Adelson, E.H., Rosenholtz, R.: Exploring features in a bayesian framework for material recognition. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2010). https://doi.org/10.1109/CVPR.2010.5540207
  * [15] Mehlig, B.: Machine learning with neural networks: an introduction for scientists and engineers. Cambridge University Press (2021)
  * [16] Müller, J., Fregin, A., Dietmayer, K.: Disparity sliding window: Object proposals from disparity images. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 5777–5784 (2018). https://doi.org/10.1109/IROS.2018.8593390
  * [17] Müller, S., Kranzlmüller, D.: Self-organising maps for efficient data reduction and visual optimisation of stereoscopic based disparity maps. In: International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision (2022). https://doi.org/10.24132/CSRN.2021.3101.3
  * [18] Nicodemus, F.E.: Directional reflectance and emissivity of an opaque surface. In: Applied Optics (1965). https://doi.org//10.1364/AO.4.000767
  * [19] Nielsen, M.A.: Neural networks and deep learning. Determination press, San Francisco, CA, USA (2015)
  * [20] Redmon, J., Farhadi, A.: Yolo9000: Better, faster, stronger. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017). https://doi.org/10.1109/CVPR.2017.690
  * [21] Redmon, J., Farhadi, A.: Yolov3: An incremental improvement. Tech. rep. (2018). https://doi.org/10.48550/arXiv.1804.02767
  * [22] Ren, S., He, K., Girshick, R., Jian, S.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: IEEE transactions on pattern analysis and machine intelligence. pp. 1137–1149 (2017). https://doi.org/10.1109/TPAMI.2016.2577031
  * [23] Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. nature 323(6088), 533–536 (1986)
  * [24] Shalev-Shwartz, S., Ben-David, S.: Understanding machine learning: From theory to algorithms. Cambridge university press (2014)
  * [25] Shukla, A., Kalnoor, G., Kumar, A., Yuvaraj, N., Manikandan, R., Ramkumar, M.: Improved recognition rate of different material category using convolutional neural networks. Materials Today: Proceedings 81, 947–950 (2023)
  * [26] Sun, Y., Gu, Z.: Using computer vision to recognize construction material: A trustworthy dataset perspective. In: Elsevier (2022). https://doi.org/10.1016/j. resconrec.2022.106362
  * [27] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1–9 (2015). https://doi.org/10.1109/CVPR.2015.7298594
  * [28] Wu, J., Lim, J.J., Zhang, H., Tenenbaum, J.B., Freeman, W.T.: Physics 101: Learning physical object properties from unlabeled videos. In: Proceedings of the British Machine Vision Conference (2016)
  * [29] Yadav, N., Yadav, A., Kumar, M.: An introduction to neural network methods for differential equations. Springer (2015). https://doi.org/10.1007/978-94-017-9816-7