# NeuralMagicEye: Learning to See and Understand the Scene Behind an
Autostereogram

Zhengxia Zou1, Tianyang Shi2, Yi Yuan2, Zhenwei Shi3
1University of Michigan, Ann Arbor, 2NetEase Fuxi AI Lab, 3Beihang University

###### Abstract

An autostereogram, a.k.a. magic eye image, is a single-image stereogram that
can create visual illusions of 3D scenes from 2D textures. This paper studies
an interesting question that whether a deep CNN can be trained to recover the
depth behind an autostereogram and understand its content. The key to the
autostereogram magic lies in the stereopsis - to solve such a problem, a model
has to learn to discover and estimate disparity from the quasi-periodic
textures. We show that deep CNNs embedded with disparity convolution, a novel
convolutional layer proposed in this paper that simulates stereopsis and
encodes disparity, can nicely solve such a problem after being sufficiently
trained on a large 3D object dataset in a self-supervised fashion. We refer to
our method as “NeuralMagicEye”. Experiments show that our method can
accurately recover the depth behind autostereograms with rich details and
gradient smoothness. Experiments also show the completely different working
mechanisms for autostereogram perception between neural networks and human
eyes. We hope this research can help people with visual impairments and those
who have trouble viewing autostereograms. Our code is available at
https://jiupinjia.github.io/neuralmagiceye/.

Figure 1: We propose NeuralMagicEye, a neural network based method for solving
autostereograms and understanding the scenes behind. Scan the recovered QR-
code with your smartphone and check out for more details about how to view an
autostereogram.

##  1 Introduction

An autostereogram is a picture within a picture where a hidden scene will
emerge in 3D when the image viewed correctly [9, 30, 33]. Autostereograms are
also called “magic eye images” - viewing autostereograms is like a brain
teaser, some people may find it easy, but others may find it very hard even
after considerable practice. The key to the magic of autostereograms lies in
the stereopsis. Human brain processes depth information from complex
mechanisms by matching each set of points in one eye’s view with the same set
of points in the other eye’s view. Autostereograms became part of visual art
and widely appear in posters and books after their great popularity in the
1990s [30, 5]. It also has attracted much attention for the research of
psychophysics and the understanding of the human vision in the past 30 years.

In this paper, we investigate a very interesting question that whether we can
train a neural network to recover the hidden depth behind the autostereograms
and understand their content. In computer vision and computer graphics, both
stereo-vision [19] and autostereogram synthesis [30, 28] are popular and
widely studied topics. However, so far, the “inverse autostereogram” problem
that we will talk about in this paper has rarely been studied. Recovering
depth from an autostereogram, which requires detecting disparity from quasi-
periodic textures, is totally different from those conventional pixel-wise
image prediction tasks, such as image segmentation [18, 3] and image super-
resolution [4]. In conventional pixel-wise prediction tasks, the spatial
correspondence in the convolution operation provides an important inductive
prior. However, when it comes to predicting the disparity in autostereograms,
the loss of spatial correspondences makes it difficult for us to directly
apply standard CNNs for such a task.

The key to our method is called “disparity convolution”, a novel convolutional
layer proposed in this paper, which simulates stereopsis by encoding the
feature discrepancy between horizontally neighboring pixels. The disparity
convolution can help re-introduce spatial correspondence to the autostereogram
decoding process. One of our important conclusions is that deep CNNs with
embedded disparity convolution are capable of solving such a problem and
greatly improves the decoding accuracy. Another good property of our method is
that our training is performed in a pure self-supervised learning fashion,
without requiring any manual annotations. To build the self-supervised
training loop, we introduce a graphic renderer and an autostereogram generator
so that we can continuously generate a large number of autostereograms and the
corresponding depth images on the fly. We train our model on ShapeNet [2], a
large 3D object shape dataset, by minimizing the difference between the
decoding output and the rendered depth map. Experiments show that after
sufficient training under the proposed self-supervised framework, our method
can accurately recover the hidden depth behind an autostereogram with rich
details and gradient smoothness. When testing on the online autostereograms
generated by different graphic engines and those with various degradations
(e.g. image blurring, jpg compression, and overexposure), our method also
shows very good generalization performance.

Our method also has some potential applications. One is that our method can be
naturally applied to the retrieval of autostereograms, where traditional
retrieval methods may fail in such cases since the image content (depth) is
typically hidden in the textures and can not be perceived directly by those
models. Another potential application is digital watermarking. We show that
our method is able to encode a hidden layer of depth into an image carrier,
and then restore this hidden layer under some noise tolerance. Finally, we
hope this research can help people with visual impairments, especially those
with amblyopia who have trouble viewing autostereograms.

Figure 2: An overview of our method. Our method consists of a graphic renderer
$G_{R}$, an autostereogram generator $G_{A}$, and a decoding network $f$. We
train our method in a self-supervised way by minimizing the difference between
its decoding outputs and the rendered depth.

##  2 Related Work

The research on autostereograms has a long history and can be traced back to
the 1960s [10]. Julesz _et al_. were the first to invent random-dot
stereograms and discovered that the impression of depth could arise purely
from stereopsis, without relying on other cues such as perspective or contours
[9]. At the time, scientists believed that depth perception occurred in the
eye itself, but now it is proved to be a complex neurological process [33]. In
the 1990s, Tyler _et al_. described a way of combining the left and right eye
random dots into one picture and invented “single eye random-dot stereograms”
(later known as autostereograms) [30], which do not require the use of special
equipment for viewing 3D effect. This new type of stereoscopic image then has
attracted the attention of many researchers [27, 28, 22, 21] and quickly
became a popular form of visual art [5].

Despite the extensive research on the synthesis algorithm and applications of
autostereograms, the recovery of the hidden depth information in an
autostereogram is still a problem that has been rarely studied. Some early
works show that self-organizing neural networks can discover disparity in
random-dot stereograms [1, 16], which suggests that neural networks may have
great potential for solving this problem. Later, some correlation-based
methods were proposed for the decoding of autostereogram in both spatial
domain [12, 24, 11] and frequency domain [29]. However, the recovered depth
image of these methods is often inaccurate and noisy. Some of these methods
[24] require a manually defined depth searching range, and some others [29, 6]
can only extract the outline of the objects and cannot get their exact depth
value. Different from all the above approaches, in this paper, we proposed to
use deep neural networks to solve this problem and our results show clear
advantages over the previous ones.

Figure 3: The geometry behind an autostereogram and the principle of
autostereogram generation.

##  3 Methodology

Fig. 2 shows an overview of our method. A complete training pipeline of our
method involves three basic modules: 1) a graphic renderer $G_{R}$ that
renders depth images from 3D object models; 2) an autostereogram generator
$G_{A}$ that encodes depth and synthesizes autostereograms; 3) a decoding
network $f$ that recovers the depth. In the following, we will first talk
about some geometry basis of an autostereogram and then introduce how each
module works in detail.

###  3.1 Autostereogram geometry

The human eye can produce a 3D illusion on 2D patterns that repeats at a
certain frequency. In an autostereogram, the depth information is encoded in a
series of 2D quasi-periodic textures - when the texture is repeated at a
higher frequency, that texture area will appear closer to the screen and vices
versa [30, 33]. Fig. 3 shows the geometry basis of autostereograms.

Suppose we have a screen placed at the location $z_{0}$. We set the near-
clipping plane and the far-clipping plane of the virtual object to the
location $z_{near}$ and $z_{far}$. Within the range of $z_{near}$ to
$z_{far}$, each point in the object’s surface can be represented as a
normalized depth value $d$ ($0\leq d\leq 1$). Through simple geometric
calculation, we can establish an approximate relationship between the depth
$d$ and disparity $s$ as follows:

$s/s_{d=1}\approx(1-\beta(1-d)),$ (1)

where $\beta=(z_{far}-z_{near})/(z_{far}-z_{0})$ is a constant. $s_{d=1}$ is
the disparity of the far-clipping plane on the screen, as well as the width of
the background texture stripes.

Given a depth image and a background stripe, to generate the autostereogram,
we first tile the background stripes to fill the entire output image, then
scan each pixel in the output image and shift it along the horizontal axis
based on the required disparity in (1). To recover the depth image from an
autostereogram, the most straightforward way is to set a small horizontal
window for each pixel in the autostereogram and search for correspondence of
row segments within a certain searching range. The shifting distance of the
best matching window can be thus used to compute the normalized depth $d$ in
the depth image. However, as shown in previous research [24], this method
requires manually determining the searching range and may produce noisy and
even wrong depth images.

###  3.2 Disparity convolution

In recent years, deep CNNs [14, 26, 7] have been extensively used in pixel-
wise image prediction tasks, in which the spatial correspondence of the
convolution operation provides important inductive priors for such tasks.
However, in our method, the networks are trained to learn a mapping from
pixels to the disparity. In this case, the spatial correspondence will be lost
in most image regions. To address this problem, we propose “disparity
convolution” by extending a standard convolution with a demodulation
operation. The basic idea is to compute the discrepancy of each feature vector
in the feature map with its horizontal neighbors and save the values to
corresponding feature channels.

Fig. 4 shows an illustration of the proposed disparity convolution layer.
Suppose we have a feature map $x$ with the size of $h\times w\times c$. For
each pixel location $(i,j)$ and each channel $k$ in the feature map $x$, we
compute the discrepancy between $x(i,j,k)$ and its horizontal neighbor pixel
$x(i,j-s,k)$ with $s$-pixel distance. Then we compute a set of difference-maps
by sum-up the discrepancy for each spatial location along their channel
dimension:

$u(i,j,s)=\sum_{k=1}^{c}|x(i,j,k)-x(i,j-s,k)|,$ (2)

where $u$ are the generated difference-maps with the size of $h\times w\times
m$. $m$ is a pre-defined the distance-bound of horizontal neighbors ($0<m\leq
w$). We finally apply a standard convolution layer on $u$ and produces the
output feature map $x^{\prime}$ as the output of our disparity convolution
layer.

To speed up computation, an equivalent but more efficient way of the above
calculation is to first circularly shift the feature map along its horizontal
axis and then make element-wise subtraction with its input, as shown in Fig.
4. In PyTorch [23], this can be further speed-up by using the “Unfold”
operation and broadcasting, without using any for-loops in the computation.
Besides, another good property of the disparity convolution is that it will
not introduce any additional parameters compared to a standard convolution
layer.

Figure 4: An illustration of the proposed disparity convolution layer.

###  3.3 Self-supervised training

We train our autostereogram decoding network under a self-supervised learning
paradigm. Given a set of 3D object models (e.g. 3D meshes), we introduce a
graphic renderer $G_{R}$ and for each object model $v$, we use $G_{R}$ to
render a depth image $I_{d}=G_{R}(v|\gamma)$ by choosing from a set of random
pose and view parameters $\gamma\in\Gamma$, where $\Gamma$ defines the
feasible range of the parameters. We then build an autostereogram generator
$G_{A}$, which takes in the rendered depth and a random texture file, and
generates an autostereogram image $I_{s}=G_{A}(I_{d})$.

For each $I_{s}$, we feed it into the decoding network $f$ and produce a
decoding output $f(I_{s})$. Since both the depth image and the generated
autostereogram are available during the rendering pipeline, we can use this
data pair to effectively train the decoding network $f$ and enforce its output
to be similar to the ground truth $I_{d}$. We train the network $f$ with
standard $\ell_{2}$ pixel regression losses and minimize the following
objective function:

$\begin{split}\mathcal{L}_{f}&=\mathbb{E}_{v\in
V,\gamma\in\Gamma}\\{{\|f(I_{s})-I_{d}\|}_{2}^{2}\\}\\\ &=\mathbb{E}_{v\in
V,\gamma\in\Gamma}\\{{\|f(G_{A}(G_{R}(v|\gamma)))-G_{R}(v|\gamma)\|}_{2}^{2}\\},\end{split}$
(3)

where $V$ is the dataset of 3D object models, on which our decoder $f$ will be
trained.

Figure 5: Some online autostereograms and their decoding results by using our
method. The autostereograms are generated by different authors using different
graphic engines.
Figure 6: A visual comparison of the autostereogram decoding results by using
different methods: “DeMagicEye” by K. Scott [11], “Magiceye-solver” by T.
Hearn [29], “Stereogram Viewer” by F. G. Loch [6], and ours.

###  3.4 Implementation Details

Network architecture. We build our decoding network on top of two popular
network architectures: resnet18 [7] and unet [25]. For both architectures, a
disparity convolution layer and a ReLU layer are inserted at the input end of
the networks. For the resnet18 backbone, we also build a 6-layer upsampling
convolution network as its pixel-wise prediction head. For the unet backbone,
we tested on different configuration versions with the input image size of
64x64, 128x128, 256x256 pixels respectively. We remove the activation layer
from the output layer since in this way we observed a better convergence. In
our disparity convolution layer, we set the max shifting distance to 1/4 of
the height of the input image. We also tested other configurations such as
layer normalization [8, 31] and feature fusion [17], where more details will
be given in our controlled experiment.

Dataset and image synthesis. We train our decoding network on “ShapeNetCore”
[2], a large 3D shape dataset, which covers 55 common object categories with
over 50,000 unique 3D models. We use Pyrender [20] to render 2D depth images
of these 3D objects. We set the size of the viewport to 1024x1024 pixels and
set yfov to 45°. We randomly split the dataset into a training set (90%) and a
testing set (10%). We also collected 718 texture tiles online (585 for
training and 133 for testing) for autostereogram generation. The generated
autostereograms are resized to 64x64, 128x128, or 256x256 pixels for training
depending on the choice of the network configuration.

Training details. We train our decoding network by using Adam optimizer [13].
We set the batch size to 32, learning rate to 2e-4, and betas to (0.9, 0.999).
We set the max epoch number to 100 and reduce the learning rate to its 1/10
after 50 epochs. Data augmentation including random cropping, flipping,
rotation, blurring, and jpg compression are performed during the training to
enhance the robustness of the decoding network.

Table 1: Quantitative comparison of the decoding accuracy of different methods on the autostereograms generated from two datasets: ShapeNet [2] and MNIST [15]. | ShapeNet | MNIST
---|---|---
Method | PSNR | SSIM | PSNR | SSIM
DeMagicEye [11] | 7.106 | 0.258 | 7.188 | 0.145
Magiceye-solver [29] | 10.95 | 0.389 | 11.269 | 0.327
Stereo-ResNet (Ours) | 23.12 | 0.903 | 27.059 | 0.937
Stereo-UNet (Ours) | 23.88 | 0.900 | 27.09 | 0.895

Figure 7: Robustness test of our method on different types of image
degradation: image blurring, jpg compression (OpenCV JPEG_QUALITY=20), and
overexposure.
Figure 8: (a) Image carrier. (b) Superimposed images by combining the clean
images in (a) and autostereograms in which some characters are encoded. (c)
Decoding output from (b). Scan the QR-code with your smartphone and check out
more details.

##  4 Experimental Analysis

###  4.1 Decoding autostereograms

Fig. 5 shows some online autostereogram images generated by different
autostereogram graphic engines and their decoding results by using our method.
We compare our method with two open-source autostereogram decoding methods
[11, 29], and a free autostereogram decoding software [6]. Fig. 6 shows the
comparison results.

In the comparison, we also test on the autostereograms generated from the
MNIST dataset [15] where the digits are used as depth images. We train our
network on MNIST for 10 epochs. Table 1 shows their decoding accuracy. We use
PSNR and SSIM [32] as metrics of recovery accuracy. Both the decoding result
and the ground truth depth are normalized to [0, 1] before the accuracy is
computed. We can see our method produces much better results in terms of both
visual quality and quantitative scores.

###  4.2 Robustness

To test the robustness of our decoding network, we apply different
degradations to the autostereograms, including image blurring, jpg
compression, and overexposure. Fig. 7 shows the degraded input and the
recovered depth. We can see that although many high-frequency details are lost
after degradation, the depth can be still nicely recovered. This shows that
the neural network has successfully learned to perform disparity prediction
and information loss recovery at the same time. Besides, the results of
overexposure show that our method is not sensitive to global intensity
changes.

###  4.3 Digital watermarking

In this experiment, we investigate whether neural networks are capable of
recovering depth from a carrier image in which the autostereogram is embedded
as hidden watermarks. We first generate a set of autostereograms based on
random characters and QR-codes. The pixel values in the character images and
QR-codes are recorded as the depth value in the autostereograms. We then train
our decoder on a set of superimposed images $I_{w}$. These images are
generated as a linear combination of a background image carrier $I_{b}$ and an
autostereogram $I_{s}$:

$I_{w}=\alpha I_{s}+(1-\alpha)I_{b}.$ (4)

We set $\alpha$ as a random number within [0.1, 0.9] during training and set
$\alpha=0.2$ during testing. It is worth noting that this task is very
challenging since the depth is first encoded as disparities within the
textures and then mixed with the image carrier. The decoder is thus trained to
deal with a “double recovery” problem of the hidden depth information. Fig. 8
shows a group of clean images (image carrier), superimposed images, and their
decoding outputs. We can see that although the autostereograms are barely
visible in the superimposed images, the hidden information can be still
clearly recovered by our decoder.

Figure 9: Some image retrieval results by using our method. We test on two
retrieval modes: (a)-(b) keywords based retrieval, and (c)-(d) query image
based retrieval.

###  4.4 Autostereogram retrieval

Our method can be also applied to autostereogram retrieval. To complete such a
task, the model has to learn to understand the semantics behind the
autostereograms. We, therefore, build an autostereogram recognition network by
replacing the upsampling convolution head in our decoding network with a fully
connected layer to predict the class probability of the input. We test on two
retrieval modes: retrieval by keywords and by query image. In the keyword
retrieval mode, we iterate through all the autostereograms in the database
(our testing set) and select the top-k matching results based on the
prediction class probability. In the query image retrieval mode, the top-k
matching results are selected by computing the feature distance between the
query and the matching images. Fig. 9 shows the retrieval results of the above
two modes. We can see although the top-k matching results have very different
texture appearance, they share the same semantics in their depth images.

Figure 10: A visual comparison of the decoding output of our method w/ and w/o using disparity convolution. Table 2: Controlled experiments on the influence of each component in our decoding model, including the choice of different backbones, layer normalization, disparity convolution, and feature fusion. The scores are computed as the mean pixel accuracy of the decoding outputs on the autostereograms generated from two datasets (ShapeNet [2] and MNIST [15]). | Configuration | ShapeNet Decode | MNIST Decode
---|---|---|---
| Backbone | Normalization | DisparityConv | FeatFusion | PSNR | SSIM | PSNR | SSIM
Res18F | resnet18 | none |  |  | 18.16 | 0.777 | 17.83 | 0.787
Res18F (BN) | resnet18 | batch |  |  | 19.92 | 0.841 | 18.99 | 0.808
Stereo-Res18F (BN) | resnet18 | batch | $\checkmark$ |  | 22.63 | 0.900 | 25.31 | 0.919
Res18F (IN) | resnet18 | instance |  |  | 20.34 | 0.851 | 18.84 | 0.812
Stereo-Res18F (IN) | resnet18 | instance | $\checkmark$ |  | 22.53 | 0.896 | 24.34 | 0.905
Res18F (BN) + FF | resnet18 | batch |  | $\checkmark$ | 20.46 | 0.863 | 19.49 | 0.802
Stereo-Res18F (BN) + FF | resnet18 | batch | $\checkmark$ | $\checkmark$ | 23.12 | 0.903 | 27.06 | 0.937
Res18F (IN) + FF | resnet18 | instance |  | $\checkmark$ | 20.69 | 0.865 | 20.08 | 0.825
Stereo-Res18F (IN) + FF | resnet18 | instance | $\checkmark$ | $\checkmark$ | 23.17 | 0.909 | 26.94 | 0.937
UNet | unet128 | none |  | – | 20.48 | 0.822 | 19.68 | 0.820
UNet (BN) | unet128 | batch |  | – | 20.85 | 0.803 | 21.32 | 0.827
Stereo-UNet (BN) | unet128 | batch | $\checkmark$ | – | 23.88 | 0.900 | 27.09 | 0.895
UNet (IN) | unet128 | instance |  | – | 20.76 | 0.764 | 22.08 | 0.830
Stereo-UNet (IN) | unet128 | instance | $\checkmark$ | – | 23.70 | 0.842 | 27.26 | 0.915
Table 3: Controlled experiments on the influence of each component in our recognition model, including the choice of different backbones, layer normalization, disparity convolution. The scores are computed as the classification accuracy on the autostereograms generated from two datasets (ShapeNet [2] and MNIST [15]). We also evaluate an upper-bound model (*Upperbound Res18) where the model is trained and tested directly on the depth images. | Configuration |  |
---|---|---|---
| Backbone | Normalization | DisparityConv | ShapeNet | MNIST
Res18 | resnet18 | none |  | – | 93.30%
Res18 (BN) | resnet18 | batch |  | 48.06% | 96.23%
Stereo-Res18 (BN) | resnet18 | batch | $\checkmark$ | 66.81% | 98.68%
Res18F (IN) | resnet18 | instance |  | 47.60% | 96.74%
Stereo-Res18 (IN) | resnet18 | instance | $\checkmark$ | 64.60% | 98.73%
*Upperbound (Res18) | resnet18 | batch | – | 77.26% | 99.41%

###  4.5 Controlled experiments

We designed a series of controlled experiments to comprehensively study the
influence of different technical components in our method. These components
include the use of our disparity convolution, layer normalization, and feature
pyramid fusion. All the controlled experiments are conducted on both ShapeNet
and MNIST dataset. We evaluate the recognition accuracy and pixel-level
decoding accuracy of our networks with different architecture configurations,
as shown in Table 2 and Table 3. Since we do not need a pixel-wise prediction
head in our classification task, in Table 3, we only evaluate the resnet18
architecture with the disparity convolution and layer normalization. Some
useful conclusions on each of these components can be drawn as follows.

Disparity convolution. Our experiments show that the disparity convolution
plays a crucial role in both decoding and classification tasks. From row 6-9,
11-14 of Table 2 and row 2-5 of Table 3, we can see that when we remove the
disparity convolution layers from our best models, we see a significant
accuracy drop on both the decoding task (resnet PSNR -2.48 on ShapeNet, -7.57
on MNIST; unet PSNR -3.03 on ShapeNet, -5.18 on MNIST) and the classification
task (resnet acc -18.75% on ShapeNet; -1.99% on MNIST). Besides, from row 2-5
of Table 2, we can see the integrating of the disparity convolution in some
other sub-optimal models also brings noticeable improvement on their
decoding/classification accuracy (resnet PSNR +2.71 on ShapeNet, +6.32 on
MNIST). In Fig. 11, we plot the validation accuracy on different training
epochs.111It should be noted that in Fig. 11, 12 and 13, the validation
accuracy is computed on 128x128 images, while in Table 2, the final testing
accuracy is computed on 512x512 images. That is why there is an accuracy gap
between these two groups of results. We can see that the integration of the
disparity convolution not only brings a higher accuracy but also a much faster
convergence speed in both resnet and unet architectures.

Normalization. In this experiment, we first start from training a baseline
model without using any layer normalization. Then we try two popular
normalization methods - batch normalization [8] and instance normalization
[31] and compare with the baselines. The impact on the resnet and unet
architectures are both evaluated. From row 1, 2, 4, 10, 11, 13 of Table 2, and
row 1, 2, 4 of Table 3, we can see that the layer normalization are important
for both tasks. For example, batch normalization brings noticeable accuracy
improvement on the decoding (resnet PSNR +1.76 on ShapeNet, +1.16 on MNIST;
unet PSNR +0.37 on ShapeNet, +1.64 on MNIST) and classification tasks (resnet
acc +2.93% on MNIST). Particularly, we notice that when there is no
normalization used in the ShapeNet classification task, the model will not
converge during the training. In Fig. 12, we show the validation accuracy w/
and w/o using layer normalization on different training epochs.

Feature fusion. In conventional pixel-wise prediction tasks, feature fusion
between layers of different depth and resolutions are crucial for generating
high-resolution outputs. The representatives of such an idea include the layer
skip connection in the unet [25] and feature pyramid networks in the object
detection networks [17]. From row 3, 5, 7, 9 of Table 2, we can see that the
feature pyramid fusion can bring a minor accuracy improvement (resnet PSNR
+0.49 on ShapeNet, +1.75 on MNIST) on our sub-optimal models. When testing on
our baseline models (w/o disparity conv), the feature fusion can also bring
similar improvement on the decoding accuracy (resnet PSNR +0.54 on ShapeNet,
+0.5 on MNIST). These results suggest that in autostereogram decoding,
although the spatial correspondences are much weaker than other image pixel
prediction tasks, feature fusion can also slightly improve the accuracy of the
decoding outputs. In Fig. 13, we show the validation accuracy (PSNR) w/ and
w/o using feature fusion on different training epochs.

Figure 11: Validation accuracy on different training epochs with (Stereo-
Res18F, Stereo-UNet) and without (Res18F, UNet) using disparity convolutions.
Figure 12: Validation accuracy during the training with different layer
normalization methods.
Figure 13: Validation accuracy on different training epochs with (+FF) and
without using feature fusion in the pixel-wise prediction head.
Figure 14: (a) An “neural autostereogram” generated by our unet-128 network
that minimizes the difference between the target depth (b) and its decoding
output (c). (d)-(f) Decoding outputs of the autostereogram in (a) with
different network architectures. We found surprisingly that although the depth
information in the generated autostereogram (a) cannot be perceived by human
eyes, the depth can be nicely recovered by neural networks with different
architectures. This experiment suggests that different neural networks may
follow a certain commonality when processing autostereograms, but the
mechanism behind may be completely different from that of human eye
stereopsis.

###  4.6 Neural autostereogram

We finally investigated an interesting question that given a depth image, what
an “optimal autostereogram” should look like in the eyes of a decoding
network. The study of this question may help us understand the working
mechanism of neural networks for autostereogram perception. To generate the
“optimal autostereogram”, we run gradient descent on the input end of our
decoding network and minimize the difference between its output and the
reference depth image. In Fig. 14 (a), we show the generated “optimal
autostereogram” on our unet decoding network. We named it “neural
autostereogram”. In Fig. 14 (b)-(c) we show the reference depth and the
decoding output.

An interesting thing we observed during this experiment is that, although the
decoding output of the network is already very similar to the target depth
image, however, human eyes still cannot perceive the depth hidden in this
neural autostereogram. Also, there are no clear periodic patterns in this
image, which is very different from those autostereograms generated by using
graphic engines. More surprisingly, when we feed this neural autostereogram to
other decoding networks with very different architectures, we found that these
networks can miraculously perceive the depth correctly. To confirm that it is
not accidental, we also tried different image initialization and smooth
constraints but still have similar observations. Fig. 14 (d)-(f) show the
decoding results on this image by using other different decoding networks.
This experiment suggests that neural networks and the human eye may use
completely different ways for stereogram perception. The mechanism and
properties behind the neural autostereograms are still open questions and need
further study.

##  5 Conclusion

We propose “NeuralMagicEye”, a deep neural network based method for perceiving
and understanding the scenes behind autostereograms. We train our networks on
a large 3D object dataset under a self-supervised learning fashion and the
results show good robustness and much better decoding accuracy than other
methods. We also show that the disparity convolution, a new form of
convolution proposed in this paper, can greatly improve the autostereogram
decoding accuracy of deep CNNs. Controlled experiments suggest the
effectiveness of our design. Finally, we show that the way a neural network
perceiving depth in an autostereogram is different from that of human eyes.
The deeper mechanism behind this needs to be further explored.

## References

  * [1] Suzanna Becker and Geoffrey E Hinton.  Self-organizing neural network that discovers surfaces in random-dot stereograms.  Nature, 355(6356):161–163, 1992.
  * [2] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  * [3] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2017.
  * [4] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Image super-resolution using deep convolutional networks.  IEEE transactions on pattern analysis and machine intelligence, 38(2):295–307, 2015.
  * [5] NE Thing Enterprises.  The Magic Eye, Volume I: A New Way of Looking at the World.  Andrews McMeel Publishing, 1993.
  * [6] Francis G. Loch.  Stereogram viewer v0.3a, 2014.  [Online; accessed 12-30-2020].
  * [7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [8] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  arXiv preprint arXiv:1502.03167, 2015.
  * [9] Bela Julesz.  Stereoscopic vision.  Vision Research, 26(9):1601–1612, 1986.
  * [10] Bela Julesz and Joan E Miller.  Automatic stereoscopic presentation of functions of two variables.  Bell System Technical Journal, 41(2):663–676, 1962.
  * [11] Katherine Scott.  Solving autostereograms aka magic eyes, 2013.  [Online; accessed 12-30-2020].
  * [12] Ron Kimmel.  3d shape reconstruction from autostereograms and stereo.  Journal of Visual Communication and Image Representation, 13(1-2):324–333, 2002.
  * [13] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [14] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84–90, 2017.
  * [15] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [16] Christopher W Lee and Bruno A Olshausen.  A nonlinear hebbian network that learns to detect disparity in random-dot stereograms.  Neural computation, 8(3):545–566, 1996.
  * [17] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [18] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [19] David Marr and Tomaso Poggio.  A computational theory of human stereo vision.  Proceedings of the Royal Society of London. Series B. Biological Sciences, 204(1156):301–328, 1979.
  * [20] Matthew Matl.  Pyrender, 2019.  [Online; accessed 12-30-2020].
  * [21] Minh Nguyen and Albert Yeap.  Stereotag: A novel stereogram-marker-based approach for augmented reality.  In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 1059–1064. IEEE, 2016.
  * [22] Jacques Ninio.  The science and craft of autostereograms.  Spatial vision, 21(1-2):185–200, 2007.
  * [23] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In Advances in neural information processing systems, pages 8026–8037, 2019.
  * [24] Kekoa Proudfoot.  An autostereogram decoder, 2003.
  * [25] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [26] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [27] Maria S Terrell and Robert E Terrell.  Behind the scenes of a random dot stereogram.  The American Mathematical Monthly, 101(8):715–724, 1994.
  * [28] Harold W. Thimbleby, Stuart Inglis, and Ian H. Witten.  Displaying 3d images: Algorithms for single-image random-dot stereograms.  Computer, 27(10):38–48, 1994.
  * [29] Tristan Hearn.  magiceye-solver, 2013.  [Online; accessed 12-30-2020].
  * [30] Christopher W Tyler and Maureen B Clarke.  Autostereogram.  In Stereoscopic displays and applications, volume 1256, pages 182–197. International Society for Optics and Photonics, 1990.
  * [31] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Instance normalization: The missing ingredient for fast stylization.  arXiv preprint arXiv:1607.08022, 2016.
  * [32] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [33] Wikipedia contributors.  Autostereogram, 2020.  [Online; accessed 12-30-2020].


# Molecular Graph Generation via Geometric Scattering

Dhananjay Bhaskar
Department of Genetics
Yale University
New Haven, CT 06511, USA
<EMAIL_ADDRESS>
& Jackson D. Grady
Department of Computer Science
Yale University
New Haven, CT 06511, USA
<EMAIL_ADDRESS>
Michael A. Perlmutter
Department of Mathematics
UCLA
Los Angeles, CA 90095, USA
<EMAIL_ADDRESS>
& Smita Krishnaswamy
Department of Computer Science
Department of Genetics
Yale University
New Haven, CT 06511, USA
<EMAIL_ADDRESS>

###### Abstract

Graph neural networks (GNNs) have been used extensively for addressing
problems in drug design and discovery. Both ligand and target molecules are
represented as graphs with node and edge features encoding information about
atomic elements and bonds respectively. Although existing deep learning models
perform remarkably well at predicting physicochemical properties and binding
affinities, the generation of new molecules with optimized properties remains
challenging. Inherently, most GNNs perform poorly in whole-graph
representation due to the limitations of the message-passing paradigm.
Furthermore, step-by-step graph generation frameworks that use reinforcement
learning or other sequential processing can be slow and result in a high
proportion of invalid molecules with substantial post-processing needed in
order to satisfy the principles of stoichiometry. To address these issues, we
propose a representation-first approach to molecular graph generation. We
guide the latent representation of an autoencoder by capturing graph structure
information with the geometric scattering transform and apply penalties that
structure the representation also by molecular properties. We show that this
highly structured latent space can be directly used for molecular graph
generation by the use of a GAN. We demonstrate that our architecture learns
meaningful representations of drug datasets and provides a platform for goal-
directed drug synthesis.

_Keywords_ Drug Discovery $\cdot$ Molecular Graph Generation $\cdot$ Geometric
Scattering

##  1 Introduction

Recently there has been a great deal of interest in developing neural networks
for graph-structured data. However, the vast majority of the literature on
graph neural networks and variants has focused on node embeddings and node
classification. Because of this, most GNNs are designed to produce features
for nodes based on neighborhood aggregation, which gives only localized
information. A smaller subset of this literature has focused on graph
classification. These networks have recently proposed using longer-range
information via skip connections or attention mechanisms (Abu-El-Haija et al.,
2019; Yun et al., 2019). However, there has been relatively little attention
given to graph generation, with the most common approaches being somewhat
cumbersome methods of sequential or reinforcement-learning based generation.
Here, we seek to remedy this by focusing on a representation-first approach to
graph generation that we call GRASSY (GRAph Scattering SYnthesis network).
GRASSY focuses on producing a latent space embedding of molecular graphs that
is organized both by molecular structure and physicochemical properties. It is
then trained adversarially to generate molecules with desirable properties
directly from this latent space.

The GRASSY framework makes use of the geometric scattering transform in order
to learn a rich representation of the graph. This transform discretizes the
original scattering transform of Mallat (2012) and uses multiscale graph
diffusion wavelets to form globally-contextualized descriptions of each graph.
Notably, the version of the scattering transform we use here collects
statistical moments of multi-scaled diffusion wavelet coefficients through
global summation. Therefore, the representation it produces is fully
permutation-invariant, and the number of moments we obtain does not depend on
the size of the original graph.

After computing the scattering transform, GRASSY reduces the dimensionality of
the resulting space with an autoencoder which is penalized by a reconstruction
penalty and a property-prediction penalty. This yields a highly structured
latent space, which we may sample from in order to generate scattering
coefficients corresponding to molecular graphs with desirable properties. To
complete the graph generation process, GRASSY utilizes an adversarial
framework which produces molecular graphs directly from the latent space.
Importantly, we note that GRASSY does not use a sequential process or
reinforcement-learning-based method for molecular synthesis but rather
immediately generates molecules based on the organization of the latent space.

In summary, the key components of GRASSY include:

  * •

A geometric scattering network to generate multiscale descriptions of graphs

  * •

A regularized autoencoder for producing representations of molecules in a
structured latent space

  * •

An adversarial molecular generation network.

We show the utility of GRASSY on two datasets described in detail in Section
5. 1) ZINC, a dataset of drug-like molecules with several properties of each
molecule, 2) BindingDB, a database of drug-target interactions. We show that
GRASSY learns the latent space of several tranches of ZINC and produces drug-
like molecules. We also show that it can learn molecules with binding
affinities to specific targets and generate molecules in this space as well.

##  2 Related work on Molecule Generation via Graph Neural Nets

A variety of notable approaches have been made to tackle the difficult problem
of molecular graph generation. GraphAF (Shi et al., 2020) leverages a flow-
based autoregressive model, which formulates graphs as a sequential decision
process and then applies reinforcement learning to generate graphs with
specified properties. MolGAN (Cao and Kipf, 2018) takes a different approach,
using a GAN-based architecture. From normally distributed samples, it
generates graphs using a discriminator which is based on the Graph
Convolutional Network (GCN) of Kipf and Welling (2017). This discriminator
also penalizes graphs based on a reward network, which encourages the
generated graphs to have specific, desired properties. Similarly, LGGAN (Fan
and Huang, 2021) uses a GAN-based architecture, and also uses a GCN as a means
to discriminate between real and fake graphs. In a different approach, Gómez-
Bombarelli et al. (2018), generate graphs by creating an auto-encoder that
encodes and decodes SMILES strings, which allowed them to sample points from
the latent space and generate new SMILES strings.

##  3 Preliminaries and Background

###  3.1 Graph diffusion

The geometric scattering transform introduced in the following subsection uses
multi-scale diffusion wavelets which are inspired, in part, by methods from
high-dimensional data analysis (Coifman and Maggioni, 2006). In many
applications, one is given a data set $\\{\mathbf{x}_{i}\\}_{i=1}^{n}$
contained in very high-dimensional Euclidean space $\mathbb{R}^{N}$. The
excessively large dimension of such a data set makes it hard to analyze and
expensive to store or process. Fortunately, in many applications, the data has
an intrinsic lower-dimensional structure and can be modeled as lying along a
$d$-dimensional Riemannian manifold for some $d\ll N$. To exploit this low-
dimensional structure, one aims to construct a graph whose geometry models the
underlying manifold. A popular method for doing this is to let the data points
$\mathbf{x}_{i}$ be the vertices of the graph and to define the affinity
between vertices using a predefined kernel function. A common choice is to
define a weighted edge between vertices $i$ and $j$, $i\neq j$ with weight
given by $W_{i,j}\coloneqq
e^{-\|\mathbf{x}_{i}-\mathbf{x}_{j}\|_{2}^{2}/\epsilon}$ for a suitably chosen
parameter $\epsilon$. One may view the $W$ as the weighted adjacency matrix of
a weighted graph $G=(V,E,W)$. (Here, we set $W_{i,i}=0$ so that the graph has
no self-loops.)

The matrix $W$ encodes _local_ information about the graph $G$. In order to
capture information about the graphs global geometry, we introduce a lazy
random walk matrix

$P\coloneqq\frac{1}{2}(I_{n}+WD^{-1}),$ (1)

where $I_{n}$ is the $n\times n$ identity matrix and $D$ is the diagonal
degree matrix whose non-zero entries are defined by
$D_{i,i}=\sum_{j=1}^{n}W_{i,j}$. Raising the matrix $P$ to different powers
captures information about the graph at various degrees of resolution. For
example, $P^{2}$ encodes information within two-step neighborhoods of each
vertex whereas $P$ raised to a very large power captures global averages. The
popular diffusion maps algorithm, introduced in Coifman and Lafon (2006), is
based on the eigendecomposition of this matrix $P$ and its powers. It has been
shown to be useful for many applications including data visualization (Moon et
al., 2019), denoising (van Dijk et al., 2017), and trajectory analysis
(Haghverdi et al., 2016).

###  3.2 Geometric Scattering

The scattering transform, originally introduced in Mallat (2012) for Euclidean
data is a wavelet-based, feed-forward network which produces a latent-space
representation of an input via an alternating sequence of convolutions and
nonlinearities. The architecture of the scattering transform is similar to a
convolutional neural network but uses predesigned filters rather than filters
learned from training data. Inspired by the rise of graph neural networks,
several works (Zou and Lerman, 2020; Gama et al., 2018; Gao et al., 2019) have
adapted the scattering transform to the graph setting and analyzed the
stability of the resulting networks (Perlmutter et al., 2019; Gama et al.,
2019). Similar to Mallat (2012), the original formulations of the graph
scattering transforms were fully designed networks using dyadic wavelets.
However, subsequent work has incorporated learning via cross-channel
convolutions (Min et al., 2020), attention mechanisms (Min et al., 2021), or
by replacing dyadic-scales with scales learned from data (Tong et al., 2021).
Most closely to our work, (Zou and Lerman, 2019) and (Castro et al., 2020)
have shown that the scattering transform can be incorporated into an encoder-
decoder type network. Here we utilize this framework for molecular graph
generation.

###  3.3 Scattering moments for graph structure representation

The primary focus of Gao et al. (2019) was applying the graph scattering
transform to graph classification. There, the authors construct the scattering
transform as an alternating cascade of wavelet transforms and pointwise
nonlinearities. In particular, they use diffusion wavelets constructed from
the matrix $P$ defined in equation 1 raised to dydadic powers. Specifically,
for $J\geq 1,$ they define

$\Psi_{0}\coloneqq I_{n}-P\,,\quad\Psi_{j}\coloneqq
P^{2^{j-1}}-P^{2^{j}}=P^{2^{j-1}}(I_{n}-P^{2^{j-1}})\,,\quad j\geq 1\,.$ (2)

Different powers of $P$ capture information about the graph at different
scales. In particular, because of the larger values of $j$, the scattering
transform is able to capture both local and long-range information in a single
layer. This is in stark contrast to many popular graph neural networks such as
GCN (Kipf and Welling, 2017) in which the convolutions are purely local
averaging operations.

Given these diffusion wavelets, first and second-order scattering coefficients
are defined by

$S\mathbf{x}(j,q)\coloneqq\sum_{l=1}^{n}|\mathbf{\Psi}_{j}\mathbf{x}(v_{l})|^{q},\quad\text{and}\quad
S\mathbf{x}(j,j^{\prime},q)\coloneqq\sum_{l=1}^{n}|\mathbf{\Psi}_{j^{\prime}}|\mathbf{\Psi}_{j}\mathbf{x}(v_{l})||^{q}.$
(3)

for $1\leq j,j^{\prime}\leq J$ and $1\leq q\leq Q.$ The authors also use
zeroth-order coefficients which are simply defined by
$S(q)\coloneqq\|\mathbf{x}\|_{q}^{q}.$ The graph scattering transform is then
defined by concatenating all of these coefficients:

$S\mathbf{x}=\\{S\mathbf{x}(q),S\mathbf{x}(j,q),S\mathbf{x}(j,j^{\prime},q),1\leq
j,j^{\prime}\leq J,1\leq q\leq Q\\}.$ (4)

We note that since these coefficients are defined via a global summation they
are fully invariant to permutations of the vertices. Moreover, the number of
coefficients does not depend on the size of the graph. This allows one to
apply the graph scattering transform to data sets consisting of many graphs
where each graph may have a different number of vertices.

###  3.4 Geometric Scattering Autoencoder

In Castro et al. (2020), the authors proposed to autoencoder scattering
transforms of folds of an single non-coding RNA molecules in order to infer
potential folding trajectories from latent space visualization. They used a
fixed three-scale scattering transform to obtain multi-level wavelet
coefficients at each node which they concatenated and used as input to a
variational autoencoder network. They showed that this framework called Graph
Scattering Autoencoder (GSAE) is able to preserve similarity structure in the
folds, as well as fold energy, in a low-dimensional embedding. The authors
also proposed a scattering inversion network (SIN), which is an autoencoder
whose middle layer is an adjacency matrix. The SIN purports to go back from
the scattering coefficients to an adjacency matrix. However, the SIN network
only produces meaningful results with very strong initializations or guided
searches around a small area of the graph space, mainly used to interpolate
between two adjacent folds of the same RNA molecule.

Unlike GSAE, our primary aim is to generate novel molecules of different sizes
from the embedded space of a neural network trained on a variety of drug-like
molecules. For this purpose, we build upon and expand GSAE in several ways. 1)
We use scattering moments, i.e., aggregation of scattering transform
coefficients which allows us to handle graphs of different sizes. 2) We use a
GAN operates directly on the learned latent space. This molecular generation
network is trained with an adversarial loss as well as a latent interpolation
loss that obviates the need for both the SIN and the variational training in
GSAE. 3) We utilize a differentiable scattering transform Tong et al. (2021)
such that the scales of scattering are not fixed, but instead learned from the
data. In the next section, we detail features of our network which we call
GRASSY.

##  4 Methods

###  4.1 Problem Formulation

Our main goal is to find a latent space representation that is smooth with
respect to various molecular properties as well as graph edit distance and to
use such a representation to generate well-formed molecular graphs. More
formally, given molecular graphs
$G_{1}=\\{E_{1},V_{1}\\},G_{2}=\\{E_{2},V_{2}\\}$, with molecular properties
$p(G_{1})$, $p(G_{2})$, we seek a embedding $f$, and a pseudo-inversion
$f^{\prime}$ such that:

  * •

If $\|f(G_{1})-f(G_{2})\|_{2}\leq\epsilon_{e}$, then
$\text{dist}(G_{1},G_{2})<\nu$, i.e., close graphs have close embedded
distances.

  * •

If $\|f(G_{1})-f(G_{2})\|_{2}\leq\epsilon_{p}$, then
$\text{dist}(p(G_{1}),p(G_{2}))<\mu$, i.e., close graphs have similar
molecular properties.

  * •

$\text{dist}(f^{\prime}(f(G_{1})+e),G_{1})\leq\epsilon_{g}$, where $e\sim
N(0,\sigma)$, i.e. a graph generated by pseudo-inversion after perturbation of
embedding $f(G_{1})$ in the latent space, is similar to the original graph.

  * •

$\text{dist}(p(f^{\prime}(f(G_{1})+e)),p(G_{1}))\leq\epsilon$, where $e\sim
N(0,\sigma)$, i.e., a graph generated by pseudo-inversion after perturbation
of embedding $f(G_{1})$ in the latent space has similar molecular properties
as the original graph.

Typically, molecular graphs are not scale-free and have complex connectivity
structure. Therefore, one cannot satisfy these goals by solely searching in
the adjacency matrix space. Indeed, removing a single edge can leave the graph
with vastly different connectivity structure and molecular properties.
Therefore, we seek an alternative latent space created via neural network
transformations that is continuous with respect to graph structure and
property.

Figure 1: Overview of the GRASSY architecture including the 1) geometric
scattering moments $\mathbf{S}$ computed on the input graph, passed to an 2)
encoder network $E$ whose latent space is regularized for property prediction
and for graph generation and trained for reconstruction via the 3) decoder
$D$, and the 4) Molecular generation network trained with GAN and
interpolation losses.

###  4.2 Learned scattering moments

Our proposed GRASSY framework uses the graph scattering transform to initially
find a high dimensional embedding of the data. For each node-labelled
molecular graph, we collect a set of signals associated with each label
$\mathbf{x}_{L_{1}},\mathbf{x}_{L_{2}},\ldots\mathbf{x}_{L_{\ell}}$.
Specifically, we define $\mathbf{x}_{L_{i}}(v_{j})=1$ if the vertex $v_{j}$
has label $L_{i}$ and $\mathbf{x}_{L_{i}}(v_{j})=0$ otherwise. We then perform
multiscale wavelet transforms on each of these signals using a learnable
scattering framework (Tong et al., 2021), and collect statistical moments by
summing over the vertices (see Eqns. 3 and 4). We let
$\mathbf{S}=\mathbf{S}(G)$ denote the collection of all scattering moments
associated to a graph $G$ and use these moments as the input into the
regularized autoencoder described below. Importantly, we note that while
collecting these statistical moments, we apply a global summation and
therefore the dimension of the the scattering embedding does not depend on the
size of the graph.

###  4.3 Autoencoder Design

We feed the scattering moments $\mathbf{S}$ into a regularized autoencoder
$F(E(X))$, which is penalized by two losses:

  * •

The reconstruction loss that penalizes for errors in the reconstruction of
scattering moments, i.e., $\mathcal{L}_{r}=\|\mathbf{S}-F(E(\mathbf{S}))\|$,
where $E$ is the encoder and $F$ is the decoder.

  * •

A regression loss which penalizes the failure of a property prediction network
$R$ to predict a physicochemical properties $p$ of a given molecule from its
latent representation, i.e., $\mathcal{L}_{p}=\|p-R(E(\mathbf{S}))\|$.

Note that $E(\mathbf{S}(G))$ is our proposed mapping function $f$ that
approximates properties described in the problem setup. The molecule
generation network, $M$, which we describe in the following subsection, is our
proposed pseudo-inversion $f^{\prime}$ in the problem setup.

###  4.4 Molecule Generation

If $\mathbf{S}(G_{i})$ and $\mathbf{S}(G_{j})$ are the scattering moments
corresponding to two graphs in our data set, we let
$z_{i}=E(\mathbf{S}(G_{i}))$ and $z_{j}=E(\mathbf{S}(G_{i}))$ and consider the
trajectory $z_{i\to j}(\alpha)\coloneqq(1-\alpha)z_{i}+\alpha z_{j}$. We let
$M$ be a multi-layer perceptron which inputs scattering coefficients and
outputs an $n\times n$ adjacency matrix and define $\hat{W}_{i\to
j}(\alpha)\coloneqq M(z_{i\to j}(\alpha))$. As alluded to earlier, we wish to
consider graphs of different sizes, and therefore we will take $n$ to be the
size of the largest graph in the data set. Smaller adjacency matrices will be
extended to be $n\times n$ via zero padding.

Inspired by the Autoencoder Adversarial Interpolation approach applied to
images in Oring et al. (2020), we train $M$ and our discriminator $D$ using
three losses adapted to the graph generation setting:

  * •

The adjacency matrix reconstruction loss,
$\mathcal{L}_{m}=\|W_{i}^{\text{pad}}-\hat{W}_{i}\|_{F}+\|W_{j}^{\text{pad}}-\hat{W}_{j}\|_{F}$,
where adjacency matrices $W_{i}$ and $W_{j}$ are padded with zero vectors up
to size $n\times n$.

  * •

An adversarial loss, $\mathcal{L}_{a}=\sum_{k=0}^{K}-\log(D(\hat{W}_{i\to
j}(k/K)))$, where discriminator, $D$, is a graph convolution network (GCN)
that outputs a scalar value.

  * •

A smoothness loss,
$\mathcal{L}_{s}=\sum_{k=0}^{K}\left\lVert\frac{\partial\hat{\mathbf{S}}_{i\to
j}(k/K)}{\partial\alpha}\right\rVert^{2}$, calculated by taking derivative of
scattering moments produced by the decoder, $\hat{\mathbf{S}}_{i\to
j}(\alpha)=F(z_{i\to j}(\alpha))$.

These losses aid in the production of valid molecular adjacency matrices which
have similar structure to nearby points.

Figure 2: Learned geometric scattering wavelet coefficients for 3 ZINC tranches. Figure 3: Latent representations of molecules in the ZINC dataset, visualized using PHATE, colored by value of physicochemical properties. Figure 4: Latent representations obtained using ablations of the GRASSY model trained on the BBAB tranche of the ZINC dataset, color-coded by QED for different models. Table 1: Molecular property prediction error Model | Tranche | Molecular Properties
---|---|---
|  | QED | TPSA | Mol. Wt. (HA) | # Rings
| BBAB | $\mathbf{0.1677\pm 0.13}$ | $\mathbf{13.20\pm 11}$ | $\mathbf{22.83\pm 18}$ | $\mathbf{0.6277\pm 0.52}$
GRASSY-AE + REGR | FBAB | 0.2090 $\pm$ 0.16 | 17.15 $\pm$ 15 | $\mathbf{32.89\pm 24}$ | $\mathbf{0.7122\pm 0.57}$
| JBCD | $\mathbf{0.1941\pm 0.15}$ | 15.18 $\pm$ 12 | $\mathbf{23.30\pm 22}$ | $\mathbf{0.6604\pm 0.46}$
| BBAB | 0.2063 $\pm$ 0.16 | 14.91 $\pm$ 13 | 56.67 $\pm$ 21 | 0.6302 $\pm$ 0.52
GRASSY-VAE + REGR | FBAB | $\mathbf{0.2041\pm 0.15}$ | $\mathbf{16.94\pm 14}$ | 36.17 $\pm$ 23 | 0.7130 $\pm$ 0.54
| JBCD | 0.2226 $\pm$ 0.17 | $\mathbf{14.74\pm 12}$ | 34.54 $\pm$ 35 | 0.6643 $\pm$ 0.54
| BBAB | 0.3947 $\pm$ 0.28 | 78.59 $\pm$ 20 | 210.0 $\pm$ 15 | 1.489 $\pm$ 0.92
GRASSY-AE | FBAB | 1.401 $\pm$ 0.97 | 108.5 $\pm$ 22 | 340.4 $\pm$ 8.6 | 6.528 $\pm$ 1.1
| JBCD | 0.4232 $\pm$ 0.26 | 126.9 $\pm$ 26 | 444.5 $\pm$ 14 | 3.152 $\pm$ 0.74
| BBAB | 0.7413 $\pm$ 1.3 | 78.53 $\pm$ 20 | 209.4 $\pm$ 15 | 1.364 $\pm$ 0.93
GRASSY-VAE | FBAB | 0.9313 $\pm$ 0.33 | 108.5 $\pm$ 22 | 339.5 $\pm$ 8.6 | 3.038 $\pm$ 0.96
| JBCD | 2.885 $\pm$ 0.92 | 127.8 $\pm$ 26 | 444.7 $\pm$ 14 | 2.974 $\pm$ 0.86
Table 2: Smoothness of physicochemical properties in latent space Model | Tranche | Molecular Properties
---|---|---
|  | QED | TPSA | Mol. Wt. (HA) | # Rings
| BBAB | 0.0952 | 0.1716 | 0.0144 | 0.7153
GRASSY-AE + REGR | FBAB | 0.1324 | 0.1353 | 0.0021 | 0.3172
| JBCD | 0.1516 | 0.0645 | 0.0028 | 0.0928
| BBAB | 0.0829 | 0.1785 | 0.0134 | 0.6415
GRASSY-VAE + REGR | FBAB | 0.1298 | 0.1333 | 0.0020 | 0.3016
| JBCD | 0.1225 | 0.0541 | 0.0031 | 0.0805
| BBAB | 0.0692 | 0.1467 | 0.0138 | 0.6114
GRASSY-AE | FBAB | 0.1203 | 0.1332 | 0.0023 | 0.3413
| JBCD | 0.1383 | 0.0697 | 0.0037 | 0.0837
| BBAB | 0.0696 | 0.1442 | 0.0147 | 0.6266
GRASSY-VAE | FBAB | 0.1201 | 0.1423 | 0.0022 | 0.3593
| JBCD | 0.1376 | 0.0713 | 0.0036 | 0.0800
Table 3: Fraction of graphs generated with molecule-like structure ZINC | # Atoms | Validity | Models
---|---|---|---
Tranche | Min. | Max. | Threshold | GRASSY | GSAE | GraphAF | MolGAN ($\lambda=0$)
BBAB | 8 | 18 | 5 | 0.86 | 0.22 | 0.79 | 0.32
FBAB | 16 | 27 | 15 | 0.94 | 0.17 | 0.76 | 0.46
JBCD | 28 | 36 | 25 | 0.73 | 0.09 | 0.54 | 0.41

##  5 Results

We trained GRASSY on large datasets of drugs and drug-like molecules from two
databases, ZINC (Irwin and Shoichet, 2005) and BindingDB (Gilson et al.,
2016). ZINC contains drug-like molecules organized into tranches by molecular
weight (abbreviated mol. wt.), solubility (logP value), reactivity and
commercial availability. BindingDB is a drug-target interaction (DTI)
database, containing pairs of molecules and proteins with known binding
affinity. We used three tranches from the ZINC database, namely FBAB, BBAB,
and JBCD, which are organized by molecular weight. Molecules in BBAB range
from 200 to 250 Daltons, molecules in FBAB range from 350 to 375 Daltons, and
molecules from JBCD range from 450 to 500 Daltons. We sampled 2500 molecules
from each tranche to use as training data and used the ChEMBL database (Davies
et al., 2015) to obtain 10 physicochemical for each molecule. These properties
include quantitative estimate of drug-likeliness (QED) introduced in Bickerton
et al. (2012), total polar surface area (TPSA), general chemical descriptors
(mol. wt. and heavy atom mol. wt.), topochemical descriptors (BalabanJ,
BertzCT, HallKierAlpha), and Lipinski parameters (number of hydrogen bond
donors, number of hydrogen bond acceptors, and ring count). We also trained on
known inhibitors of two protein targets on BindingDB, henceforth referred to
by their UniProtKB identifiers. These targets are P14416, a D(2) dopamine
receptor, and P00918, Carbonic anhydrase 2. All BindingDB results are shown in
the Appendix.

To calculate the scattering moments for an individual molecular graph, we
labeled each node by atom type and used the signals
$\mathbf{x}_{L_{1}},\ldots,\mathbf{x}_{L_{\ell}}$ as described in Section 4.1.
We set the number of moments $Q=2$ and the number of scales $J=4$. Since our
tranches ranged in molecular weight, we decided to use a learnable version of
graph scattering proposed in Tong et al. (2021) to individually learn
diffusion scales for each tranche rather than just using dyadic powers
$2^{j}$. The learned wavelet coefficients for each of the three ZINC tranches
can be seen in Figure 2. We then passed the scattering moments into our
autoencoder, which applied regression penalties with respect to 10
physicochemical properties in latent space.

We adjusted the learning rates so that our network was properly able to learn
the input data on each tranche, and we used an early stopping mechanism that
monitored the validation loss to prevent the models from over-fitting our
data. Figure 3 shows the latent space representations of each tranche trained
on GRASSY, where the latent space is colored by four different properties,
each of which was part of the latent space regression task.

We compare GRASSY to three state-of-the art frameworks for molecular graph
generation: GraphAF (Shi et al., 2020), MolGAN (Cao and Kipf, 2018) and GSAE
(Castro et al., 2020). For GraphAF, we trained the TorchDrug implementation
using a custom dataloader but without any architectural modifications. For
MolGAN, we set the hyperparameter $\lambda$ to $0$ (full RL version). We also
compared with experimented with ablations of the GRASSY architecture, turning
off and on the regression penalty (with or without REG), as well as toggling
variational vs vanilla autoencoder training (GRASSY-VAE vs. GRASSY-AE).

We tested these GRASSY variations according to three criteria:

  * •

Accuracy of physicochemical prediction in latent space across each of these
models, measuring by absolute error. (See Tables 1 and 4.)

  * •

Smoothness of the physicochemical properties in latent space, which we
calculated from the graph Laplacian, $L$, of the diffusion potential graph
obtained from the PHATE transform of the latent space embeddings. We
calculated smoothness by $s\coloneqq\frac{p^{T}Lp}{p^{T}p}$, where $p$ is a
vector containing properties of the points embedded. (See Tables 2 and 5)

  * •

Fraction of valid molecules generated. (See Table 3)

Tables 2 and 5 show smoothness, as described above, of physicochemical
properties in the latent space of various versions of our architecture. We
have made bold values representing the highest smoothness in the latent space
for each tranch (or protein target for BindingDB) trained on each property,
across the various architectures. We can see that there is no version of our
architecture that significantly outperforms the others across the board.
Though, in general, the Figure 4 shows that each version of the GRASSY
produces a smooth latent space with respect to the physicochemical properties
listed.

Tables 1 and 4 show the molecular property prediction errors across different
versions of our architecture on four different physicochemical properties. On
both tables, we made bold the lowest error values for each property, across
the various versions of GRASSY. Table 1 shows the prediction error across the
three different tranches of ZINC we trained, BBAB, FBAB, and JBCD. We can see
that GRASSY-AE + REGR significantly outperforms the other three versions of
our architecture with respect to property prediction in the latent space on
this dataset. Table 4 shows that GRASSY-VAE + REGR outperformed our other
architectures when trained on the BindingDB data, across protein targets
P14416 and P00918.

Molecule-like graphs were created by thresholding the adjacency matrix output
of the generator of GRASSY and identifying the largest connected component.
The resulting molecule was considered valid if it satisfied the following
criteria:

  * •

Number of vertices in the graph must exceed a “validity threshold", set
according to size of molecules in the training tranche. Specifically, single
atoms, diatomic and tri-atomic molecules are unlikely to function as drugs or
inhibit the activity of any protein target.

  * •

Any cycles/rings must not contain more than 10 vertices/atoms. In practice,
most drugs contain 5 or fewer rings (Aldeghi et al., 2014).

  * •

No vertex should have degree (covalent bonds) exceeding 5. Our data
predominantly contains hydrogen, carbon, nitrogen, oxygen, sulphur, chlorine
and fluorine atoms, none of which have enough valence electrons to bond with
more than 5 atoms simultaneously.

Table 3 shows the fraction of valid molecules generated by GRASSY, GSAE
(Castro et al., 2020), GraphAF (Shi et al., 2020) and
MolGAN111https://github.com/nicola-decao/MolGAN (full RL version with
hyperparameter $\lambda=0$ (Cao and Kipf, 2018), according to the criteria
above. The minimum and maximum number of atoms in the training molecules are
listed to justify the choice of the validity threshold used to mark small
generated molecules as invalid.

##  6 Conclusions

We have introduced GRASSY, a novel method for molecule generation. Given a
dataset of molecules represented as graphs, we first collect a sequence of
scattering moments for each graph. We then train a regularized autoencoder on
these scattering moments to produce a latent representation which respects the
physicochemical properties of each molecule. Finally, we produce new molecules
by interpolating in latent space an applying a GAN which is trained to produce
chemically valid molecules. In our experiments, we see that our network
produces a higher proportion of realistic molecules than other methods. We
also see that physicochemical properties of the molecules vary smoothly in our
latent space and therefore that our network can be used to predict such
properties.

## References

  * Abu-El-Haija et al. [2019] Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan.  Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing.  In _international conference on machine learning_ , pages 21–29. PMLR, 2019.
  * Yun et al. [2019] Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim.  Graph transformer networks.  _Advances in Neural Information Processing Systems_ , 32:11983–11993, 2019.
  * Mallat [2012] Stéphane Mallat.  Group invariant scattering.  _Communications on Pure and Applied Mathematics_ , 65(10):1331–1398, 2012.
  * Shi et al. [2020] Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang.  GraphAF: a Flow-based Autoregressive Model for Molecular Graph Generation.  _arXiv:2001.09382 [cs, stat]_ , February 2020.  URL http://arxiv.org/abs/2001.09382.  arXiv: 2001.09382.
  * Cao and Kipf [2018] Nicola De Cao and Thomas Kipf.  MolGAN: An implicit generative model for small molecular graphs.  _CoRR_ , abs/1805.11973, 2018.  URL http://arxiv.org/abs/1805.11973.
  * Kipf and Welling [2017] Thomas N. Kipf and Max Welling.  Semi-Supervised Classification with Graph Convolutional Networks.  _arXiv:1609.02907 [cs, stat]_ , February 2017.  URL http://arxiv.org/abs/1609.02907.  arXiv: 1609.02907.
  * Fan and Huang [2021] Shuangfei Fan and Bert Huang.  Labeled Graph Generative Adversarial Networks.  _arXiv:1906.03220 [cs, stat]_ , February 2021.  URL http://arxiv.org/abs/1906.03220.  arXiv: 1906.03220.
  * Gómez-Bombarelli et al. [2018] Rafael Gómez-Bombarelli, Jennifer N. Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D. Hirzel, Ryan P. Adams, and Alán Aspuru-Guzik.  Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules.  _ACS Central Science_ , 4(2):268–276, February 2018.  ISSN 2374-7943, 2374-7951.  doi:10.1021/acscentsci.7b00572.  URL https://pubs.acs.org/doi/10.1021/acscentsci.7b00572.
  * Coifman and Maggioni [2006] Ronald R. Coifman and Mauro Maggioni.  Diffusion wavelets.  _Applied and Computational Harmonic Analysis_ , 21(1):53–94, July 2006.  ISSN 1063-5203.  doi:10.1016/j.acha.2006.04.004.  URL https://www.sciencedirect.com/science/article/pii/S106352030600056X.
  * Coifman and Lafon [2006] Ronald R. Coifman and Stéphane Lafon.  Diffusion maps.  _Applied and Computational Harmonic Analysis_ , 21(1):5–30, 2006.  ISSN 1063-5203.  doi:https://doi.org/10.1016/j.acha.2006.04.006.  URL https://www.sciencedirect.com/science/article/pii/S1063520306000546.  Special Issue: Diffusion Maps and Wavelets.
  * Moon et al. [2019] Kevin R. Moon, David van Dijk, Zheng Wang, Scott Gigante, Daniel B. Burkhardt, William S. Chen, Kristina Yim, Antonia van den Elzen, Matthew J. Hirn, Ronald R. Coifman, Natalia B. Ivanova, Guy Wolf, and Smita Krishnaswamy.  Visualizing structure and transitions in high-dimensional biological data.  _Nature Biotechnology_ , 37(12):1482–1492, December 2019.
  * van Dijk et al. [2017] David van Dijk, Juozas Nainys, Roshan Sharma, Pooja Kaithail, Ambrose J Carr, Kevin R Moon, Linas Mazutis, Guy Wolf, Smita Krishnaswamy, and Dana Pe’er.  Magic: A diffusion-based imputation method reveals gene-gene interactions in single-cell rna-sequencing data.  _BioRxiv_ , page 111591, 2017.
  * Haghverdi et al. [2016] Laleh Haghverdi, Maren Büttner, F Alexander Wolf, Florian Buettner, and Fabian J Theis.  Diffusion pseudotime robustly reconstructs lineage branching.  _Nature methods_ , 13(10):845, 2016.
  * Zou and Lerman [2020] Dongmian Zou and Gilad Lerman.  Graph convolutional neural networks via scattering.  _Applied and Computational Harmonic Analysis_ , 49(3):1046–1074, 2020.
  * Gama et al. [2018] Fernando Gama, Alejandro Ribeiro, and Joan Bruna.  Diffusion scattering transforms on graphs.  _arXiv preprint arXiv:1806.08829_ , 2018.
  * Gao et al. [2019] Feng Gao, Guy Wolf, and Matthew Hirn.  Geometric scattering for graph data analysis.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2122–2131. PMLR, 09–15 Jun 2019.  URL https://proceedings.mlr.press/v97/gao19e.html.
  * Perlmutter et al. [2019] Michael Perlmutter, Feng Gao, Guy Wolf, and Matthew Hirn.  Understanding Graph Neural Networks with Asymmetric Geometric Scattering Transforms.  _arXiv:1911.06253 [cs, stat]_ , November 2019.  URL http://arxiv.org/abs/1911.06253.  arXiv: 1911.06253.
  * Gama et al. [2019] Fernando Gama, Alejandro Ribeiro, and Joan Bruna.  Stability of graph scattering transforms.  _Advances in Neural Information Processing Systems_ , 32:8038–8048, 2019.
  * Min et al. [2020] Yimeng Min, Frederik Wenkel, and Guy Wolf.  Scattering GCN: Overcoming oversmoothness in graph convolutional networks.  _arXiv preprint arXiv:2003.08414_ , 2020.
  * Min et al. [2021] Yimeng Min, Frederik Wenkel, and Guy Wolf.  Geometric scattering attention networks.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 8518–8522. IEEE, 2021.
  * Tong et al. [2021] Alexander Tong, Frederik Wenkel, Kincaid MacDonald, Smita Krishnaswamy, and Guy Wolf.  Data-Driven Learning of Geometric Scattering Networks.  _arXiv:2010.02415 [cs, stat]_ , February 2021.  URL http://arxiv.org/abs/2010.02415.  arXiv: 2010.02415.
  * Zou and Lerman [2019] Dongmian Zou and Gilad Lerman.  Encoding robust representation for graph generation.  In _2019 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–9. IEEE, 2019.
  * Castro et al. [2020] Egbert Castro, Andrew Benz, Alexander Tong, Guy Wolf, and Smita Krishnaswamy.  Uncovering the folding landscape of RNA secondary structure using deep graph embeddings.  In _2020 IEEE International Conference on Big Data (Big Data)_ , pages 4519–4528, 2020.  doi:10.1109/BigData50022.2020.9378305.
  * Oring et al. [2020] Alon Oring, Zohar Yakhini, and Yacov Hel-Or.  Autoencoder Image Interpolation by Shaping the Latent Space.  _arXiv:2008.01487 [cs, stat]_ , October 2020.  URL http://arxiv.org/abs/2008.01487.  arXiv: 2008.01487.
  * Irwin and Shoichet [2005] John J. Irwin and Brian K. Shoichet.  ZINC – A Free Database of Commercially Available Compounds for Virtual Screening.  _Journal of chemical information and modeling_ , 45(1):177–182, 2005.  ISSN 1549-9596.  doi:10.1021/ci049714.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1360656/.
  * Gilson et al. [2016] Michael K. Gilson, Tiqing Liu, Michael Baitaluk, George Nicola, Linda Hwang, and Jenny Chong.  BindingDB in 2015: A public database for medicinal chemistry, computational chemistry and systems pharmacology.  _Nucleic Acids Research_ , 44(D1):D1045–1053, January 2016.  ISSN 1362-4962.  doi:10.1093/nar/gkv1072.
  * Davies et al. [2015] Mark Davies, Michał Nowotka, George Papadatos, Nathan Dedman, Anna Gaulton, Francis Atkinson, Louisa Bellis, and John P. Overington.  ChEMBL web services: streamlining access to drug discovery data and utilities.  _Nucleic Acids Research_ , 43(W1):W612–W620, July 2015.  ISSN 0305-1048.  doi:10.1093/nar/gkv352.  URL https://doi.org/10.1093/nar/gkv352.
  * Bickerton et al. [2012] G. Richard Bickerton, Gaia V. Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L. Hopkins.  Quantifying the chemical beauty of drugs.  _Nature Chemistry_ , 4(2):90–98, February 2012\.  ISSN 1755-4349.  doi:10.1038/nchem.1243.  URL https://www.nature.com/articles/nchem.1243.
  * Aldeghi et al. [2014] Matteo Aldeghi, Shipra Malhotra, David L Selwood, and Ah Wing Edith Chan.  Two- and Three-dimensional Rings in Drugs.  _Chemical Biology & Drug Design_, 83(4):450–461, January 2014.  ISSN 1747-0277.  doi:10.1111/cbdd.12260.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4233953/.

##  Appendix A Appendix

Figures and tables associated with the BindingDB dataset are included below:

Figure 5: Latent representations obtained using ablations of the GRASSY model trained on the BindingDB dataset, color-coded by QED. Table 4: Accuracy of physicochemical property prediction for BindingDB Model | Protein | Molecular Properties
---|---|---
| Target | QED | TPSA | Mol. Wt. (HA) | # Rings
GRASSY-AE + REGR | P14416 | 0.3072 $\pm$ 0.43 | 22.52 $\pm$ 30 | $\mathbf{49.51\pm 66}$ | $\mathbf{0.8053\pm 1.6}$
| P00918 | $\mathbf{0.2911\pm 0.22}$ | 35.50 $\pm$ 34 | 62.91 $\pm$ 71 | 0.8908 $\pm$ 0.75
GRASSY-VAE + REGR | P14416 | $\mathbf{0.2430\pm 0.20}$ | $\mathbf{18.99\pm 31}$ | 52.02 $\pm$ 66 | 0.9096 $\pm$ 1.9
| P00918 | 0.3395 $\pm$ 0.27 | $\mathbf{34.80\pm 33}$ | $\mathbf{56.72\pm 64}$ | $\mathbf{0.7981\pm 0.66}$
GRASSY-AE | P14416 | 0.5672 $\pm$ 0.83 | 56.90 $\pm$ 74 | 399.1 $\pm$ 196 | 3.533 $\pm$ 1.4
| P00918 | 0.6871 $\pm$ 0.63 | 116.8 $\pm$ 57 | 373.3 $\pm$ 151 | 2.483 $\pm$ 1.5
GRASSY-VAE | P14416 | 0.7300 $\pm$ 0.35 | 56.86 $\pm$ 74 | 399.7 $\pm$ 197 | 4.154 $\pm$ 1.3
| P00918 | 0.6001 $\pm$ 0.30 | 117.1 $\pm$ 57 | 372.3 $\pm$ 150 | 2.630 $\pm$ 1.4
Table 5: Smoothness of physicochemical properties in latent space for BindingDB Model | Protein | Molecular Properties
---|---|---
| Target | QED | TPSA | Mol. Wt. (HA) | # Rings
GRASSY-AE + REGR | P14416 | 0.1846 | 0.2813 | 0.0822 | 0.1494
| P00918 | 0.3786 | 0.4469 | 0.1611 | 0.5016
GRASSY-VAE + REGR | P14416 | 0.1746 | 0.3032 | 0.0716 | 0.1475
| P00918 | 0.3345 | 0.3991 | 0.1294 | 0.3926
GRASSY-AE | P14416 | 0.1811 | 0.2385 | 0.0995 | 0.1505
| P00918 | 0.2405 | 0.2436 | 0.1496 | 0.2954
GRASSY-VAE | P14416 | 0.2282 | 0.2688 | 0.1148 | 0.1818
| P00918 | 0.3685 | 0.4035 | 0.2368 | 0.4820