# Predicting small molecules solubility on endpoint devices using deep
ensemble neural networks

Mayk Caldas Ramos
Department of Chemical Engineering
University of Rochester
<EMAIL_ADDRESS>
Andrew D. White
Department of Chemical Engineering
University of Rochester
<EMAIL_ADDRESS>

###### Abstract

Aqueous solubility is a valuable yet challenging property to predict.
Computing solubility using first-principles methods requires accounting for
the competing effects of entropy and enthalpy, resulting in long computations
for relatively poor accuracy. Data-driven approaches, such as deep learning,
offer improved accuracy and computational efficiency but typically lack
uncertainty quantification. Additionally, ease of use remains a concern for
any computational technique, resulting in the sustained popularity of group-
based contribution methods. In this work, we addressed these problems with a
deep learning model with predictive uncertainty that runs on a static website
(without a server). This approach moves computing needs onto the website
visitor without requiring installation, removing the need to pay for and
maintain servers. Our model achieves satisfactory results in solubility
prediction. Furthermore, we demonstrate how to create molecular property
prediction models that balance uncertainty and ease of use. The code is
available at https://github.com/ur-whitelab/mol.dev, and the model is usable
at https://mol.dev.

_Keywords_ Solubility, Small Molecule, Deep Ensemble, Recurrent Neural Network

##  1 Introduction

Aqueous solubility measures the maximum quantity of matter that can be
dissolved in a given volume of water. It depends on several conditions, such
as temperature, pressure, pH, and the physicochemical properties of the
compound being solvated.[1] The solubility of molecules is essential in many
chemistry-related fields, including drug development[2, 3, 4, 5], protein
design[6], chemical[7, 8] and separation[9] processes. In drug development,
for instance, compounds with biological activity may not have enough
bioavailability due to inadequate aqueous solubility.

Solubility prediction is essential, driving the development of various
methods, from physics-based approaches — including first principles[10, 11],
semi-empirical equations[12, 13, 14], molecular dynamics (MD)[15, 16, 17, 18],
and quantum computations[19] — to empirical methods like quantitative
structure-property relationship (QSPR)[20, 21, 22, 23] and multiple linear
regression (MLR)[24, 25]. Despite their sophistication, physics-based models
often present complexity that limits accessibility to advanced users[26] and
do not guarantee higher accuracy than empirical methods[27]. Data-driven
models emerge as efficient alternatives, capable of outperforming physics-
based models[26]. However, achieving accurate and reliable solubility
predictions remains a significant challenge[26, 28].

To address the persistent issues of systematic bias and non-reproducibility in
aqueous solubility datasets, Llinàs et al.[29, 30] introduced two solubility
challenges featuring consistent data. The first challenge red participants
based on the root mean square error (RMSE) and the accuracy within a $\pm 0.5$
logS error range[31]. The second challenge revealed that despite the freedom
in method selection, all entries relied on QSPR or machine learning (ML)
techniques[32], yet did not achieve a notable improvement over the first
challenge[31]. These challenges highlighted the importance of data quality
over model selection for accurate solubility predictions.[32] Sorkun et al.
[28] further emphasized this by demonstrating how data quality assessments on
subsets of the AqSolDB[1] significantly impacted model performance[28].

McDonagh et al. [27] demonstrated that cheminformatic methods surpass first
principle theoretical calculations in calculating solubilization free
energies, highlighting the superior accuracy of cheminformatics and the
efficacy of Random Forest models, evidenced by an RMSE of 0.93 using Llinàs’
first dataset[29]. Data-driven approaches, particularly feature-based models,
have contributed to accurate aqueous solubility prediction. Delaney [25] used
MLR to develop a model called Estimated SOLubility (ESOL) adjusted on a 2874
small organic molecules dataset with an average absolute error (AAE) of 0.83.
Comparable performance has been achieved using various methods including
MLR[24], Gaussian processes[33], undirected graph recurrent neural networks
(UG-RNN)[34], deep neural networks (DNN)[35], and random forests (RF)[36, 37].

Recently, transformers[38] models have been applied to compute solubility of
small molecules.[39, 40, 41, 42, 43]. Francoeur and Koes [41] developed the
SolTranNet, a transformers model trained on AqSolDB[1] solubility data.
Notably, this architecture results in an RMSE of only $0.278$ when trained and
evaluated on the original ESOL[25] dataset using random split. Nevertheless,
it shows an RMSE of $2.99$ when trained using the AqSolDB[1] and evaluated
using ESOL. It suggests that the molecules present in ESOL may have low
variability, meaning that samples in the test set are similar to samples in
the training set. Hence, models trained on the ESOL training set performed
excellently when evaluated on the ESOL test set.

Solubility models should ideally combine accuracy with ease of access. Thus, a
common idea is to use web servers to provide easier public access. However,
web servers demand continuous financial and time investments for maintenance,
leading to the eventual disappearance of some, despite having institutional or
government backing[44]. For instance, eight out of 89 web server tools
featured in the 2020 Nucleic Acids Research special web server issue were
offline by the end of 2022[45]. Moreover, computational demands can be
significant, with tools like RoseTTAFold[46] and ATB[47] requiring hours to
days for job completion, thus creating potential delays due to long queues and
wait times[48].

An alternative approach is to perform the computation directly on the user’s
device, removing the need for the server’s maintenance and cost. This method
allows hosting the website as a static file on platforms such as GitHub, with
potential archiving on the Internet Archive111https://archive.org/. We
explored this approach in Ansari and White [49] for bioinformatics. Our web
application implements a deep ensemble[50] recurrent neural network (RNN)
capable of extracting data directly from molecular string representations,
such as SMILES[51] or SELFIES[52], which can be easily quickly accessed.[53,
54]

The primary difficulty lies in the application’s dependence on the device’s
capabilities, which is crucial for smartphones with limited resources.
Balancing performance in low-resource settings, the use of transformer
models[38] becomes impractical due to their large size, incompatible with
smartphone memory and prolonged inference times. Additionally, our model
implements a deep ensemble to calibrate uncertainties, making the application
of transformers even more unfeasible. In contrast, using descriptors is an
easy way to convey physical information to the model and, consequently,
enables smaller models. However, descriptor computation is time-intensive. In
our tests, using PaDEL to compute descriptors for all molecules in AqSolDB
took roughly $\sim$ 20 hours. Furthermore, feature-based model development
requires specialized knowledge for feature selection[55], and is limited by
the regions of the chemical space these descriptors cover[56]. Even
application usage may need specialized data, as Kurotani et al. [37]
illustrate. RNNs present an alternative for property extraction directly from
string representations while allowing for adaptable computational resource
management.

In this work, we developed a front-end application using a JavaScript (JS)
implementation of TensorFlow framework[57]. Our application can be used to
predict the solubility of small molecules with uncertainty. To calibrate the
confidence of the prediction, our model implements a deep ensemble
approach[50] which allows reporting model uncertainty when reporting the
prediction. Our solution implements a deep ensemble of RNN models specially
designed to achieve satisfactory performance while being able to run in an
environment without strong computational resources. This application runs
locally on the user’s device and can be accessed at https://mol.dev/. Mol.dev
does not save data input for predictions in any way.

##  2 Methods

###  2.1 Dataset

The data used for training the models were obtained from AqSolDB[1]. This
database combined and curated data from 9 different aqueous solubility
datasets. The main concern in using a large, curated database is to avoid
problems with the generalizability of the model[58] and with the fidelity of
the data[59]. AqSolDB consists of aqueous solubility (LogS) values for 9982
unique molecules extended with 17 topological and physicochemical 2D
descriptors calculated by RDKit[60].

We augmented AqSolDB to 96,625 molecules using SMILES randomization[61, 62].
Each entry of AqSolDB was used to generate at most ten new unique randomized
SMILES strings. Training the model on multiple representations of the same
molecule improves its ability to learn the chemical space constraints of the
training set, as demonstrated in previous studies [61, 62]. Duplicates were
removed.

After shuffling, the augmented dataset was split into 80%/20% for the training
and test datasets, respectively. The curated datasets for the solubility
challenges[29, 32] were used as withheld validation data to evaluate the
model’s ability to predict solubility for unseen compounds. To refer to the
validation datasets, we labeled the first solubility challenge dataset as
"solubility challenge 1" and the two sets from the second solubility challenge
as "solubility challenge 2_1" and "solubility challenge 2_2", respectively.
Molecules in these three datasets were not found in train and test datasets.

###  2.2 Model architecture

Our model uses a deep ensemble approach as described by Lakshminarayanan et
al. [50]. This technique was selected due to its ability to estimate
prediction uncertainty, thus enhancing the predictive capability of our model.
The uncertainty of a model can be divided into two sources: aleatoric
uncertainty (AU) and epistemic uncertainty (EU).[63, 64] These uncertainties
quantify the intrinsic uncertainty inherent in data observations and the
disagreement among model estimations, respectively.[65]

Given a model that outputs two values – $\hat{\mu}_{m}$ and $\hat{\sigma}_{m}$
– that characterize a normal distribution
$\mathcal{N}(\hat{\mu}_{m},\hat{\sigma}_{m})$, a deep ensemble creates an
ensemble of $m$ models that can estimate prediction uncertainty. For a given
data point $\vec{x}$, the estimates for the ensemble predictions are computed
as follows:

$\hat{\mu}(\vec{x})=\frac{1}{N}\sum_{m}\hat{\mu}_{m}(\vec{x})$ (1)

$\hat{\sigma}^{2}_{ale}(\vec{x})=\frac{1}{N}\sum_{m}\hat{\sigma}^{2}_{m}(\vec{x})\
,\ \
\hat{\sigma}^{2}_{epi}(\vec{x})=\frac{1}{N}\sum_{m}\left(\hat{\mu}(\vec{x})-\hat{\mu}_{m}(\vec{x})\right)^{2}$
(2)

where $\hat{\sigma}^{2}_{ale}$ is AU, $\hat{\sigma}^{2}_{epi}$ is EU, N is the
ensemble size, and m indexes the models in the ensemble.

We used a deep neural network (DNN) implemented using Keras[66] and
TensorFlow[67] to build the deep ensemble. Our DNN model uses Self-referencing
embedded strings (SELFIES)[52] tokens as input. A pre-defined vocabulary was
created by analyzing all training data. Each unique SELFIES group was assigned
to a corresponding integer, yielding 273 distinct tokens. Simplified
molecular-input line-entry system (SMILES)[51] or SELFIES[52] molecule
representations are converted to tokens based on the pre-defined vocabulary.
Figure 1 illustrates the model architecture. The network can be divided into
three sections: ($i$) Embedding, ($ii$) bi-RNN, and ($iii$) fully connected
NN.

Figure 1: Scheme of the deep learning DNN. The molecule is input using the
SMILES or SELFIES representation. This representation is converted to a
tokenized input based on a vocabulary obtained using the training dataset. A
set of models represents the deep ensemble model. Each model consists of an
embed layer, two bidirectional RNN (bi-RNN) layers, a normalization layer, and
three fully connected layers being down-sized in three steps. Dropout layers
are present after the embed and after each fully connected layer during
training, but they were not represented in this scheme. Predictions of the
models in the ensemble are then aggregated.

The embedding layer converts a list of discrete tokens into a fixed-length
vector space. Working on a continuous vector space has two main advantages: it
uses a more compact representation, and semantically similar symbols can be
described closely in vector space. Our embedding layer has an input dimension
of 273 (vocabulary size) and an output dimension of 64.

Following the embedding layer, the data are fed into the bidirectional
Recurrent Neural Network (RNN) layer. We used two RNN layers, each containing
64 units. The effects of using Gated Recurrent Unit (GRU) or Long Short-Term
Memory (LSTM)[68] layers as the RNN layers were investigated (refer to Section
3.1). Using bi-RNN was motivated based on our previous work[49] in which LSTM
helped improve the model’s performance for predicting peptide properties using
its sequences. More details regarding RNN, LSTM, and GRU layers can be found
in Ref. 69.

The output from the bi-LSTM stack undergoes normalization via Layer
Normalization[70]. There is no agreement on why Layer Normalization improves
the model’s performance.[71, 72, 73, 74] The absence of a comprehensive
theoretical understanding of normalization effects hinders the evolution of
novel regularization schemes.[75] Despite the limited understanding, Layer
Normalization is employed due to its demonstrated effectiveness.[74]

After normalization, data is processed through three dense layers containing
32, 16, and 1 units, respectively. The 16-unit layer’s output goes to two
different 1-unit layers. One layer uses a linear function and the other uses a
softplus function, producing $\hat{\mu}_{m}$ and $\hat{\sigma}_{m}$,
respectively.

Negative log-likelihood loss $l$ was used to train the model. It is defined as
the probability of observing the label $y$ given the input $\vec{x}$:

$l(\vec{x},y)=\frac{log(\hat{\sigma}^{2}_{m}(\vec{x}))}{2}+\frac{\left(y-\hat{\mu}_{m}(\vec{x})\right)^{2}}{2\hat{\sigma}^{2}_{m}(\vec{x})}$
(3)

During the training phase, dropout layers with 0.35 dropout rate were
incorporated after the embedding and each dense layer to mitigate over-
fitting.[76] Models were trained using the Adam[77] optimizer with a fixed
learning rate of 0.0001 and default values for $\beta_{1}$ and $\beta_{2}$
(0.9 and 0.999, respectively).

Our model employs adversarial training, following the approach proposed by
Lakshminarayanan et al. [50] to improve the robustness of our model
predictions. Because the input for our model is a discrete sequence, we
generate adversarial examples by modifying the embedded representation of the
input data. Each iteration in the training phase consists of first computing
the loss using Equation 3 and a second step with a new input
$\vec{x}^{\prime}$ to smooth the model’s prediction:

$\vec{x}^{\prime}=\vec{x}+\epsilon\textrm{sign}(\nabla_{x}l(\vec{x},y))$ (4)

where $\epsilon$ is the strength of the adversarial perturbation.

Details of the model performance, limitations, training data, ethical
considerations, and caveats are available as a model card[78] at
http://mol.dev/.

##  3 Results

In order to evaluate the performance of our model using deep ensembles, two
baseline models were created: ($i$) an XGBoost Random Forest (RF) model using
the 17 descriptors available on AqSolDB plus 1809 molecular descriptors
calculated by PaDELPy, a python wrapper for the PaDEL-Descriptor[79] software,
and ($ii$) a model with the same architecture used on our deep ensemble using
RMSE as the loss function and no ensemble (referred to as DNN). RFs are the
state-of-the-art (SOTA) of solubility prediction. We used this baseline as a
comparison to prove that our model is able to achieve SOTA performance using
only molecular string representations. In addition, we evaluate the effects of
($i$) the bi-RNN layer (either GRU or LSTM), ($ii$) using an augmented dataset
to train, ($iii$) the adversarial training, and ($iv$) the ensemble size in
the model’s performance. Table 1 shows the performance of each one of our
trained models.

| Solubility Challenge 1 | Solubility Challenge 2_1 | Solubility Challenge 2_2
---|---|---|---
Model | RMSE | MAE | r | RMSE | MAE | r | RMSE | MAE | r
RF | 1.121 | 0.914 | 0.547 | 0.950 | 0.727 | 0.725 | 1.205 | 1.002 | 0.840
DNN | 1.540 | 1.214 | 0.433 | 1.315 | 1.035 | 0.651 | 1.879 | 1.381 | 0.736
DNNAug | 1.261 | 1.007 | 0.453 | 1.371 | 1.085 | 0.453 | 2.189 | 1.710 | 0.386
kde4GRU | 1.610 | 1.145 | 0.462 | 1.413 | 1.114 | 0.604 | 1.488 | 1.220 | 0.704
kde4LSTM | 1.554 | 1.191 | 0.507 | 1.469 | 1.188 | 0.650 | 1.523 | 1.161 | 0.706
kde4GRU-NoAdv | 1.729 | 1.348 | 0.525 | 1.483 | 1.235 | 0.622 | 1.954 | 1.599 | 0.517
kde4LSTM-NoAdv | 1.425 | 1.114 | 0.505 | 1.258 | 0.972 | 0.610 | 1.719 | 1.439 | 0.609
kde4${}^{GRU}_{Aug}$ | 1.329 | 1.148 | 0.426 | 1.354 | 1.157 | 0.674 | 1.626 | 1.340 | 0.623
kde4${}^{LSTM}_{Aug}$ | 1.273 | 0.984 | 0.473 | 1.137 | 0.932 | 0.639 | 1.511 | 1.128 | 0.717
kde8${}^{LSTM}_{Aug}$ | 1.247 | 0.984 | 0.542 | 1.044 | 0.846 | 0.701 | 1.418 | 1.118 | 0.729
kde10${}^{LSTM}_{Aug}$-NoAdv | 1.689 | 1.437 | 0.471 | 1.451 | 1.238 | 0.676 | 1.599 | 1.405 | 0.699
kde10${}^{LSTM}_{Aug}$ | 1.095 | 0.843 | 0.559 | 0.983 | 0.793 | 0.724 | 1.263 | 1.051 | 0.792
Table 1: Summary of the metrics for each trained model. We used the Root Mean
Squared Error(RMSE($\downarrow$)), Mean Absolute Error (MAE($\downarrow$)),
and Pearson correlation coefficient (r($\uparrow$)) to evaluate our models.
The arrows indicate the direction of improvement. Deep ensemble models are
referred to as “kde$N$”, where $N$ is the ensemble size. Baseline models using
random forest (RF) and the DNN model employed for deep ensemble (DNN) are also
displayed. DNN model was trained as described in Section 2. The models in
which data augmentation was used were subscribed with the flag $Aug$. A
superscript indicates if the bidirectional layer implements a $GRU$ or a
$LSTM$ layer. In addition, models trained not using adversarial perturbation
are flagged with “-NoAdv”. The columns show the results of each model
evaluated on each solubility challenge dataset. 2_1 represents the tight
dataset (set-1), while 2_2 represents the loose dataset (set-2) as described
in the original paper (See Ref. 30). $r$ stands for the Pearson correlation
coefficient. The best-performing model in each dataset is displayed in bold.

###  3.1 Gated layer

The most common RNN layers are the GRU and the LSTM. GRU layers use two gates,
reset and update, to control the cell’s internal state. On the other hand,
LSTM layers use three gates: forget, input, and output, with the same
objective. Available studies compare GRU and LSTM performances in RNNs for
different applications, for instance: forecasting[80], cryptocurrency[81, 82],
wind speed[83, 84], condition of a paper press[85], motive classification in
thematic apperception tests[86] and music and raw speech[87]. Nevertheless, it
is not clear which of those layers would perform better at a given task.

We trained models with four elements in the deep ensemble using GRU or LSTM.
Metrics can be found in Table 1; for an explanation of the naming syntax used
in this work, refer to Table 1 caption. Using LSTM resulted in a decrease in
RMSE and MAE and an increase in the correlation coefficient, indicating better
performance. For Solubility Challenges 1, 2_1, and 2_2, the
kde4${}^{GRU}_{Aug}$ model yielded RMSE values of 1.329, 1.354, and 1.626,
respectively, while the kde4${}^{LSTM}_{Aug}$ model achieved 1.273, 1.137, and
1.511, respectively. This trend was also observed for the models trained
without data augmentation (See Table 1). Considering that LSTM performs better
regarding this model and data, we will consider only bi-LSTM layers for
further discussion. Those results are in accordance with our previous work[49]
in which using LSTM helped improve the model’s performance.

###  3.2 Data augmentation

Our model is not intrinsically invariant with respect to the SELFIES
representation input. For instance, both “C(C(C1C(=C(C(=O)O1)O)O)O)O” and
“O=C1OC(C(O)CO)C(O)=C1O” are valid SMILES representations for the ascorbic
acid (See Figure 1) that will be encoded for different SELFIES tokens. Hence,
the model should learn to be invariant concerning changes in the string
representation during training. It can be achieved by augmenting the dataset
with SMILES randomization and training the model using different
representations with the same label. Therefore, the model can learn relations
in the chemical space instead of correlating the label with a specific
representation.[61] With this aim, we evaluated the effects of augmenting the
dataset by generating new randomized SMILES representations for each sample.

Augmenting the dataset had a significant impact on the metrics. It could be
seen improvements of $\sim 0.5$ in the RMSE when evaluating on challenge
datasets 1 and 2_1, and a gain of $\sim 0.2$ on 2_2 (See Table 1). Concerning
the first two datasets, augmenting data improved every model used in this
study. However, surprisingly, data augmentation led to a deprecation of the
DNN model on the solubility challenge 2_2 dataset. This behavior was not
further investigated.

###  3.3 Adversarial training

Using adversarial training improved performance in Lakshminarayanan et al.[50]
studies. Hence, they suggested that it should be used in future applications
of their deep learning algorithm. Thus, we tested the effects of adversarial
perturbation on training models with ensemble sizes of 4 and 10.

Comparing kde4LSTM-NoAdv and kde4LSTM, using adversarial training decreases
model performance. It can be seen in Table 1 that using adversarial
perturbation increased the RMSE from $1.425$ to $1.554$ and $1.258$ to $1.469$
in solubility challenges dataset 1 and 2_1, respectively. However, the RMSE
decreased from $1.719$ to $1.523$ in dataset 2_2. Using adversarial
perturbation affected our kde4LSTM’s performance by a change in RMSE of $\pm
0.2$.

The inconsistent performance improvement observed when using adversarial
training was further investigated with models in which the dataset was
augmented. Due to the lack of multiple string representations in the training
dataset, it is known that kde4LSTM may have generalization problems. A
generalization issue could direct the adversarial perturbation in a non-
physical direction because the model does not have complete knowledge about
the chemical representation space. This hypothesis is reinforced when we
compare kde10${}^{LSTM}_{Aug}$-NoAdv and kde10${}^{LSTM}_{Aug}$. When using
adversarial training on a model trained with an augmented dataset, the
performance improvement is more evident ($\sim 0.5$) and consistent for all
the test datasets.

###  3.4 Deep ensemble size

To investigate the effects of increasing the ensemble size, we trained models
with an ensemble of 4, 8, and 10 models. Given the previous results, these
models used LSTM as the bi-RNN layer and were trained on the augmented
dataset. Specifically for the solubility challenge 2_2, the most complex set
to predict, these models presented an RMSE of $1.511$, $1.418$, and $1.263$,
respectively. Therefore, increasing the ensemble size consistently improved
performance. We also observed this improvement on the other datasets (See
Table 1).

kde4${}^{LSTM}_{Aug}$

kde10${}^{LSTM}_{Aug}$

Figure 2: Density distribution of the aleatoric (AU) and epistemic variances
(EU) for the: ($i$) kde4${}^{LSTM}_{Aug}$ (top six panels) and ($ii$)
kde10${}^{LSTM}_{Aug}$ (bottom six panels). Increasing ensemble size reduces
the extent of the distribution’s tail, decreasing uncertainty about
predictions. However, the ensemble size does not noticeably affect the
distribution center.

Besides the immediate improvement in RMSE, increasing the ensemble size also
improves the uncertainty of the model. Figure 2 shows the density distribution
of the aleatoric variance and the epistemic variance (respectively related to
AU and EU) for kde4${}^{LSTM}_{Aug}$ (top 6 panels) and kde10${}^{LSTM}_{Aug}$
(bottom six panels).

The increase in ensemble size led to a decrease in both uncertainties. AU
distributions for the kde4${}^{LSTM}_{Aug}$ are centered around 4 logS2 ,
displaying a long tail that extends to values as high as 20 logS2 in the worst
case (solubility challenge 2_2). A similar trend is observed in EU
distributions. On the other hand, the kde10${}^{LSTM}_{Aug}$ model results in
narrower distributions. The mean of these distributions remains relatively
unchanged, but a noticeable reduction in the extent of their tails can be
observed. AU distribution ends in values around 10 logS2.

##  4 Discussion

| SolChal1 | SolChal2_1 | SolChal2_2 | ESOL
---|---|---|---|---
Model | RMSE | MAE | RMSE | MAE | RMSE | MAE | RMSE | MAE
RF | 1.121 | 0.914 | 0.950 | 0.727 | 1.205 | 1.002 |  |
DNN | 1.540 | 1.214 | 1.315 | 1.035 | 1.879 | 1.381 |  |
DNNAug | 1.261 | 1.007 | 1.371 | 1.085 | 2.189 | 1.710 |  |
kde4${}^{LSTM}_{Aug}$ | 1.273 | 0.984 | 1.137 | 0.932 | 1.511 | 1.128 | 1.397 | 1.131
kde8${}^{LSTM}_{Aug}$ | 1.247 | 0.984 | 1.044 | 0.846 | 1.418 | 1.118 | 1.676 | 1.339
kde10${}^{LSTM}_{Aug}$ | 1.095 | 0.843 | 0.983 | 0.793 | 1.263 | 1.051 | 1.316 | 1.089
Linear regression[25] |  |  |  |  |  |  | 0.75 |
UG-RNN[34] | 0.90 | 0.74 |  |  |  |  |  |
RF w/ CDF descriptors[27] | 0.93 |  |  |  |  |  |  |
RF w/ Morgan fingerprints[36] |  | 0.64 |  |  |  |  |  |
Consensus[88] | 0.91 |  |  |  |  |  |  |
GNN[89] | $\sim 1.10$ |  | 0.91 |  | 1.17 |  |  |
SolvBert[90] | 0.925 |  |  |  |  |  |  |
SolTranNeta[41] |  |  | 1.004 |  | 1.295 |  | 2.99 |
SMILES-BERTb[91] |  |  |  |  |  |  | 0.47 |
MolBERTb[40] |  |  |  |  |  |  | 0.531 |
RTb[42] |  |  |  |  |  |  | 0.73 |
MolFormerb[43] |  |  |  |  |  |  | 0.278 |
Table 2: Metrics for the best models found in the current study (upper
section) and for other state-of-art models available in the literature (lower
section). Values were taken from the cited references. Missing values stand
for entries that the cited authors did not study. SolChal columns stand for
the Solubility Challenges. 2_1 represents the tight dataset (set-1), while 2_2
represents the loose dataset (set-2) as described in the original paper (See
Ref. 30). The best-performing model in each dataset has its RMSE value in
bold.
a Has overlap between training and test sets.
b Pre-trained model was fine-tuned on ESOL.

After extensively investigating the hyperparameter selection, we compared our
model with available state-of-art models from the literature. Performance
metrics on the solubility challenge datasets can be found in Table 2. Parity
plots for our chosen models are presented in Figure 3.

kde4${}^{LSTM}_{Aug}$

kde10${}^{LSTM}_{Aug}$

Figure 3: Parity plots for two selected models being evaluated on the
solubility challenge datasets: ($i$) kde4${}^{LSTM}_{Aug}$ (top row), and
($ii$) kde10${}^{LSTM}_{Aug}$ (botom row). The left, middle, and right columns
show the parity plots for solubility challenge 1[29], 2-set1, and 2-set2[30],
respectively. Pearson correlation coefficient is displayed together with RMSE
and MAE. “acc-0.5” stands for the $\pm 0.5\textrm{log}\%$ metric. Red dashed
lines show the limits for molecules considered a correct prediction when
computing the $\pm 0.5\textrm{log}\%$. The correlation between predicted
values and labels increases when more models are added to the ensemble. RMSE
and MAE also follow this pattern. However, the $\pm 0.5\textrm{log}\%$
decreases in set-2 of the second solubility challenge dataset (SolChal2-set2).
While kde10${}^{LSTM}_{Aug}$ improved the prediction of molecules that were
being poorly predicted by kde4${}^{LSTM}_{Aug}$, the prediction of molecules
with smaller errors was not greatly improved.

Comparing the performance of different models is a complex task, as
performance metrics cannot be directly compared across models evaluated on
distinct datasets. To address this issue, Panapitiya et al. [89] curated a
large and diverse dataset to train models with various architectures and
molecular representations. They also compared the performance of these models
on datasets from the literature[92, 24, 93, 94, 25, 95, 29, 88, 30, 96, 97,
98]. Although their models achieved an RMSE of $\sim 1.1$ on their test set,
using descriptors as molecular representations resulted in RMSE values ranging
from $0.55$ to $\sim 1.35$ when applied to other datasets from the literature.
According to their study, the Solubility Challenge datasets by Llinàs et
al.[29, 30] were found to be particularly challenging due to their more
significant reproducibility error. Therefore, we focused on the Llinàs
datasets to compare our performance with the literature.

Focusing on the solubility challenge 1 dataset[29], kde10${}^{LSTM}_{Aug}$ is
only $\sim 0.2$ RMSE units worse than the best model available in the
literature[34]. The RMSE of the participants of the challenge was not
reported.[31] The primary metric used to evaluate models was the percentage of
predictions within an error of 0.5 LogS units (called $\pm
0.5\textrm{log}\%$). Computing the same metric, kde10${}^{LSTM}_{Aug}$ has a
percentage of correct prediction of 44.4%. This result would place our model
among the 35% best participants. The participant with the best performance
presented a $\pm 0.5\textrm{log}\%$ of 60.7%.

The architecture of the models was not published in the findings of the first
challenge.[31] Nevertheless, the findings for the second challenge[32]
investigated the participants more thoroughly. Participants were asked to
identify their models’ architecture and descriptors used. The challenge is
divided into two datasets. Set-1 contains LogS values with an average
interlaboratory reproducibility of 0.17 LogS. Our kde10${}^{LSTM}_{Aug}$
achieve an RMSE of 0.983 and a $\pm 0.5\textrm{log}\%$ of 40.0% in this
dataset. Therefore, our model performs better than 62% of the published RMSE
values and 50% of the $\pm 0.5\textrm{log}\%$. In addition, the model with the
best performance is an artificial neural network (ANN) that correctly
predicted 61% ($\pm 0.5\textrm{log}\%$) of the molecule’s LogS using a
combination of molecule descriptors and fingerprints. The second dataset
(set-2) contains molecules whose solubility measurements are more challenging,
reporting an average error in reproducibility of 0.62 LogS. The
kde10${}^{LSTM}_{Aug}$ achieves an RMSE of 1.263 and a $\pm 0.5\textrm{log}\%$
of 23.3%. It performs better than 82% of the candidates when considering the
RMSE. Surprisingly, $\pm 0.5\textrm{log}\%$ does not follow this outstanding
performance, which is more significant than only 32% Regarding the literature,
kde10${}^{LSTM}_{Aug}$ has an RMSE only $\sim 0.1$ higher than a GNN that used
an extensive set of numeric and one-hot descriptors in their feature
vector.[89] Our model performs better than a transformer model that uses
SMILES-string and an adjacency matrix and inputs.[41] The performance of those
models is available in Table 2.

Notably, all participants in the solubility challenge 2 submitted a kind of
QSPR or descriptor-based ML model. Using descriptors provides an easy way to
ensure model invariance concerning molecule representation and is more
informative since they can be physical quantities. However, selecting
appropriate descriptors is crucial for developing descriptor-based ML models.
It often requires specialists with a strong intuition about the relevant
physical and chemical properties for predicting the target quantity. Feature-
based models are still being considered to be the SOTA of solubility
prediction. Recently, studies investigating different descriptors and
fingerprints were performed.[36, 99] These studies showed that similarly to
the impacts of data quality[28], molecular representation also has a great
impact on models’ performance. Despite Tayyebi et al. [36] being able to
achieve an MAE of 0.64 on solubility challenge 1 when using Morgan
fingerprints (MF), Zagidullin et al. [99] reported poor performance when using
MF. Our approach, on the other hand, is based on extracting information from
simple string representations, a more straightforward raw data. Furthermore,
we could achieve SOTA performance while balancing the model size and
complexity and using a raw input (a simple string). This simplified usage
enables running the model on devices with limited computing power.

Lastly, transformer models have been used to address the issue of accurately
predicting the solubility of small compounds. The typical workflow for
transformers involves pre-training the model using a large dataset and
subsequently fine-tuning it for a specific downstream task using a smaller
dataset. Most existing models were either pre-trained on the ESOL[25] dataset
or pre-trained on a larger dataset and fine-tuned using ESOL. Hence, the
generalizability of those models cannot be verified. In a study by Francoeur
and Koes [41], they considered two versions of their model, SolTranNet. The
first version of SolTranNet was trained with the ESOL dataset using random
splits. This approach achieved an RMSE of 0.278. Subsequently, the deployed
version of SolTranNet was trained with the AqSolDB[1]. When ESOL was used to
evaluate their deployed version, the model presented an RMSE of 2.99. While
our model achieved an RMSE of 1.316 on ESOL, outperforming the SolTranNet
deployed version, it cannot be compared with other models trained on ESOL.

##  5 Conclusions

We used TensorFlow.js, the JavaScript library of TensorFlow, to develop a deep
ensemble recurrent neural network (RNN) capable of accurately predicting LogS
values from SMILES or SELFIES string representations. This model is accessible
via a static website at https://mol.dev/. The significant contributions of
this research include: (1) Demonstrating the feasibility of using string
representations for solubility prediction; (2) establishing that using string
representations does not significantly compromise performance, with our models
achieving results comparable to state-of-the-art (SOTA) models on the datasets
by Llinas et al.; (3) enhancing prediction reliability and practical
applicability by incorporating uncertainty estimates; and (4) significantly
facilitating model usability through the deployment on a static website,
eliminating the need for domain-specific data or expertise.

Our based on a deep ensemble of recurrent neural networks (RNNs) model was
trained using SMILES randomization for data augmentation on the AqSolDB
dataset and validated using the solubility challenges by Llínas et al. It
directly processes molecular string representations, such as SMILES or
SELFIES, to predict solubility without relying on pre-selected descriptors.
This approach not only simplifies the prediction process but also enhances its
applicability across a broader chemical space. In addition, we show that this
deep ensemble RNN model could achieve similar performance compared to a random
forest (RF) using PaDEL descriptors. RFs with descriptors were shown to
perform relatively well in other datasets.

By carefully compromising between performance and complexity, we developed a
model with acceptable performance and that is not computationally intensive.
It enables us to host the model on a static website using TensorFlow JS. Our
model was designed to operate on devices with limited computational resources,
aiming to broaden the accessibility of advanced solubility prediction tools.
This application can satisfactorily run on any device with limited
computational resources, such as laptops and smartphones. This approach
ensures wider applicability, catering to the needs of users without access to
high-performance computing facilities, improving usability and flexibility,
and decreasing implementation costs. We believe this is a considerable step in
improving the usability of deep learning models and promoting such models to a
broader scientific community.

## Data and code availability

All code needed to reproduce those results is publicly available on the
following GitHub repository: https://github.com/ur-whitelab/mol.dev. The model
is also publicly accessible at the following address: https://mol.dev/.

## Authors contribution

M.C.R. implemented the deep learning model, performed the training and
hyperparameters optimization, tested the model’s performance, analyzed the
results, and wrote this manuscript. A.D.W. idealized the project, proposed the
model to be used, implemented the deep learning approach, and developed
http://mol.dev/.

### Conflicts of interest

The authors have no conflicts to declare.

### Acknowledgments

The authors acknowledge the National Institute of General Medical Sciences of
the National Institutes of Health (NIH) under award number R35GM137966. This
research used the computational resources and structure provided by the Center
for Integrated Research Computing (CIRC) at the University of Rochester.

## References

  * Sorkun et al. [2019] Murat Cihan Sorkun, Abhishek Khetan, and Süleyman Er.  AqSolDB, a curated reference set of aqueous solubility and 2D descriptors for a diverse set of compounds.  _Sci Data_ , 6(1):143, August 2019.  ISSN 2052-4463.  doi:10.1038/s41597-019-0151-1.
  * Dajas [2012] Federico Dajas.  Life or death: neuroprotective and anticancer effects of quercetin.  _J. Ethnopharmacol._ , 143(2):383–396, September 2012.  ISSN 0378-8741, 1872-7573.  doi:10.1016/j.jep.2012.07.005.
  * Di et al. [2012] Li Di, Paul V Fish, and Takashi Mano.  Bridging solubility between drug discovery and development.  _Drug Discov. Today_ , 17(9-10):486–495, May 2012\.  ISSN 1359-6446, 1878-5832.  doi:10.1016/j.drudis.2011.11.007.
  * Docherty et al. [2015] Robert Docherty, Klimentina Pencheva, and Yuriy A Abramov.  Low solubility in drug development: de-convoluting the relative importance of solvation and crystal packing.  _J. Pharm. Pharmacol._ , 67(6):847–856, June 2015\.  ISSN 0022-3573, 2042-7158.  doi:10.1111/jphp.12393.
  * Barrett et al. [2022] Jaclyn A Barrett, Wenzhan Yang, Suzanne M Skolnik, Lisa M Belliveau, and Kellyn M Patros.  Discovery solubility measurement and assessment of small molecules with drug development in mind.  _Drug Discov. Today_ , 27(5):1315–1325, May 2022\.  ISSN 1359-6446, 1878-5832.  doi:10.1016/j.drudis.2022.01.017.
  * Sormanni et al. [2015] Pietro Sormanni, Francesco A Aprile, and Michele Vendruscolo.  The CamSol method of rational design of protein mutants with enhanced solubility.  _J. Mol. Biol._ , 427(2):478–490, January 2015\.  ISSN 0022-2836, 1089-8638.  doi:10.1016/j.jmb.2014.09.026.
  * Herrero-Martínez et al. [2005] José M Herrero-Martínez, Meritxell Sanmartin, Martí Rosés, Elisabeth Bosch, and Clara Ràfols.  Determination of dissociation constants of flavonoids by capillary electrophoresis.  _Electrophoresis_ , 26(10):1886–1895, 2005.  ISSN 0173-0835.  doi:10.1002/elps.200410258.
  * Diorazio et al. [2016] Louis J Diorazio, David R J Hose, and Neil K Adlington.  Toward a more holistic framework for solvent selection.  _Org. Process Res. Dev._ , 20(4):760–773, April 2016.  ISSN 1083-6160.  doi:10.1021/acs.oprd.6b00015.
  * Sheikholeslamzadeh and Rohani [2012] Ehsan Sheikholeslamzadeh and Sohrab Rohani.  Solubility prediction of pharmaceutical and chemical compounds in pure and mixed solvents using predictive models.  _Ind. Eng. Chem. Res._ , 51(1):464–473, January 2012.  ISSN 0888-5885.  doi:10.1021/ie201344k.
  * Yalkowsky and Valvani [1980] S H Yalkowsky and S C Valvani.  Solubility and partitioning i: Solubility of nonelectrolytes in water.  _J. Pharm. Sci._ , 69(8):912–922, August 1980\.  ISSN 0022-3549.  doi:10.1002/jps.2600690814.
  * Ran and Yalkowsky [2001] Y Ran and S H Yalkowsky.  Prediction of drug solubility by the general solubility equation (GSE).  _J. Chem. Inf. Comput. Sci._ , 41(2):354–357, March 2001.  ISSN 0095-2338.  doi:10.1021/ci000338c.
  * Fredenslund et al. [1975] A Fredenslund, R L Jones, and J M Prausnitz.  Group-contribution estimation of activity coefficients in nonideal liquid mixtures.  _AIChE J._ , 1975.  ISSN 0001-1541.
  * Abrams and Prausnitz [1975] Denis S Abrams and John M Prausnitz.  Statistical thermodynamics of liquid mixtures: A new expression for the excess gibbs energy of partly or completely miscible systems.  _AIChE J._ , 21(1):116–128, January 1975.  ISSN 0001-1541, 1547-5905.  doi:10.1002/aic.690210115.
  * Maurer and Prausnitz [1978] G Maurer and J M Prausnitz.  On the derivation and extension of the uniquac equation.  _Fluid Phase Equilib._ , 2(2):91–99, January 1978\.  ISSN 0378-3812, 1879-0224.  doi:10.1016/0378-3812(78)85002-x.
  * Lüder et al. [2007a] Kai Lüder, Lennart Lindfors, Jan Westergren, Sture Nordholm, and Roland Kjellander.  In silico prediction of drug solubility. 3. free energy of solvation in pure amorphous matter, 2007a.
  * Lüder et al. [2007b] Kai Lüder, Lennart Lindfors, Jan Westergren, Sture Nordholm, and Roland Kjellander.  In silico prediction of drug solubility: 2. free energy of solvation in pure melts, 2007b.
  * Boothroyd et al. [2018] Simon Boothroyd, Andy Kerridge, Anders Broo, David Buttar, and Jamshed Anwar.  Solubility prediction from first principles: a density of states approach.  _Phys. Chem. Chem. Phys._ , 20(32):20981–20987, August 2018.  ISSN 1463-9076, 1463-9084.  doi:10.1039/c8cp01786g.
  * Boothroyd and Anwar [2019] Simon Boothroyd and Jamshed Anwar.  Solubility prediction for a soluble organic molecule via chemical potentials from density of states.  _J. Chem. Phys._ , 151(18):184113, November 2019\.  ISSN 0021-9606, 1089-7690.  doi:10.1063/1.5117281.
  * Tomasi et al. [2005] Jacopo Tomasi, Benedetta Mennucci, and Roberto Cammi.  Quantum mechanical continuum solvation models.  _Chem. Rev._ , 105(8):2999–3093, August 2005\.  ISSN 0009-2665.  doi:10.1021/cr9904009.
  * Yu et al. [2006] Xinliang Yu, Xueye Wang, Hanlu Wang, Xiaobing Li, and Jinwei Gao.  Prediction of solubility parameters for polymers by a QSPR model.  _QSAR Comb. Sci._ , 25(2):156–161, February 2006\.  ISSN 1611-020X, 1611-0218.  doi:10.1002/qsar.200530138.
  * Ghasemi and Saaidpour [2007] Jahanbakhsh Ghasemi and Saadi Saaidpour.  QSPR prediction of aqueous solubility of Drug-Like organic compounds, 2007.
  * Duchowicz and Castro [2009] Pablo R Duchowicz and Eduardo A Castro.  QSPR studies on aqueous solubilities of drug-like compounds.  _Int. J. Mol. Sci._ , 10(6):2558–2577, June 2009\.  ISSN 1422-0067.  doi:10.3390/ijms10062558.
  * Louis et al. [2009] Bruno Louis, Jyoti Singh, Basheerulla Shaik, Vijay K Agrawal, and Padmakar V Khadikar.  QSPR study on the estimation of solubility of drug-like organic compounds: A case of barbiturates, 2009.
  * Huuskonen [2000] J Huuskonen.  Estimation of aqueous solubility for a diverse set of organic compounds based on molecular topology.  _J. Chem. Inf. Comput. Sci._ , 40(3):773–777, May 2000.  ISSN 0095-2338.  doi:10.1021/ci9901338.
  * Delaney [2004] John S Delaney.  ESOL: estimating aqueous solubility directly from molecular structure.  _J. Chem. Inf. Comput. Sci._ , 44(3):1000–1005, May 2004.  ISSN 0095-2338.  doi:10.1021/ci034243x.
  * Skyner et al. [2015] R E Skyner, J L McDonagh, C R Groom, T van Mourik, and J B O Mitchell.  A review of methods for the calculation of solution free energies and the modelling of systems in solution.  _Phys. Chem. Chem. Phys._ , 17(9):6174–6191, March 2015.  ISSN 1463-9076, 1463-9084.  doi:10.1039/c5cp00288e.
  * McDonagh et al. [2014] James L McDonagh, Neetika Nath, Luna De Ferrari, Tanja van Mourik, and John B O Mitchell.  Uniting cheminformatics and chemical theory to predict the intrinsic aqueous solubility of crystalline druglike molecules.  _J. Chem. Inf. Model._ , 54(3):844–856, March 2014.  ISSN 1549-9596, 1549-960X.  doi:10.1021/ci4005805.
  * Sorkun et al. [2021] Murat Cihan Sorkun, J M Vianney A Koelman, and Süleyman Er.  Pushing the limits of solubility prediction via quality-oriented data selection.  _iScience_ , 24(1):101961, January 2021.  ISSN 2589-0042.  doi:10.1016/j.isci.2020.101961.  URL http://dx.doi.org/10.1016/j.isci.2020.101961.
  * Llinàs et al. [2008] Antonio Llinàs, Robert C Glen, and Jonathan M Goodman.  Solubility challenge: can you predict solubilities of 32 molecules using a database of 100 reliable measurements?  _J. Chem. Inf. Model._ , 48(7):1289–1303, July 2008.  ISSN 1549-9596.  doi:10.1021/ci800058v.
  * Llinas and Avdeef [2019] Antonio Llinas and Alex Avdeef.  Solubility challenge revisited after ten years, with multilab shake-flask data, using tight (SD 0.17 log) and loose (SD 0.62 log) test sets.  _J. Chem. Inf. Model._ , 59(6):3036–3040, June 2019.  ISSN 1549-9596, 1549-960X.  doi:10.1021/acs.jcim.9b00345.
  * Hopfinger et al. [2009] Anton J Hopfinger, Emilio Xavier Esposito, A Llinàs, R C Glen, and J M Goodman.  Findings of the challenge to predict aqueous solubility.  _J. Chem. Inf. Model._ , 49(1):1–5, January 2009\.  ISSN 1549-9596.  doi:10.1021/ci800436c.
  * Llinas et al. [2020] Antonio Llinas, Ioana Oprisiu, and Alex Avdeef.  Findings of the second challenge to predict aqueous solubility.  _J. Chem. Inf. Model._ , 60(10):4791–4803, October 2020.  ISSN 1549-9596, 1549-960X.  doi:10.1021/acs.jcim.0c00701.
  * Schwaighofer et al. [2007] Anton Schwaighofer, Timon Schroeter, Sebastian Mika, Julian Laub, Antonius ter Laak, Detlev Sülzle, Ursula Ganzer, Nikolaus Heinrich, and Klaus-Robert Müller.  Accurate solubility prediction with error bars for electrolytes: a machine learning approach.  _J. Chem. Inf. Model._ , 47(2):407–424, March 2007.  ISSN 1549-9596.  doi:10.1021/ci600205g.
  * Lusci et al. [2013] Alessandro Lusci, Gianluca Pollastri, and Pierre Baldi.  Deep architectures and deep learning in chemoinformatics: the prediction of aqueous solubility for drug-like molecules.  _J. Chem. Inf. Model._ , 53(7):1563–1575, July 2013.  ISSN 1549-9596, 1549-960X.  doi:10.1021/ci400187y.
  * Ye and Ouyang [2021] Zhuyifan Ye and Defang Ouyang.  Prediction of small-molecule compound solubility in organic solvents by machine learning algorithms.  _J. Cheminform._ , 13(1):98, December 2021.  ISSN 1758-2946.  doi:10.1186/s13321-021-00575-3.  URL http://dx.doi.org/10.1186/s13321-021-00575-3.
  * Tayyebi et al. [2023] Arash Tayyebi, Ali S Alshami, Zeinab Rabiei, Xue Yu, Nadhem Ismail, Musabbir Jahan Talukder, and Jason Power.  Prediction of organic compound aqueous solubility using machine learning: a comparison study of descriptor-based and fingerprints-based models.  _J. Cheminform._ , 15(1):99, October 2023.  ISSN 1758-2946.  doi:10.1186/s13321-023-00752-6.  URL http://dx.doi.org/10.1186/s13321-023-00752-6.
  * Kurotani et al. [2021] Atsushi Kurotani, Toshifumi Kakiuchi, and Jun Kikuchi.  Solubility prediction from molecular properties and analytical data using an in-phase deep neural network (Ip-DNN).  _ACS Omega_ , 6(22):14278–14287, June 2021.  ISSN 2470-1343.  doi:10.1021/acsomega.1c01035.  URL http://dx.doi.org/10.1021/acsomega.1c01035.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  June 2017.  URL http://arxiv.org/abs/1706.03762.
  * Wang et al. [2019] S Wang, Y Guo, Y Wang, H Sun, and J Huang.  SMILES-BERT: large scale unsupervised pre-training for molecular property prediction.  _Proceedings of the 10th ACM_ , 2019.
  * Fabian et al. [2020] B Fabian, T Edlich, H Gaspar, M Segler, and others.  Molecular representation learning with language models and domain-relevant auxiliary tasks.  _arXiv preprint arXiv_ , 2020.
  * Francoeur and Koes [2021] Paul G Francoeur and David R Koes.  SolTranNet–A machine learning tool for fast aqueous solubility prediction.  _J. Chem. Inf. Model._ , 61(6):2530–2536, June 2021.  ISSN 1549-9596.  doi:10.1021/acs.jcim.1c00331.
  * Born and Manica [2022] Jannis Born and Matteo Manica.  Regression transformer: Concurrent sequence regression and generation for molecular language modeling.  February 2022.  URL http://arxiv.org/abs/2202.01338.
  * Ross et al. [2022] Jerret Ross, Brian Belgodere, Vijil Chenthamarakshan, Inkit Padhi, Youssef Mroueh, and Payel Das.  Molformer: Large scale chemical language representations capture molecular structure and properties.  May 2022.
  * Zdrazil and Guha [2017] Barbara Zdrazil and Rajarshi Guha.  The rise and fall of a scaffold: a trend analysis of scaffolds in the medicinal chemistry literature.  _Journal of Medicinal Chemistry_ , 61(11):4688–4703, 2017.
  * Seelow [2020] D Seelow.  Editorial: the 18th annual nucleic acids research web server issue 2020\.  _Nucleic Acids Res_ , 48:W1–W4, 2020.
  * Baek et al. [2021] Minkyung Baek, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N Kinch, R Dustin Schaeffer, Claudia Millán, Hahnbeom Park, Carson Adams, Caleb R Glassman, Andy DeGiovanni, Jose H Pereira, Andria V Rodrigues, Alberdina A van Dijk, Ana C Ebrecht, Diederik J Opperman, Theo Sagmeister, Christoph Buhlheller, Tea Pavkov-Keller, Manoj K Rathinaswamy, Udit Dalwadi, Calvin K Yip, John E Burke, K Christopher Garcia, Nick V Grishin, Paul D Adams, Randy J Read, and David Baker.  Accurate prediction of protein structures and interactions using a three-track neural network.  _Science_ , 373(6557):871–876, 2021.  ISSN 0036-8075.  doi:10.1126/science.abj8754.
  * Stroet et al. [2018] Martin Stroet, Bertrand Caron, Koen M Visscher, Daan P Geerke, Alpeshkumar K Malde, and Alan E Mark.  Automated topology builder version 3.0: Prediction of solvation free enthalpies in water and hexane.  _J. Chem. Theory Comput._ , 14(11):5834–5845, 2018.  ISSN 1549-9618, 1549-9626.  doi:10.1021/acs.jctc.8b00768.
  * Smith et al. [2021] Daniel G A Smith, Doaa Altarawy, Lori A Burns, Matthew Welborn, Levi N Naden, Logan Ward, Sam Ellis, Benjamin P Pritchard, and T Daniel Crawford.  The MolSSI QCA rchive project: An open-source platform to compute, organize, and share quantum chemistry data.  _Wiley Interdiscip. Rev. Comput. Mol. Sci._ , 11(2), March 2021.  ISSN 1759-0876, 1759-0884.  doi:10.1002/wcms.1491.
  * Ansari and White [2023] Mehrad Ansari and Andrew D. White.  Serverless prediction of peptide properties with recurrent neural networks.  _Journal of Chemical Information and Modeling_ , 63(8):2546–2553, April 2023.  doi:10.1021/acs.jcim.2c01317.  URL https://doi.org/10.1021/acs.jcim.2c01317.
  * Lakshminarayanan et al. [2016] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  December 2016.
  * Weininger [1988] David Weininger.  SMILES, a chemical language and information system. 1. introduction to methodology and encoding rules.  _J. Chem. Inf. Model._ , 28(1):31–36, February 1988.  ISSN 1549-9596, 1549-960X.  doi:10.1021/ci00057a005.
  * Krenn et al. [2022] Mario Krenn, Qianxiang Ai, Senja Barthel, Nessa Carson, Angelo Frei, Nathan C Frey, Pascal Friederich, Théophile Gaudin, Alberto Alexander Gayle, Kevin Maik Jablonka, Rafael F Lameiro, Dominik Lemm, Alston Lo, Seyed Mohamad Moosavi, José Manuel Nápoles-Duarte, Akshatkumar Nigam, Robert Pollice, Kohulan Rajan, Ulrich Schatzschneider, Philippe Schwaller, Marta Skreta, Berend Smit, Felix Strieth-Kalthoff, Chong Sun, Gary Tom, Guido Falk von Rudorff, Andrew Wang, Andrew D White, Adamo Young, Rose Yu, and Alán Aspuru-Guzik.  SELFIES and the future of molecular string representations.  _Patterns (N Y)_ , 3(10):100588, October 2022\.  ISSN 2666-3899.  doi:10.1016/j.patter.2022.100588.  URL http://dx.doi.org/10.1016/j.patter.2022.100588.
  * Kim et al. [2018] Sunghwan Kim, Paul A Thiessen, Tiejun Cheng, Bo Yu, and Evan E Bolton.  An update on PUG-REST: RESTful interface for programmatic access to PubChem.  _Nucleic Acids Res._ , 46(W1):W563–W570, July 2018.  ISSN 0305-1048, 1362-4962.  doi:10.1093/nar/gky294.  URL http://dx.doi.org/10.1093/nar/gky294.
  * Schilter et al. [2024] Oliver Tobias Schilter, Teodoro Laino, and Philippe Schwaller.  CMD + V for chemistry: Image to chemical structure conversion directly done in the clipboard.  _Appl. AI Lett._ , 5(1), February 2024.  ISSN 2689-5595.  doi:10.1002/ail2.91.  URL https://onlinelibrary.wiley.com/doi/10.1002/ail2.91.
  * Beltran et al. [2018] Jesus A Beltran, Longendri Aguilera-Mendoza, and Carlos A Brizuela.  Optimal selection of molecular descriptors for antimicrobial peptides classification: an evolutionary feature weighting approach.  _BMC Genomics_ , 19(Suppl 7):672, September 2018\.  ISSN 1471-2164.  doi:10.1186/s12864-018-5030-1.
  * Maggiora [2006] Gerald M Maggiora.  On outliers and activity cliffs why qsar often disappoints, 2006.
  * Smilkov et al. [2019] Daniel Smilkov, Nikhil Thorat, Yannick Assogba, Charles Nicholson, Nick Kreeger, Ping Yu, Shanqing Cai, Eric Nielsen, David Soegel, Stan Bileschi, and Others.  Tensorflow. js: Machine learning for the web and beyond.  _Proceedings of Machine Learning and Systems_ , 1:309–321, 2019.
  * Wang et al. [2009] Junmei Wang, Tingjun Hou, and Xiaojie Xu.  Aqueous solubility prediction based on weighted atom type counts and solvent accessible surface areas.  _J. Chem. Inf. Model._ , 49(3):571–581, March 2009.  ISSN 1549-9596.  doi:10.1021/ci800406y.
  * Wang and Hou [2011] Junmei Wang and Tingjun Hou.  Recent advances on aqueous solubility prediction.  _Comb. Chem. High Throughput Screen._ , 14(5):328–338, June 2011.  ISSN 1386-2073, 1875-5402.  doi:10.2174/138620711795508331.
  * [60] Landrum.  Rdkit documentation.  _RELease 1.0_.  ISSN 1047-935X, 1533-3752.
  * Arús-Pous et al. [2019] Josep Arús-Pous, Simon Viet Johansson, Oleksii Prykhodko, Esben Jannik Bjerrum, Christian Tyrchan, Jean-Louis Reymond, Hongming Chen, and Ola Engkvist.  Randomized SMILES strings improve the quality of molecular generative models.  _J. Cheminform._ , 11(1):71, November 2019.  ISSN 1758-2946.  doi:10.1186/s13321-019-0393-0.
  * Schwaller et al. [2020] Philippe Schwaller, Alain C Vaucher, Teodoro Laino, and Jean-Louis Reymond.  Data augmentation strategies to improve reaction yield predictions and estimate uncertainty.  _ChemRxiv_ , November 2020.  doi:10.26434/chemrxiv.13286741.v1.
  * Shaker and Hüllermeier [2020] Mohammad Hossein Shaker and Eyke Hüllermeier.  Aleatoric and epistemic uncertainty with random forests.  In _Advances in Intelligent Data Analysis XVIII_ , pages 444–456. Springer International Publishing, 2020.  doi:10.1007/978-3-030-44584-3_35.
  * Ghoshal et al. [2021] Biraja Ghoshal, Allan Tucker, Bal Sanghera, and Wai Lup Wong.  Estimating uncertainty in deep learning for reporting confidence to clinicians in medical image segmentation and diseases detection.  _Comput. Intell._ , 37(2):701–734, May 2021.  ISSN 0824-7935, 1467-8640.  doi:10.1111/coin.12411.
  * Scalia et al. [2020] Gabriele Scalia, Colin A Grambow, Barbara Pernici, Yi-Pei Li, and William H Green.  Evaluating scalable uncertainty estimation methods for deep Learning-Based molecular property prediction.  _J. Chem. Inf. Model._ , 60(6):2697–2717, June 2020.  ISSN 1549-9596, 1549-960X.  doi:10.1021/acs.jcim.9b00975.
  * Chollet and others [2018] François Chollet and others.  Keras: The python deep learning library, June 2018.
  * Abadi et al. [2015] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, and Others.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.
  * Hochreiter and Schmidhuber [1997] S Hochreiter and J Schmidhuber.  Long short-term memory.  _Neural Comput._ , 9(8):1735–1780, November 1997\.  ISSN 0899-7667.  doi:10.1162/neco.1997.9.8.1735.
  * Zhang et al. [2023] Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola.  _Dive into Deep Learning_.  Cambridge University Press, 2023.  https://D2L.ai.
  * Ba et al. [2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  July 2016.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Francis Bach and David Blei, editors, _Proceedings of the 32nd International Conference on Machine Learning_ , volume 37 of _Proceedings of Machine Learning Research_ , pages 448–456, Lille, France, 2015. PMLR.
  * Awais et al. [2021] Muhammad Awais, Md Tauhid Bin Iqbal, and Sung-Ho Bae.  Revisiting internal covariate shift for batch normalization, 2021.
  * Santurkar et al. [2018] Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry.  How does batch normalization help optimization?  In S Bengio, H Wallach, H Larochelle, K Grauman, N Cesa-Bianchi, and R Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc., 2018.
  * Xu et al. [2019] Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin.  Understanding and improving layer normalization.  November 2019.
  * Tian and Zhang [2022] Yingjie Tian and Yuqi Zhang.  A comprehensive survey on regularization strategies in machine learning.  _Inf. Fusion_ , 80:146–166, April 2022.  ISSN 1566-2535.  doi:10.1016/j.inffus.2021.11.005.
  * Gal and Ghahramani [2016] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In Maria Florina Balcan and Kilian Q Weinberger, editors, _Proceedings of The 33rd International Conference on Machine Learning_ , volume 48 of _Proceedings of Machine Learning Research_ , pages 1050–1059, New York, New York, USA, 2016. PMLR.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  December 2014.
  * Mitchell et al. [2019] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru.  Model cards for model reporting.  In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ , FAT* ’19, pages 220–229, New York, NY, USA, January 2019. Association for Computing Machinery.  ISBN 9781450361255.  doi:10.1145/3287560.3287596.
  * Yap [2011] Chun Wei Yap.  PaDEL-descriptor: An open source software to calculate molecular descriptors and fingerprints, 2011.
  * Gao et al. [2020] Shuai Gao, Yuefei Huang, Shuo Zhang, Jingcheng Han, Guangqian Wang, Meixin Zhang, and Qingsheng Lin.  Short-term runoff prediction with GRU and LSTM networks without requiring time step optimization during sample generation.  _J. Hydrol._ , 589:125188, October 2020.  ISSN 0022-1694.  doi:10.1016/j.jhydrol.2020.125188.
  * Kim et al. [2021a] Jongyeop Kim, Seongsoo Kim, Hayden Wimmer, and Hong Liu.  A cryptocurrency prediction model using LSTM and GRU algorithms.  In _2021 IEEE/ACIS 6th International Conference on Big Data, Cloud Computing, and Data Science (BCD)_ , pages 37–44. ieeexplore.ieee.org, September 2021a.  doi:10.1109/BCD51206.2021.9581397.
  * Encean and Zinca [2022] Andrei-Alexandru Encean and Daniel Zinca.  Cryptocurrency price prediction using LSTM and GRU networks, 2022\.
  * Kumar et al. [2021] V Bharat Kumar, V Bharat Kumar, V Mallikarjuna Nookesh, B Satya Saketh, S Syama, and J Ramprabhakar.  Wind speed prediction using deep Learning-LSTM and GRU, 2021.
  * Liu et al. [2021] Xiaolei Liu, Zi Lin, and Ziming Feng.  Short-term offshore wind speed forecast by seasonal ARIMA - a comparison against GRU and LSTM.  _Energy_ , 227:120492, July 2021.  ISSN 0360-5442.  doi:10.1016/j.energy.2021.120492.
  * Mateus et al. [2021] Balduíno César Mateus, Mateus Mendes, José Torres Farinha, Rui Assis, and António Marques Cardoso.  Comparing LSTM and GRU models to predict the condition of a pulp paper press.  _Energies_ , 14(21):6958, October 2021.  ISSN 1996-1073, 1996-1073.  doi:10.3390/en14216958.
  * Gruber and Jockisch [2020] Nicole Gruber and Alfred Jockisch.  Are GRU cells more specific and LSTM cells more sensitive in motive classification of text?  _Front Artif Intell_ , 3:40, June 2020.  ISSN 2624-8212.  doi:10.3389/frai.2020.00040.
  * Chung et al. [2014] Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  December 2014.
  * Boobier et al. [2017] Samuel Boobier, Anne Osbourn, and John B O Mitchell.  Can human experts predict solubility better than computers?  _J. Cheminform._ , 9(1):63, December 2017.  ISSN 1758-2946.  doi:10.1186/s13321-017-0250-y.
  * Panapitiya et al. [2022] Gihan Panapitiya, Michael Girard, Aaron Hollas, Jonathan Sepulveda, Vijayakumar Murugesan, Wei Wang, and Emily Saldanha.  Evaluation of deep learning architectures for aqueous solubility prediction.  _ACS Omega_ , 7(18):15695–15710, May 2022.  ISSN 2470-1343.  doi:10.1021/acsomega.2c00642.
  * Yu et al. [2023] Jiahui Yu, Chengwei Zhang, Yingying Cheng, Yun-Fang Yang, Yuan-Bin She, Fengfan Liu, Weike Su, and An Su.  SolvBERT for solvation free energy and solubility prediction: a demonstration of an NLP model for predicting the properties of molecular complexes.  _Digital Discovery_ , 2(2):409–421, April 2023\.  ISSN 2635-098X.  doi:10.1039/D2DD00107A.  URL https://pubs.rsc.org/en/content/articlelanding/2023/dd/d2dd00107a.
  * Kim et al. [2021b] Hyunseob Kim, Jeongcheol Lee, Sunil Ahn, and Jongsuk Ruth Lee.  A merged molecular representation learning for molecular properties prediction with a web-based service.  _Sci. Rep._ , 11(1):11028, May 2021b.  ISSN 2045-2322.  doi:10.1038/s41598-021-90259-7.
  * Yalkowsky and Banerjee [1992] Samuel Hyman Yalkowsky and Sujit Banerjee.  Aqueous solubility: Methods of estimation for organic compounds.  _(No Title)_ , 1992.
  * Klopman and Zhu [2001] G Klopman and H Zhu.  Estimation of the aqueous solubility of organic molecules by the group contribution approach.  _J. Chem. Inf. Comput. Sci._ , 41(2):439–445, 2001.  ISSN 0095-2338.  doi:10.1021/ci000152d.
  * Hou et al. [2004] T J Hou, K Xia, W Zhang, and X J Xu.  ADME evaluation in drug discovery. 4. prediction of aqueous solubility based on atom contribution approach.  _J. Chem. Inf. Comput. Sci._ , 44(1):266–275, 2004.  ISSN 0095-2338.  doi:10.1021/ci034184n.
  * Wang et al. [2007] Junmei Wang, George Krudy, Tingjun Hou, Wei Zhang, George Holland, and Xiaojie Xu.  Development of reliable aqueous solubility models and their application in druglike analysis.  _J. Chem. Inf. Model._ , 47(4):1395–1404, June 2007.  ISSN 1549-9596.  doi:10.1021/ci700096r.
  * Boobier et al. [2020] Samuel Boobier, David R J Hose, A John Blacker, and Bao N Nguyen.  Machine learning with physicochemical relationships: solubility prediction in organic solvents and water.  _Nat. Commun._ , 11(1):5753, November 2020.  ISSN 2041-1723.  doi:10.1038/s41467-020-19594-z.
  * Tang et al. [2020] Bowen Tang, Skyler T Kramer, Meijuan Fang, Yingkun Qiu, Zhen Wu, and Dong Xu.  A self-attention based message passing neural network for predicting molecular lipophilicity and aqueous solubility.  _J. Cheminform._ , 12(1):15, February 2020.  ISSN 1758-2946.  doi:10.1186/s13321-020-0414-z.
  * Cui et al. [2020] Qiuji Cui, Shuai Lu, Bingwei Ni, Xian Zeng, Ying Tan, Ya Dong Chen, and Hongping Zhao.  Improved prediction of aqueous solubility of novel compounds by going deeper with deep learning.  _Front. Oncol._ , 10:121, February 2020.  ISSN 2234-943X.  doi:10.3389/fonc.2020.00121.
  * Zagidullin et al. [2021] B Zagidullin, Z Wang, Y Guan, E Pitkänen, and J Tang.  Comparative analysis of molecular fingerprints in prediction of drug combination effects.  _Brief. Bioinform._ , 22(6), November 2021.  ISSN 1467-5463, 1477-4054.  doi:10.1093/bib/bbab291.  URL http://dx.doi.org/10.1093/bib/bbab291.