list/template=description qHTS short = qHTS, long = quantitative high-
throughput screening, LLM short = LLM, long = large language model, MT-DNN
short = MT-DNN, long = multi-task deep neural network, jpg short = JPEG , sort
= jpeg , alt = JPG , long = Joint Photographic Experts Group ML short = ML,
long = machine learning DL short = DL, long = deep learning MCC short = MCC,
long = Matthews correlation coefficient Sp short = Sp, long = specificity Sn
short = Sn, long = sensitivity BA short = BA, long = balanced accuracy AP
short = AP, long = average precision ROC_AUC short = ROC AUC, long = receiver
operating characteristic area under curve PR_AUC short = PR AUC, long =
precision recall area under curve DPR_AUC short = $\Delta$PR AUC, long =
$\Delta$ in precision recall area under curve TPR short = TPR, long = true
positive rate TNR short = TNR, long = true negative rate FPR short = FPR, long
= false positive rate FNR short = FNR, long = false negative rate TP short =
TP, long = true positive FN short = FN, long = false negative FP short = FP,
long = false positive TN short = TN, long = true negative RF short = RF, long
= random forest AID short = AID, long = bioassay identifier HTS short = HTS,
long = high-throughput screening MMP short = MMP, alt =
$\Delta\Psi_{\text{m}}$, long = mitochondrial membrane potential m-MPI short =
m-MPI, long = mitochondrial membrane potential indicator ECACC short = ECACC,
long = European Collection of Authenticated Cell Cultures DMEM short = DMEM,
long = Dulbecco’s modified eagle medium FCS short = FCS, long = fetal calf
serum RT short = RT, long = room temperature FCCP short = FCCP, long =
carbonylcyanid-4-(trifluormethoxy)phenylhydrazon, DMSO short = DMSO, long =
dimethyl sulfoxide, ddH2O short = ddH2O, long = double destilled water,
sort=ddH2O, PBS short = PBS, long = phosphate-buffered saline, EC50 short =
EC50, long = half maximal effective concentration, AI short = AI, long =
artificial intelligence, DTI short = DTI, long = drug–target interaction, DDI
short = DDI, long = drug–drug interaction, DNA short = DNA, long =
deoxyribonucleic acid, ECFP short = ECFP, long = extended-connectivity
fingerprint, FCFP short = FCFP, long = functional-class fingerprint, MAP4
short = MAP4, long = MinHashed atom-pair fingerprint, SVM short = SVM, long =
support vector machine, DNN short = DNN, long = deep neural network, GCNN
short = GCNN, long = graph convolutional neural networks, GHS short = GHS,
long = globally harmonized system of classification and labelling of
chemicals, SMILES short = SMILES, long = simplified molecular-input line-entry
system, CLI short = CLI, long = command-line interpreter, GUI short = GUI,
long = graphical user interface, ATP short = ATP, long = adenosine
triphosphate, AMP short = AMP, long = adenosine monophosphate, PPi short =
PPi, long = pyrophosphate, sort=PPi Lu short = LH2, long = luciferin, sort=LH2
oLu short = oxy-L, long = oxy-luciferin, sort=oxylu SMOTE short = SMOTE, long
= synthetic minority over-sampling technique, SHAP short = SHAP, long =
Shapley additive explanation, CPU short = CPU, long = central processing unit,
RAM short = RAM, long = random-access memory, GPU short = GPU, long = graphics
processing unit, CAS short = CAS, long = Chemical Abstracts Service, UMAP
short = UMAP, long = uniform manifold approximation and projection, Tox21
short = Tox21, long = Toxicology in the 21st Century, GOSS short = GOSS, long
= gradient-based one-side sampling, QSAR short = QSAR, long = quantitative
structure–activity relationship, EFB short = EFB, long = exclusive feature
bundling, MTT short = MTT, long = 3-(4,
5-dimethylthiazolyl-2)-2,5-diphenyltetrazolium bromide, SSL short = SSL, long
= self-supervised learning, GBM short = GBM, long = gradient boosting machine,
MLP short = MLP, long = multilayer perceptron, API short = API, long =
application programming interface, CHOP short = CHOP, long = C/EBP homologous
protein UPR short = UPR, long = unfolded protein response ER short = ER, long
= endoplasmic reticulum PN short = PN, long = prototypical network FH short =
FH, long = frequent hitters LSA short = LSA, long = latent semantic analysis
{}$missing\text{\,}\mathrm{missing}$

#  TwinBooster: Synergising Large Language Models with Barlow Twins and
Gradient Boosting for Enhanced Molecular Property Prediction

Maximilian G. Schuh
<EMAIL_ADDRESS>
&Davide Boldini
<EMAIL_ADDRESS>
0009-0008-2415-8810 0000-0002-4109-3528 Stephan A. Sieber
Chair of Organic Chemistry II
TUM School of Natural Sciences
Technical University of Munich
<EMAIL_ADDRESS>
0000-0002-9400-906X

###### Abstract

The success of drug discovery and development relies on the precise prediction
of molecular activities and properties. While in silico molecular property
prediction has shown remarkable potential, its use has been limited so far to
assays for which large amounts of data are available. In this study, we use a
fine-tuned LLM to integrate biological assays based on their textual
information, coupled with Barlow Twins, a Siamese neural network using a novel
SSL approach. This architecture uses both assay information and molecular
fingerprints to extract the true molecular information. TwinBooster enables
the prediction of properties of unseen bioassays and molecules by providing
state-of-the-art zero-shot learning tasks. Remarkably, our AI pipeline shows
excellent performance on the FS-Mol benchmark. This breakthrough demonstrates
the application of DL to critical property prediction tasks where data is
typically scarce. By accelerating the early identification of active molecules
in drug discovery and development, this method has the potential to help
streamline the identification of novel therapeutics.

##  1 Introduction

Accurate prediction of biomolecular properties, such as toxicity,Zhang et al.
(2019) is a critical factor in accelerating the drug discovery and development
process.Deng et al. (2023); Chithrananda et al. (2020); Jain et al. (2021);
Walter et al. (2022) However, the reliance on traditional laboratory
experiments presents significant challenges. These methods are not only time-
consuming and expensive, but resource constraints make them impractical when
scaled up to large numbers of molecules.Yang et al. (2019); Shen and Nicolaou
(2019)

To bridge this gap and improve predictive accuracy, it is essential to collect
a significant amount of data. In biomolecular research, the quantity and
quality of data points are critical to the development of robust predictive
models. Without a large dataset, models may lack the precision and reliability
needed to identify potential drug candidates and assess their safety
profiles.Shen and Nicolaou (2019); Gaulton et al. (2012)

To address these challenges, the in silico analysis of chemical structures and
bioassays is emerging as a promising solution.Jain et al. (2021) This
computational approach uses large data sets to train more effective predictive
models.Seidl et al. (2023) By virtual modelling experiments, it bypasses the
limitations of traditional lab-based methods and offers a faster, more cost-
effective and scalable alternative for studying a wide range of biomolecules.
This innovative method not only streamlines the drug development process, but
also enhances the predictive capabilities critical to identifying viable drug
candidates.Merkwirth and Lengauer (2005)

Advances in LLM technology are opening up new ways of reinterpreting large
datasets, particularly in the field of bioassays.Young et al. (2018) Our
research exploits this potential by fine-tuning an LLM specifically for the
task of integrating and understanding textual information from assay titles,
descriptions and protocols to predict molecular properties.Young et al. (2018)
This approach, which is unique in its application, leverages PubChem’s
comprehensive data repository of over $1\,500\,000$ bioassays.Kim et al.
(2023)

Our method applies a fine-tuned LLM to accurately capture and interpret the
semantic nuances of bioassay text.He et al. (2021, 2023) The LLM extracts and
integrates complex information to generate meaningful semantic embeddings.
This advanced capability enhances the depth and quality of our molecular
property predictions, providing a novel and effective way to analyse bioassay
data.

Figure 1: The TwinBooster architecture. This Siamese neural network provides
an information-rich and bias-free representation of molecules in the context
of bioassays.Zbontar et al. (2021)

In our pursuit of enhancing QSAR, we introduce TwinBooster, a classification
architecture inspired by Barlow Twins.Zbontar et al. (2021) The primary
advantage of Barlow Twins over other SSL techniques lies in its novel
objective function, which measures the cross-correlation matrix between the
outputs of two identical networks processing different representations of a
molecule. We are using ECFP and the corresponding bioassay text embedded by
the fine-tuned LLM.Landrum et al. (2020); He et al. (2023, 2021) The aim is to
make this matrix as close as possible to the identity matrix. This approach
not only ensures the similarity of the embedding vectors for distorted
versions of a molecule, but also minimises the redundancy between the
components of these vectors, thereby revealing a representation that is rich
in information and free of bias (shown in fig. 1).Hadsell et al. (2006)
Notably, Barlow Twins does not require a large number of negative samples,
allowing it to work effectively on smaller batches. It also performs better on
very high-dimensional embeddings compared to current methods.Zbontar et al.
(2021)

QSAR modelling is essential in cheminformatics research, enabling in silico
predictions of molecular properties. With information-rich representations
generated by the Barlow Twins architecture decision tree ensembles such as GBM
are used in this study due to their remarkable performance, ability to rank
features, and scalability.Biau and Scornet (2016); Jiang et al. (2021);
Boldini et al. (2023); Stanley et al. (2021) In recent years, GBM have become
increasingly popular in cheminformatics for a range of tasks, such as
predicting toxicity, analysing drug sensitivity, modelling anti-cancer
activity, and identifying drug-target interactions.Zhang et al. (2019);
Breiman (2017) GBM are able to tackle broad ranges of dataset sizes and class-
imbalance ratios, ideal for scenarios in drug discovery and development
applications.Zheng et al. (2021); Winter et al. (2019) Combining GBM with the
information-rich representation provided by the Barlow Twins architecture
results in state-of-the-art performance in the zero-shot classification task.
To enhance the robustness and predictive power of our model, we move from a
conventional zero-shot framework to a novel pseudo-proteochemometric approach.
Here, a GBM is trained on the information bottleneck embeddingsTishby and
Zaslavsky (2015) derived from the Barlow Twins architecture.Zbontar et al.
(2021); Xian et al. (2020) This strategic shift enables the GBM to operate
effectively in zero-shot tasks, where its predictive capabilities are tested
on bioassays beyond the scope of its training dataset. These assays, which are
new to the model, encompass previously unseen biological targets and assay
types, providing a rigorous test of the model’s ability to capture and analyse
previously unseen data.

In conclusion, TwinBooster and this study contribute significantly to drug
discovery and development through:

  1. 1.

Achieve state-of-the-art performance in zero-shot classification tasks,
critical for drug discovery pre-screening.

  2. 2.

Provide an intuitive user experience for experimentalists in molecular
property prediction using ML, DL and LLM technology, enabling faster and more
cost-effective drug discovery.

  3. 3.

Present a conformal prediction implementation that assesses the confidence of
molecular property predictions.

  4. 4.

Present a case study of an in silico pre-screening experiment, emphasising the
design of experiments for increased efficiency and higher chances of
discovering desired hits.

##  2 Materials and Methods

###  2.1 Dataset

##### FS-Mol

The FS-Mol datasetStanley et al. (2021) proposes a new approach to drug
discovery using few-shot learning, to analyse small datasets, which are common
in drug discovery due to high data generation costs and ethical
considerations. The classification dataset and benchmarking procedure are
designed to simulate the challenges of machine learning in drug discovery,
where typically only a few hundred compounds can be tested. FS-Mol evaluates
single-task, multi-task and meta-learning approaches and contains ML
baselines. It provides training, validation as well as testing data, which are
sourced from ChEMBL.Mendez et al. (2019) In the context of few-shot learning a
set from 16 up to 256 support molecules, alongside binary activity labels are
provided.Stanley et al. (2021)

####  2.1.1 Molecular representation

##### Bioassay-based LLM text embeddings

The pipeline in this study requires titles, descriptions and protocols as
additional representation for each molecule and assay. Therefore, this text
information is extracted from PubChem.Kim et al. (2023) Using both API from
PubChem and ChEMBL a mapping of AID to ChEMBL IDs is performed.Kim et al.
(2023); Mendez et al. (2019) This is done to retrieve the information rich
text information of PubChem in combination with the ChEMBL-based FS-Mol
benchmark.

Finally, the text is converted into a vector (of shape 768) using our fine-
tuned LLM PubChemDeBERTa.

#####  ECFP

All molecules are handled in SMILES strings then converted to ECFP 1024 bits
and a radius of 2, using the Pythonvan Rossum (1995) RdkitLandrum et al.
(2020) implementation.

###  2.2 Models

####  2.2.1 LLM

##### Fine-tuning

The DeBERTa V3 base modelHe et al. (2023) is fine-tuned on the PubChem corpus
using $\sim 14\text{\,}\mathrm{GB}$ video RAM for $\sim
15\text{\,}\mathrm{h}$. In the augmented version, the description is shuffled
(“.” as delimiter) and 5 augmentations are used as the training corpus.
Therefore, the Pythonvan Rossum (1995) TransformersWolf et al. (2020) library
is used. The OptunaAkiba et al. (2019) hyperparameter optimisation library is
used to find the best combination of hyperparameters for the LLM (ref. table
1). After 20 optimisation procedure trials the best hyperparameters shown in
table 2 were found.

Table 1: LLM hyperparameter optimisation space. Hyperparameter | Range
---|---
learning_rate | {$1.5\text{\times}{10}^{-5}$, $2\text{\times}{10}^{-5}$, $2.5\text{\times}{10}^{-5}$, $3\text{\times}{10}^{-5}$}
batch_size | {16, 32}
max_length | {64, 128}
num_train_epochs | 1.0
Table 2: Best LLM fine-tuning hyperparameters. Hyperparameter | Value
---|---
ampere | True
num_train_epochs | $3.0$
learning_rate | $3\text{\times}{10}^{-5}$
weight_decay | $0.01$
batch_size | $32$
max_length | $128$
adam_beta1 | $0.9$
adam_beta2 | $0.999$
adam_epsilon | $1\text{\times}{10}^{-6}$
warmup_steps | $500$

##### Performance evaluation

For the evaluation, the perplexity is used as an evaluation metric for fine-
tuning the LLM ($\mathrm{perplexity}\in[0,\infty)$, lower values indicate
better performance).Jelinek et al. (1977) The LLM evaluation was performed on
the complete training corpus with a token masking rate of
$15\text{\,}\mathrm{\char 37\relax}$. Since our investigation focuses
exclusively on the LLM behaviour in-distribution and not out-of-distribution,
other performance metrics are not considered or evaluated.Meister and
Cotterell (2021)

The training and evaluation procedures are conducted using the PubChem
corpus.Kim et al. (2023) This corpus is selected to generate optimal
embeddings that represent the bioassays for our model.

####  2.2.2 TwinBooster

#####  MLP

Barlow Twins use MLP for both the encoders and the projector design. The
network architecture is altered from the original by having two encoders a
molecule and a text encoder. Finally, the projector is shared for both
representations.

Both encoders as well as the projector have the following structure

$\boldsymbol{l_{i+1}}=\mathrm{Linear}\left({\phi\left(\mathrm{BatchNorm}\left(\mathrm{Linear}\left(\boldsymbol{Wl_{i}+b}\right)\right)\right)}^{n}\right),$

where $\boldsymbol{l_{i}}$ is the input layer and $\boldsymbol{l_{i+1}}$ is
its output, with a flexible number of layers $n$ and adjustable dimensionality
of input and output. Furthermore, variables $\boldsymbol{W}$, $\boldsymbol{b}$
represent learnable weights and biases. A linear layer is followed by batch
normalisation,Ioffe and Szegedy (2015) an activation function $\phi$,Agarap
(2019); Ramachandran et al. (2017) and the last linear layer. The network is
constructed using PyTorch.Paszke et al. (2019)

For training the network is using Barlow Twins lossZbontar et al. (2021) and
the AdamW optimiser.Loshchilov and Hutter (2019) Manual hyperparameter tuning
was performed on a range and set of parameters listed in table 3. The model is
trained for 25 epochs or until early stopping was engaged if a validation set
is provided.

Furthermore, the model is trained using ECFP and LLM embeddings. For inactive
molecules, the embeddings are sign changed.

Table 3: Barlow Twins Hyperparameters. The range of parameters is listed and the best are highlighted in bold. Hyperparameter | Range
---|---
enc_n_neurons | {512, 1024, 2048, 4096, 8192}
enc_n_layers | {2, 3, 4}
proj_n_neurons | {512, 1024, 2048, 4096, 8192}
proj_n_layers | {2, 3, 4}
embedding_dim | {512, 1024, 2048, 4096}
act_function | {ReLU,Agarap (2019) SwishRamachandran et al. (2017)}
batch_size | {1024, 2048}
learning_rate | $\\{$5\text{\times}{10}^{-3}$,\textbf{$1\text{\times}{10}^{-4}$}\\}$
weight_decay | $\\{$1\text{\times}{10}^{-3}$,\textbf{$5\text{\times}{10}^{-3}$}\\}$

#####  GBM

The GBM package LightGBM is used for for training based on the informational
bottleneck embeddings provided by the Barlow Twins model.Tishby and Zaslavsky
(2015) Performing zero-shot predictions is done by feeding the Barlow Twins
model with ECFP and text information of the desired molecules.Ke et al. (2017)
To achieve optimal performance, the SMAC3Lindauer et al. (2022) hyperparameter
optimisation library is applied to find the optimal combination of
hyperparameters (ref. table 4), using $80\text{\,}\mathrm{\char 37\relax}$ of
the “train” data of the FS-Mol dataset for training. Optimisation is set to
200 trials. The evaluation is performed by assessing the PR_AUC and ROC_AUC on
the “valid” and the remaining $20\text{\,}\mathrm{\char 37\relax}$ of the
“train” data of the FS-Mol benchmark. SMAC3’s multi-fidelity implementation is
used with the budget parameter being represented by the n_estimators parameter
of LightGBM.Lindauer et al. (2022); Ke et al. (2017)

Table 4: GBM SMAC3 hyperparameter optimisation space. Hyperparameter | Range
---|---
budget (n_estimators) | $[200,2000]$
num_leaves |  $[62,256]$ (step size 64)
learning_rate |  $[$1\text{\times}{10}^{-8}$,1.0]$ (log scale)
min_child_samples | $[5,100]$
subsample | $[0.4,1.0]$
subsample_freq | $[0,7]$
reg_lambda | $[$1\text{\times}{10}^{-8}$,10.0]$

Finally, the LightGBM is trained using the full “train” data of the FS-Mol
dataset and the best hyperparameters listed in table 5.

Table 5: Best GBM hyperparameters after optimisation. Hyperparameter | Value
---|---
budget (n_estimators) | $2000$
num_leaves | $256$
learning_rate | $0.0711$
min_child_samples | $60$
subsample | $0.941$
subsample_freq | $1$
reg_lambda | $3.78$

##### Performance evaluation

When comparing models, we are using intersecting tasks of the “test” data, to
ensure a scientific comparison. Metric selection is based on the FS-Mol
benchmark.Stanley et al. (2021) ROC_AUC are commonly used for classifier
evaluation in the presence of class imbalance, but they can be less reliable
for rare classes due to small sample sizes.Fawcett (2006); Tharwat (2020)

ROC_AUC
$\displaystyle=\int_{0}^{1}f_{\text{\acs{TPR}}}\left(f_{\text{\acs{FPR}}}\right)\differential
f_{\text{\acs{FPR}}}$ $\displaystyle\mathrm{TPR}$
$\displaystyle=\dfrac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$
$\displaystyle\mathrm{FPR}$
$\displaystyle=\dfrac{\mathrm{FP}}{\mathrm{FP}+\mathrm{TN}}$

PR_AUC are recommended for highly skewed classes, as they provide a more
realistic view of classifier performance than ROC_AUC.201 (2018); Branco et
al. (2015); Piryonesi and El-Diraby (2020); Tharwat (2020) Moreover, both
metrics can be calculated based on the probability of the prediction rather
than the prediction itself, where a classification threshold problem can
arise.Tharwat (2020) These metrics are also used in the FS-Mol
benchmark.Stanley et al. (2021)

PR_AUC
$\displaystyle=\int_{0}^{1}f_{\text{Precision}}\left(f_{\text{Recall}}\right)\differential
f_{\text{Recall}}$ Precision
$\displaystyle=\dfrac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$ Recall
$\displaystyle=\dfrac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$

In the context of zero and few-shot learning, a different form of PR_AUC,
known as DPR_AUC, is used. Here $t_{i}$ denotes a particular task or bioassay
within the total set of $i$ tasks. The expression $\sum t_{i}$ represents the
sum of all activity endpoints for a given task, indicating the number of
active molecules. In addition, $|t_{i}|$ corresponds to the size of the task.
This metric shows sensitivity to the balance between classes, allowing for
straightforward comparisons with a baseline benchmark. This is due to the
performance of a random classifier reflecting the percentage of positive
endpoints.Stanley et al. (2021)

$\displaystyle\text{\acs{DPR_AUC}}\left(t_{i}\right)=\text{\acs{PR_AUC}}\left(t_{i}\right)-\frac{\sum
t_{i}}{|t_{i}|}$

##### Conformal prediction

In our study, we apply the conformal prediction method using the LightGBM
classifier.Ke et al. (2017); Cortés-Ciriano and Bender (2019) This technique
involves a two-step process: calibration with cross-validation on training
data (5 fold), because no calibration set is provided, and prediction on test
data.Stone (1974) Then GBM predictions are analysed while the confidence level
is set to $\epsilon=0.80$. This method is valuable in providing both
predictive outputs and insights into the certainty of each prediction.Cortés-
Ciriano and Bender (2019)

###  2.3 Case study

To highlight the zero-shot capabilities of TwinBooster a case study of
biological HTS was conducted. Therefore, the primary screen (AID
2732111https://pubchem.ncbi.nlm.nih.gov/bioassay/2732) is analysed by
TwinBooster to predict the desired properties. Then it is analysed against the
confirmatory screen (AID
504437222https://pubchem.ncbi.nlm.nih.gov/bioassay/504437). The columns
PUBCHEM_EXT_DATASOURCE_SMILES, PUBCHEM_ACTIVITY_OUTCOME as well as the text
information are used for the TwinBooster prediction pipeline.Kim et al. (2023)

##### Performance evaluation

Recall measures the proportion of relevant instances that are retrieved, this
refers to the active compounds in this case study.Tharwat (2020) It can be
expressed at the ratio of the found active molecules (TP) and all active
molecules (TP and FN).

##### Similarity estimation

The Tanimoto similarity of compounds is calculated using the corresponding
Rdkit function.Landrum et al. (2020) To highlight structural similarities and
differences, 50 compounds are randomly selected for visual reasons.

##  3 Results and Discussion

##### Fine-tuned LLM on bioassay corpus

In this study, we used Microsoft’s developed DeBERTa V3He et al. (2023) as the
underlying architecture for our LLM and fine-tuned it on a comprehensive
bioassay corpus obtained from PubChem.He et al. (2021, 2023); Kim et al.
(2023) We chose DeBERTaV3, a pre-trained LLM, for this research due to its
superior performance compared to the original DeBERTa or BERT model,
respectivly. DeBERTa V3 uses the replaced token detection pre-training task,
which is more sample-efficient than the traditional masked LLM approach. This
innovation enhances both training efficiency and model quality by removing the
“tug-of-war” dynamics present in the vanilla embedding sharing method used in
ELECTRA.He et al. (2021, 2023) The fine-tuning process aimed to enhance the
model’s performance in predicting biomolecular properties.

Through the fine-tuning of DeBERTa V3, a remarkable reduction in average
perplexity is achieved, decreasing from $10.7\text{\times}{10}^{6}$ to $1.52$
(refer to table 6).He et al. (2021, 2023) In conclusion, the performance is
unmatched by other LLM like BERT or BioBERT.Devlin et al. (2019); Lee et al.
(2020) This improvement indicates that the LLM has gained a deeper
understanding of the data, resulting in more accurate predictions and a better
fit to our specific task, by understanding terminology like cell lines,
technical equipment as well as chemicals. This ability represents a critical
foundation for further analyses.

Table 6: LLM performance evaluation. The best value is highlighted in bold. Model | Perplexity
---|---
BERT base uncasedDevlin et al. (2019) | $14.9$
DeBERTa baseHe et al. (2021) | $12.5\text{\times}{10}^{4}$
DeBERTa V3 baseHe et al. (2023) | $10.7\text{\times}{10}^{6}$
BioBERT V1.2 base casedLee et al. (2020) | $4.47$
PubChemDeBERTa | $2.32$
PubChemDeBERTa augmented | $1.52$

The chosen evaluation metric, perplexity, provides a reliable measure of the
performance of the LLM. Its use is well established in the field of language
modelling.Jelinek et al. (1977) Regarding the choice of performance metrics,
the primary goal of this analysis is to analyse the behaviour of the LLM
within its known data distribution. We are not concerned with its performance
on unseen (out-of-distribution) data. As a result, other evaluation metrics
that typically assess generalisation to out-of-distribution scenarios are not
necessary for the scope of this investigation.Meister and Cotterell (2021)

##### Zero-shot benchmark

The FS-Mol benchmark is used as the standard of measurement when assessing
zero-shot capabilities, where the aim is to predict unseen tasks. Our approach
demonstrated strong performance, achieving a DPR_AUC of $20.84\pm
0.24\text{\,}\mathrm{\char 37\relax}$, as shown in both figs. 2 and 7. For a
more detailed analysis, comparisons are made against two baselines: the zero-
shot algorithm CLAMP,Seidl et al. (2023) which reported a DPR_AUC of $19.37\pm
0.2\text{\,}\mathrm{\char 37\relax}$, and the few-shot learning approach PN of
FS-Mol.Stanley et al. (2021) TwinBooster in zero-shot learning outperforms the
best few-shot baseline of the FS-Mol benchmark, PN, at 16 support molecules
with a DPR_AUC of $20.17\pm 0.08\text{\,}\mathrm{\char 37\relax}$
(WilcoxonVirtanen et al. (2020) test $\alpha=0.05$).Stanley et al. (2021)

Table 7: Comparing different zero- and few-shot model performances across different metrics on FS-Mol. In zero-shot mode no “test” molecules are provided, in the case of the few-shot performance of PN 16 molecules of the “test” set are provided. 10 replicates each are performed. Results that are both the best and statistically significant (WilcoxonVirtanen et al. (2020) test $\alpha=0.05$) are highlighted in bold. | TwinBooster | CLAMP333It is not possible to make direct comparisons as only mean values and standard deviations are provided.Seidl et al. (2023) |  PNStanley et al. (2021)
---|---|---|---
Mode | zero-shot | zero-shot | few-shot (16)
ROC_AUC ($\mathrm{\char 37\relax}$) | $71.11\pm 0.29$ | $69.26\pm 0.20$ | —
PR_AUC ($\mathrm{\char 37\relax}$) | $68.56\pm 0.24$ | $66.55\pm 0.20$ | $67.72\pm 0.08$
DPR_AUC ($\mathrm{\char 37\relax}$) | $20.84\pm 0.24$ | $19.37\pm 0.20$ | $20.17\pm 0.08$
Figure 2: Zero- and few-shot FS-Mol benchmark performance of various ML/DL
models.Stanley et al. (2021); Seidl et al. (2023) Standard deviations are
shown between replicates.

In addition, performance is assessed on confident predictions, which are
evaluated using conformal prediction. Across all metrics, performance could be
significantly improved, e.g. with a relative DPR_AUC increase of $\sim
10\text{\,}\mathrm{\char 37\relax}$ (shown in table 8). The average ratio of
confident predictions across all bioassays is $65\text{\,}\mathrm{\char
37\relax}$.

Table 8: Comparing zero-shot performances with or without conformal prediction on FS-Mol. The confidence level is set to $\epsilon=0.80$. 10 replicates each are performed. Results that are both the best and statistically significant (WilcoxonVirtanen et al. (2020) test $\alpha=0.05$) are highlighted in bold. TwinBooster | Conformal Prediction
---|---
✗ | ✓
ROC_AUC ($\mathrm{\char 37\relax}$) | $71.11\pm 0.29$ | $73.76\pm 0.30$
PR_AUC ($\mathrm{\char 37\relax}$) | $68.56\pm 0.24$ | $71.04\pm 0.31$
DPR_AUC ($\mathrm{\char 37\relax}$) | $20.84\pm 0.24$ | $22.81\pm 0.30$

The improved performance is due to the fine-tuned LLM, which effectively
transforms bioassay text into numerical data, surpassing the capabilities of
LSA.Deerwester et al. (1990) In addition, the Barlow Twins method produces
superior embeddings that capture both bioassay and molecular data.He et al.
(2023); Zbontar et al. (2021) In the SSL framework, the Barlow Twins
architecture uses the information bottleneck principleTishby and Zaslavsky
(2015) to optimise representations, maximising molecular information while
minimising extraneous details from ECFP and LLM text embeddings. This approach
focuses on preserving important molecular details and reducing noise.Zbontar
et al. (2021) Furthermore, the Barlow Twins model trains on negative as well
as positive examples, which should help generalisation. Finally, the use of a
GBM in zero-shot inference provides fast, efficient and powerful results in
predictive drug discovery.Boldini et al. (2023); Ke et al. (2017); Zhang et
al. (2019)

###### Ablation study

The ablation study is carried out to identify the differential effects on FS-
Mol performance metrics attributable to each ablation. The first step in this
exploration involves the combination of ECFP and PubChemDeBERTa bioassay
embeddings for each molecule under investigation. The GBM is then trained
using a methodology analogous to that used in TwinBooster, which involves
hyperparameter optimisation. Subsequently, the experiment is repeated with one
modification: the original text embeddings are replaced by LSA embeddings.
These substitutions aim at evaluating the comparative effectiveness of
different text embedding techniques as well as using the Barlow Twins
architecture in the context of the GBM framework. The results of these
investigations are shown in table 9.

Table 9: Performance of different ablation experiments on FS-Mol. All results are tested pairwise using the Wilcoxon test with Bonferroni correction.Virtanen et al. (2020); Bonferroni (1936) 10 replicates each are performed. Significance in WilcoxonVirtanen et al. (2020) test is indicated at $\alpha=\frac{0.05}{3}$ for all results except those in italics, which are not significant. Results that are both the best and statistically significant are highlighted in bold. | TwinBooster | ECFP + PubChemDeBERTa | ECFP + LSA
---|---|---|---
ROC_AUC ($\mathrm{\char 37\relax}$) |  $71.11\pm 0.00$.29) |  $70.88\pm 0.00$.27) |  $70.20\pm 0.00$.22)
PR_AUC ($\mathrm{\char 37\relax}$) |  $68.57\pm 0.00$.24) |  $68.13\pm 0.00$.29) |  $67.51\pm 0.00$.21)
DPR_AUC ($\mathrm{\char 37\relax}$) |  $20.84\pm 0.00$.24) |  $20.41\pm 0.00$.29) |  $19.78\pm 0.00$.21)

The ablation results show a significant improvement when using PubChemDeBERTa
embeddings as opposed to LSA embeddings. In addition, the use of embeddings
derived from the Barlow Twins architecture as implemented in TwinBooster shows
significant performance improvements, surpassing the combined use of ECFP with
text embeddings.

##### Zero-shot case study

In the study conducted by Flaherty et al., the research team used HTS to
identify compounds that selectively activate the CHOP pathway in the context
of ER stress. The UPR is a cellular response to ER stress, primarily induced
by the accumulation of misfolded proteins within the ER.Lekstrom-Himes and
Xanthopoulos (1998); Ghosh et al. (2012); Oyadomari and Mori (2004) A key
component of the UPR is the CHOP, which is upregulated in response to
prolonged ER stress and plays a critical role in the initiation of
apoptosis.Vij et al. (2008); Ghosh et al. (2012); Goodall et al. (2010) This
pathway becomes particularly relevant in pathological conditions such as
cancer, where ER stress and UPR dysregulation are often observed.Flaherty et
al. (2014); Fawcett (2006); Oyadomari and Mori (2004)

In this study, the authors performed a systematic HTS of a diverse chemical
library to identify molecules that specifically induce the CHOP pathway.
Multiple screens were performed and curated, and also published on
PubChem.Flaherty et al. (2014); Kim et al. (2023) This screening led to the
discovery of a class of sulfonamidebenzamide compounds that effectively
activate the CHOP pathway. Subsequent investigations, including
structure–activity relationship studies, allowed these compounds to be
optimised for improved potency and selectivity.Flaherty et al. (2014)

The work of Flaherty et al. serves as an exemplary demonstration of the
effectiveness of the TwinBooster pipeline in practical scenarios. This
research is particularly valuable as it includes both primary and confirmatory
CHOP HTS, and all associated data are publicly available. Crucially, this
publication is not included in the ChEMBL database, ensuring its complete
exclusion from the FS-Mol dataset and consequently from the training data
used.Mendez et al. (2019) This clear separation underlines the suitability of
this case study to illustrate the zero-shot learning capabilities of the
TwinBooster pipeline.

The molecular and textual information from the primary screen (AID 2732) is
extracted and then predictions are made by TwinBooster. Zero-shot predictions
aim to predict a single endpoint for an unseen task.

(a) Prediction ranking

(b) Recall

Figure 3: Zero-shot predictions on the primary screen of the case study. (a)
Ranked molecules based on zero-shot prediction: highlighting earlier discovery
of confirmatory screening hits. (b) Recall curve: retrieved active compounds
as a percentage of all active compounds based on the percentage of the primary
screen provided.

TwinBooster correctly prioritises the majority of hits. It is highlighted in
fig. 3, which shows its zero-shot predictions for approximately $220\,000$
molecules for activities related to CHOP and the UPR pathway. The ability of
the model to accurately classify these molecules, particularly in the upper
likelihood range, is demonstrated in fig. 3(a), where a notable enrichment in
the discovery of confirmatory screening hits is observed among the actively
classified molecules. Looking at the performance metrics of the primary screen
in table 10, the use of conformal predictions can relatively improve the
DPR_AUC of the model by up to $72\text{\,}\mathrm{\char 37\relax}$. The
proportion of confident predictions in this zero-shot case study bioassay is
$23\text{\,}\mathrm{\char 37\relax}$. Using this in the model during pre-
screening can increase the overall hit rate and lead to a higher proportion of
active molecules showing the desired biological effects.

Table 10: Performance metrics of TwinBooster on the primary screen with or without conformal prediction. The confidence level is set to $\epsilon=0.80$. TwinBooster | Conformal Prediction
---|---
✗ | ✓
ROC_AUC ($\mathrm{\char 37\relax}$) | $58.82$ | $62.02$
PR_AUC ($\mathrm{\char 37\relax}$) | $7.10$ | $11.64$
DPR_AUC ($\mathrm{\char 37\relax}$) | $3.34$ | $5.73$

Furthermore, results demonstrate the decent efficiency of the model: using
only a $20\text{\,}\mathrm{\char 37\relax}$ subset of the screening data, it
is possible to accurately identify $49\text{\,}\mathrm{\char 37\relax}$ of all
active compounds (refer to fig. 3(b)). This efficiency is further highlighted
when the provided data is increased to $50\text{\,}\mathrm{\char 37\relax}$,
at which point approximately $73\text{\,}\mathrm{\char 37\relax}$ of active
molecules are correctly identified. This finding highlights the potential of
the model to streamline the screening process by requiring significantly less
molecules to achieve meaningful results. Recall is a crucial metric in this
context as it measures the ability of the model to correctly identify all
relevant instances (in this case active molecules).

Figure 4: Tanimoto similarity of active compounds from the confirmatory
screen. Shown is a random selection of 50 compounds, to highlight the
structural similarities and differences.

Furthermore, this study includes a systematic evaluation of whether the
observed results can be attributed to the structural similarity of the
compounds. For this purpose, the Tanimoto similarity is calculated as shown in
fig. 4. This analysis reveals a significantly diverse structural distribution
among the compounds, suggesting that similarities are not driving the observed
performance. In fig. 5 the number of unique Murcko scaffolds from confirmatory
screen relative to percentage from primary screen is shown. TwinBooster is
capable to reach an area under the curve of $60.77\text{\,}\mathrm{\char
37\relax}$ compared to $50\text{\,}\mathrm{\char 37\relax}$ in case of a
random selection, i.e. with a $25\text{\,}\mathrm{\char 37\relax}$ subset of
the screening data it is possible to capture $>39\text{\,}\mathrm{\char
37\relax}$ of unique Murcko scaffolds.Landrum et al. (2020) The ability of
TwinBooster to discriminate and identify a wide range of potentially active
compounds highlights its utility in streamlining the drug discovery process,
paving the way for more targeted and expedient identification of promising
compounds.

##  4 Conclusion and Outlook

In this study, we have taken a step towards improving the capabilities of drug
discovery and development. By integrating a fine-tuned LLM with the Barlow
Twins architecture, and further employing GBM for training and prediction, our
zero-shot TwinBooster framework represents a novel approach to molecular
property prediction, particularly in scenarios where data is scarce.Zbontar et
al. (2021)

The effectiveness of TwinBooster in zero-shot learning tasks, as evidenced by
its performance on the FS-Mol benchmark as well as in a HTS case study,
suggests that this methodology could be an important tool in the early stages
of drug discovery. It is able to outperform the best performing few-shot
baseline at 16 support molecules provided by FS-Mol.Stanley et al. (2021)

In addition, TwinBooster’s prediction model can help improving efficiency and
economics of drug discovery. By diminishing the number of molecules that
require empirical screening, it accelerates the research timeline and cuts
down both time and associated costs, thereby enhancing the overall efficiency
and cost-effectiveness of drug development.

However, it is important to recognise the complexity of predicting very
different assays in comparison to the training data in a zero-shot scenario.
While the results are promising, they represent one step in an ongoing journey
of scientific exploration and innovation.

## Data and Code Availability

The system used for computational work has an AMD Ryzen Threadripper PRO
5995WX CPU with 64/128 cores/threads with $1024\text{\,}\mathrm{GB}$ RAM.
Additionally, the server is equipped with a NVIDIA RTX 4090 GPU with
$24\text{\,}\mathrm{GB}$ VRAM.

The fine-tuned DeBERTa V3 model on the PubChem corpus is available on
HuggingFace https://huggingface.co/mschuh/PubChemDeBERTa, as well as the
augmented version https://huggingface.co/mschuh/PubChemDeBERTa-augmented.

As a Python package, it can be installed using $ pip install twinbooster. The
code is available on GitHub https://github.com/maxischuh/TwinBooster, where
you can also find the model data.

## References

  * Zhang et al. (2019) Jin Zhang, Daniel Mucs, Ulf Norinder, and Fredrik Svensson.  LightGBM: An Effective and Scalable Algorithm for Prediction of Chemical Toxicity–Application to the Tox21 and Mutagenicity Data Sets.  _Journal of Chemical Information and Modeling_ , 59(10):4150–4158, October 2019.  ISSN 1549-9596.  10.1021/acs.jcim.9b00633.
  * Deng et al. (2023) Jianyuan Deng, Zhibo Yang, Hehe Wang, Iwao Ojima, Dimitris Samaras, and Fusheng Wang.  A systematic study of key elements underlying molecular property prediction.  _Nature Communications_ , 14(1):6395, October 2023\.  ISSN 2041-1723.  10.1038/s41467-023-41948-6.
  * Chithrananda et al. (2020) Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar.  ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction, October 2020.
  * Jain et al. (2021) Sankalp Jain, Vishal B. Siramshetty, Vinicius M. Alves, Eugene N. Muratov, Nicole Kleinstreuer, Alexander Tropsha, Marc C. Nicklaus, Anton Simeonov, and Alexey V. Zakharov.  Large-Scale Modeling of Multispecies Acute Toxicity End Points Using Consensus of Multitask Deep Learning Methods.  _Journal of Chemical Information and Modeling_ , 61(2):653–663, February 2021.  ISSN 1549-9596.  10.1021/acs.jcim.0c01164.
  * Walter et al. (2022) Moritz Walter, Luke N. Allen, Antonio de la Vega de León, Samuel J. Webb, and Valerie J. Gillet.  Analysis of the benefits of imputation models over traditional QSAR models for toxicity prediction.  _Journal of Cheminformatics_ , 14(1):1–27, December 2022.  ISSN 1758-2946.  10.1186/s13321-022-00611-w.
  * Yang et al. (2019) Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, Andrew Palmer, Volker Settels, Tommi Jaakkola, Klavs Jensen, and Regina Barzilay.  Analyzing Learned Molecular Representations for Property Prediction.  _Journal of Chemical Information and Modeling_ , 59(8):3370–3388, August 2019.  ISSN 1549-9596.  10.1021/acs.jcim.9b00237.
  * Shen and Nicolaou (2019) Jie Shen and Christos A. Nicolaou.  Molecular property prediction: Recent trends in the era of artificial intelligence.  _Drug Discovery Today: Technologies_ , 32–33:29–36, December 2019.  ISSN 1740-6749.  10.1016/j.ddtec.2020.05.001.
  * Gaulton et al. (2012) Anna Gaulton, Louisa J. Bellis, A. Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, and John P. Overington.  ChEMBL: A large-scale bioactivity database for drug discovery.  _Nucleic Acids Research_ , 40(Database issue):D1100–1107, January 2012.  ISSN 1362-4962.  10.1093/nar/gkr777.
  * Seidl et al. (2023) Philipp Seidl, Andreu Vall, Sepp Hochreiter, and Günter Klambauer.  Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Language, June 2023.
  * Merkwirth and Lengauer (2005) Christian Merkwirth and Thomas Lengauer.  Automatic Generation of Complementary Descriptors with Molecular Graph Networks.  _Journal of Chemical Information and Modeling_ , 45(5):1159–1168, September 2005.  ISSN 1549-9596.  10.1021/ci049613b.
  * Young et al. (2018) Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria.  Recent Trends in Deep Learning Based Natural Language Processing [Review Article].  _IEEE Computational Intelligence Magazine_ , 13(3):55–75, August 2018.  ISSN 1556-6048.  10.1109/MCI.2018.2840738.
  * Kim et al. (2023) Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin A. Shoemaker, Paul A. Thiessen, Bo Yu, Leonid Zaslavsky, Jian Zhang, and Evan E. Bolton.  PubChem 2023 update.  _Nucleic Acids Research_ , 51(D1):D1373–D1380, January 2023.  ISSN 1362-4962.  10.1093/nar/gkac956.
  * He et al. (2021) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen.  DeBERTa: Decoding-enhanced BERT with Disentangled Attention, October 2021.
  * He et al. (2023) Pengcheng He, Jianfeng Gao, and Weizhu Chen.  DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing, March 2023.
  * Zbontar et al. (2021) Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow Twins: Self-Supervised Learning via Redundancy Reduction, June 2021.
  * Landrum et al. (2020) Greg Landrum, Paolo Tosco, Brian Kelley, sriniker, gedeck, NadineSchneider, Riccardo Vianello, Ric, Andrew Dalke, Brian Cole, AlexanderSavelyev, Matt Swain, Samo Turk, Dan N, Alain Vaucher, Eisuke Kawashima, Maciej Wójcikowski, Daniel Probst, guillaume godin, David Cosgrove, Axel Pahl, JP, Francois Berenger, strets123, JLVarjo, Noel O’Boyle, Patrick Fuller, Jan Holst Jensen, Gianluca Sforna, and DoliathGavid.  Rdkit/rdkit: 2020_03_1 (Q1 2020) Release.  Zenodo, March 2020.
  * Hadsell et al. (2006) R. Hadsell, S. Chopra, and Y. LeCun.  Dimensionality Reduction by Learning an Invariant Mapping.  In _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , volume 2, pages 1735–1742, June 2006.  10.1109/CVPR.2006.100.
  * Biau and Scornet (2016) Gérard Biau and Erwan Scornet.  A random forest guided tour.  _TEST_ , 25(2):197–227, June 2016.  ISSN 1863-8260.  10.1007/s11749-016-0481-7.
  * Jiang et al. (2021) Dejun Jiang, Zhenxing Wu, Chang-Yu Hsieh, Guangyong Chen, Ben Liao, Zhe Wang, Chao Shen, Dongsheng Cao, Jian Wu, and Tingjun Hou.  Could graph neural networks learn better molecular representation for drug discovery? A comparison study of descriptor-based and graph-based models.  _Journal of Cheminformatics_ , 13(1):1–23, December 2021.  ISSN 1758-2946.  10.1186/s13321-020-00479-8.
  * Boldini et al. (2023) Davide Boldini, Francesca Grisoni, Daniel Kuhn, Lukas Friedrich, and Stephan A. Sieber.  Practical guidelines for the use of gradient boosting for molecular property prediction.  _Journal of Cheminformatics_ , 15(1):1–13, December 2023.  ISSN 1758-2946.  10.1186/s13321-023-00743-7.
  * Stanley et al. (2021) Megan Stanley, John F. Bronskill, Krzysztof Maziarz, Hubert Misztela, Jessica Lanini, Marwin Segler, Nadine Schneider, and Marc Brockschmidt.  FS-Mol: A Few-Shot Learning Dataset of Molecules.  In _Thirty-Fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , August 2021\.
  * Breiman (2017) Leo Breiman.  _Classification and Regression Trees_.  Routledge, New York, October 2017.  ISBN 978-1-315-13947-0.  10.1201/9781315139470.
  * Zheng et al. (2021) Shuyu Zheng, Jehad Aldahdooh, Tolou Shadbahr, Yinyin Wang, Dalal Aldahdooh, Jie Bao, Wenyu Wang, and Jing Tang.  DrugComb update: A more comprehensive drug sensitivity data repository and analysis portal.  _Nucleic Acids Research_ , 49(W1):W174–W184, July 2021.  ISSN 0305-1048.  10.1093/nar/gkab438.
  * Winter et al. (2019) Robin Winter, Floriane Montanari, Frank Noé, and Djork-Arné Clevert.  Learning continuous and data-driven molecular descriptors by translating equivalent chemical representations.  _Chemical Science_ , 10(6):1692–1701, February 2019.  ISSN 2041-6539.  10.1039/C8SC04175J.
  * Tishby and Zaslavsky (2015) Naftali Tishby and Noga Zaslavsky.  Deep Learning and the Information Bottleneck Principle, March 2015\.
  * Xian et al. (2020) Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata.  Zero-Shot Learning – A Comprehensive Evaluation of the Good, the Bad and the Ugly, September 2020.
  * Mendez et al. (2019) David Mendez, Anna Gaulton, A Patrícia Bento, Jon Chambers, Marleen De Veij, Eloy Félix, María Paula Magariños, Juan F Mosquera, Prudence Mutowo, Michal Nowotka, María Gordillo-Marañón, Fiona Hunter, Laura Junco, Grace Mugumbate, Milagros Rodriguez-Lopez, Francis Atkinson, Nicolas Bosc, Chris J Radoux, Aldo Segura-Cabrera, Anne Hersey, and Andrew R Leach.  ChEMBL: Towards direct deposition of bioassay data.  _Nucleic acids research_ , 47(D1):D930–D940, January 2019.  ISSN 1362-4962.  10.1093/nar/gky1075.
  * van Rossum (1995) Guido van Rossum.  Python tutorial.  (R 9526), January 1995.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush.  Transformers: State-of-the-Art Natural Language Processing.  In Qun Liu and David Schlangen, editors, _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online, October 2020\. Association for Computational Linguistics.  10.18653/v1/2020.emnlp-demos.6.
  * Akiba et al. (2019) Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama.  Optuna: A Next-generation Hyperparameter Optimization Framework, July 2019.
  * Jelinek et al. (1977) F. Jelinek, R. L. Mercer, L. R. Bahl, and J. K. Baker.  Perplexity—a measure of the difficulty of speech recognition tasks.  _The Journal of the Acoustical Society of America_ , 62(S1):S63, August 1977.  ISSN 0001-4966.  10.1121/1.2016299.
  * Meister and Cotterell (2021) Clara Meister and Ryan Cotterell.  Language Model Evaluation Beyond Perplexity.  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5328–5339, Online, August 2021. Association for Computational Linguistics.  10.18653/v1/2021.acl-long.414.
  * Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, March 2015.
  * Agarap (2019) Abien Fred Agarap.  Deep Learning using Rectified Linear Units (ReLU), February 2019.
  * Ramachandran et al. (2017) Prajit Ramachandran, Barret Zoph, and Quoc V. Le.  Searching for Activation Functions, October 2017.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  PyTorch: An Imperative Style, High-Performance Deep Learning Library, December 2019.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter.  Decoupled Weight Decay Regularization, January 2019.
  * Ke et al. (2017) Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu.  LightGBM: A Highly Efficient Gradient Boosting Decision Tree.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.
  * Lindauer et al. (2022) Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhopf, René Sass, and Frank Hutter.  SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization, February 2022.
  * Fawcett (2006) Tom Fawcett.  An introduction to ROC analysis.  _Pattern Recognition Letters_ , 27(8):861–874, June 2006.  ISSN 01678655.  10.1016/j.patrec.2005.10.010.
  * Tharwat (2020) Alaa Tharwat.  Classification assessment methods.  _Applied Computing and Informatics_ , 17(1):168–192, January 2020.  ISSN 2210-8327.  10.1016/j.aci.2018.08.003.
  * 201 (2018) _Learning from Imbalanced Data Sets_.  Springer Science+Business Media, New York, NY, 2018.  ISBN 978-3-319-98073-7.
  * Branco et al. (2015) Paula Branco, Luis Torgo, and Rita Ribeiro.  A Survey of Predictive Modelling under Imbalanced Distributions, May 2015.
  * Piryonesi and El-Diraby (2020) S. Madeh Piryonesi and Tamer E. El-Diraby.  Data Analytics in Asset Management: Cost-Effective Prediction of the Pavement Condition Index.  _Journal of Infrastructure Systems_ , 26(1):04019036, March 2020.  ISSN 1943-555X.  10.1061/(ASCE)IS.1943-555X.0000512.
  * Cortés-Ciriano and Bender (2019) Isidro Cortés-Ciriano and Andreas Bender.  Concepts and Applications of Conformal Prediction in Computational Drug Discovery, August 2019.
  * Stone (1974) M. Stone.  Cross-Validatory Choice and Assessment of Statistical Predictions.  _Journal of the Royal Statistical Society. Series B (Methodological)_ , 36(2):111–147, 1974.  ISSN 0035-9246.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, May 2019.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang.  BioBERT: A pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240, February 2020\.  ISSN 1367-4803, 1367-4811.  10.1093/bioinformatics/btz682.
  * Virtanen et al. (2020) Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C. J. Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, and Paul van Mulbregt.  SciPy 1.0: Fundamental algorithms for scientific computing in Python.  _Nature Methods_ , 17(3):261–272, March 2020\.  ISSN 1548-7105.  10.1038/s41592-019-0686-2.
  * Deerwester et al. (1990) Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman.  Indexing by latent semantic analysis.  _Journal of the American Society for Information Science_ , 41(6):391–407, 1990.  ISSN 1097-4571.  10.1002/(SICI)1097-4571(199009)41:6¡391::AID-ASI1¿3.0.CO;2-9.
  * Bonferroni (1936) C.E. Bonferroni.  _Teoria Statistica Delle Classi e Calcolo Delle Probabilità_.  Pubblicazioni Del R. Istituto Superiore Di Scienze Economiche e Commerciali Di Firenze. Seeber, 1936.
  * Flaherty et al. (2014) Daniel P. Flaherty, Justin R. Miller, Danielle M. Garshott, Michael Hedrick, Palak Gosalia, Yujie Li, Monika Milewski, Eliot Sugarman, Stefan Vasile, Sumeet Salaniwal, Ying Su, Layton H. Smith, Thomas D. Y. Chung, Anthony B. Pinkerton, Jeffrey Aubé, Michael U. Callaghan, Jennifer E. Golden, Andrew M. Fribley, and Randal J. Kaufman.  Discovery of Sulfonamidebenzamides as Selective Apoptotic CHOP Pathway Activators of the Unfolded Protein Response.  _ACS Medicinal Chemistry Letters_ , 5(12):1278–1283, December 2014.  10.1021/ml5003234.
  * Lekstrom-Himes and Xanthopoulos (1998) Julie Lekstrom-Himes and Kleanthis G. Xanthopoulos.  Biological Role of the CCAAT/Enhancer-binding Protein Family of Transcription Factors *.  _Journal of Biological Chemistry_ , 273(44):28545–28548, November 1998.  ISSN 0021-9258, 1083-351X.  10.1074/jbc.273.44.28545.
  * Ghosh et al. (2012) Arindam P. Ghosh, Barbara J. Klocke, Mary E. Ballestas, and Kevin A. Roth.  CHOP Potentially Co-Operates with FOXO3a in Neuronal Cells to Regulate PUMA and BIM Expression in Response to ER Stress.  _PLOS ONE_ , 7(6):e39586, June 2012.  ISSN 1932-6203.  10.1371/journal.pone.0039586.
  * Oyadomari and Mori (2004) S. Oyadomari and M. Mori.  Roles of CHOP/GADD153 in endoplasmic reticulum stress.  _Cell Death & Differentiation_, 11(4):381–389, April 2004.  ISSN 1476-5403.  10.1038/sj.cdd.4401373.
  * Vij et al. (2008) Neeraj Vij, Martha O. Amoako, Steven Mazur, and Pamela L. Zeitlin.  CHOP Transcription Factor Mediates IL-8 Signaling in Cystic Fibrosis Bronchial Epithelial Cells.  _American Journal of Respiratory Cell and Molecular Biology_ , 38(2):176–184, February 2008.  ISSN 1044-1549.  10.1165/rcmb.2007-0197OC.
  * Goodall et al. (2010) Jane C. Goodall, Changxin Wu, Yongsheng Zhang, Louise McNeill, Lou Ellis, Vladimir Saudek, and J. S. Hill Gaston.  Endoplasmic reticulum stress-induced transcription factor, CHOP, is crucial for dendritic cell IL-23 expression.  _Proceedings of the National Academy of Sciences_ , 107(41):17698–17703, October 2010.  10.1073/pnas.1011736107.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011\.

##  Appendix A Appendix – Results

###  A.1 Zero-shot benchmark

The zero-shot performances of 10 replicates are averaged and compared with PN
from FS-Mol.Stanley et al. (2021) Only means per task are used for the
Wilcoxon test as only 5 replicates were performed on the PN at 16 support
molecules from FS-Mol. The Wilcoxon test yields $p\simeq 0.0478$, for PR_AUC
and DPR_AUC.Virtanen et al. (2020) Significance is indicated at $p<\alpha$,
where $\alpha=0.05$. In addition, only the $122$ intersecting tasks (tasks
that are present in both FS-Mol and PubChem and therefore have an assay
description) are evaluated.

The AID are
$52168988188389912151394154017085215215215215215215215215215216896896896896896896896896896898818818818818818818818818818818838838838838838838838838838838998998998998998998998998998991215121512151215121512151215121512151215139413941394139413941394139413941394139415401540154015401540154015401540154015401708170817081708170817081708170817081708521521521521521521521521521521521521521521521521521521521521521521521$.

Comparing zero-shot performances of 10 replicates with or without conformal
prediction on FS-Mol yields $p\simeq 0.0020$, for ROC_AUC, PR_AUC and DPR_AUC
on a Wilcoxon test.Virtanen et al. (2020) Significance is indicated at
$p<\alpha$, where $\alpha=0.05$.

###  A.2 Ablation study

The LSA modelDeerwester et al. (1990) is pre-trained on the PubChem corpus.
This process is carried out in a manner similar to the approach used in Seidl
et al., using the Python packages TfidfVectorizer and TruncatedSVD.Seidl et
al. (2023); Pedregosa et al. (2011) The resulting text embeddings for each
bioassay have a dimensionality of 355.

In parallel, the hyperparameter optimisation for the Gradient Boosting Machine
(GBM) is performed in a similar way as described in section 2.2.2.

The $p$-values of the statistical tests using the Wilcoxon test with
Bonferroni correction are provided in table 11.Virtanen et al. (2020);
Bonferroni (1936)

Table 11: All $p$-value results are tested pairwise using the Wilcoxon test
with Bonferroni correction of different ablation experiments on FS-
Mol.Virtanen et al. (2020); Bonferroni (1936) 10 replicates each are
performed. Significance is indicated at $p<\alpha$, where
$\alpha=\frac{0.05}{3}$. If $p\geq\alpha$, values are in italics.

TwinBooster TwinBooster ECFP + PubChemDeBERTa $p$-values vs. vs. vs. ECFP +
PubChemDeBERTa ECFP + LSA ECFP + LSA ROC_AUC 0.1055 0.0020 0.0020 PR_AUC
0.0059 0.0020 0.0020 DPR_AUC 0.0059 0.0020 0.0020

###  A.3 Case study

Figure 5 shows an enrichment of the number of unique Murcko scaffolds
identified by TwinBooster and their proportional representation from the
primary screen. This indicates that TwinBooster does not prioritise certain
scaffolds, but enriches the scaffold diversity compared to random scaffold
selection. Highlighting the importance of scaffold variability in early drug
development, this increase in diversity is critical to the identification of
potential leads.

Figure 5: Number of detected unique Murcko scaffolds in relation to the
percentage of scaffolds from the primary screen.

##### Primary screen

This represents title, description and protocol used for the zero-shot
prediction of the primary screen (AID
2732444https://pubchem.ncbi.nlm.nih.gov/bioassay/2732), based of the PubChem
entry:Kim et al. (2023)

> “HTS for small molecule inhibitors of CHOP to regulate the unfolded protein
> response to ER stress. Many genetic and environmental diseases result from
> defective protein folding within the secretory pathway so that aberrantly
> folded proteins are recognized by the cellular surveillance system and
> retained within the endoplasmic reticulum (ER). Under conditions of
> malfolded protein accumulation, the cell activates the Unfolded Protein
> Response (UPR) to clear the malfolded proteins, and if unsuccessful,
> initiates a cell death response. Preliminary studies have shown that CHOP is
> a crucial factor in the apoptotic arm of the UPR; XBP1 activates genes
> encoding ER protein chaperones and thereby mediates the adaptive UPR
> response to increase clearance of malfolded proteins. Inhibition of CHOP is
> hypothesized to enhance survival by preventing UPR programmed cell death.
> There are currently no known small molecule CHOP inhibitors either for
> laboratory or clinical use. To identify small molecule inhibitors of the UPR
> pathway, mediated by CHOP, a cell-based luciferase reporter assay using
> stably transfected CHO-K1 cells with luciferase driven by the CHOP promoter
> has been developed. The assay have been optimized and validated in 384-well
> format and used to screen for inhibitors of tunicamycin-induced CHOP in HTS.
> These identified compounds will have potential therapeutic application to
> diverse disease states ranging from diabetes, Alzheimer’s disease, and
> Parkinson’s disease, to hemophilia, lysosomal storage diseases, and alpha-1
> antitrypsin deficiency. Reagents: 1\. Cell line: CHO-CHOP cells with a
> luciferase reporter driven by the CHOP promoter (provided by assay PI) 2\.
> Cell growth media (Ham’s F12 + Glutamax, 10% FBS, 1X non-essential amino
> acids, and penicillin:streptomycin) (Invitrogen) 3\. Tunicamycin
> (Calbiochem) 4\. SteadyGlo reagent (Promega) Protocol: 1\. 40 uL of medium
> containing CHO-CHOP cells (3000-4000) were dispensed to 384 well white
> opaque plates (Corning #3570) using a Multidrop combi (Thermo-Fisher
> Scientific). Plates were then incubated for 24 hrs at 37 degrees C, 5% CO2.
> 2\. 0.5 uL of library compounds (1 mM in DMSO) was added to wells using
> Sciclone (Caliper LifeSciences). The final concentration of compound is 10
> uM. 3\. 10 uL of fresh medium containing tunicamycin (Tm) (2.0 ug/ml, final
> concentration,) was then added and the plates were incubated for 15-18 hrs.
> 4\. Medium was aspirated with an Elx405 plate washer (BioTek), leaving 10 uL
> of medium in the well. 10 uL of Steady-Glo was added to each well using a
> multildrop combi. 5\. Luminescence signal was measured on an Envision
> Multilable plate reader (PerkinElmer).”