# Is Meta-training Really Necessary for Molecular Few-Shot Learning ?

Philippe Formont  Hugo Jeannin  Pablo Piantanida  Ismail Ben Ayed

###### Abstract

Few-shot learning has recently attracted significant interest in drug
discovery, with a recent, fast-growing literature mostly involving convoluted
meta-learning strategies. We revisit the more straightforward fine-tuning
approach for molecular data, and propose a regularized quadratic-probe loss
based on the the Mahalanobis distance. We design a dedicated block-coordinate
descent optimizer, which avoid the degenerate solutions of our loss.
Interestingly, our simple fine-tuning approach achieves highly competitive
performances in comparison to state-of-the-art methods, while being applicable
to black-box settings and removing the need for specific episodic pre-training
strategies. Furthermore, we introduce a new benchmark to assess the robustness
of the competing methods to domain shifts. In this setting, our fine-tuning
baseline obtains consistently better results than meta-learning methods.

Machine Learning, ICML

Figure 1:  Performances of different few-shot learning methods evaluated with
library screening (HTS) out-of-distribution tasks from the Therapeutic Data
Commons dataset. Models are evaluated with the hitrate in the top-5% of the
predictions (left). A ranking of the models on these tasks across different
proportions of selected molecules reveals the competitiveness of fine-tuning
baselines such as the linear probe (l-probe) or our proposed quadratic probe
(q-probe) in comparisons to meta-learning methods (right). Complete results
are available in the section 4.

##  1 Introduction

Drug discovery is a process that aims to answer the following question: “How
do we design molecules with biological activity on a specific therapeutic
target and other necessary properties to become a drug?”. To identify such
compounds, scientists rely on a combination of experimental and computational
methods (Xu & Hagler, 2002; Eckert & Bajorath, 2007). This lengthy and costly
process takes on average 12 years and 1.8 billion dollars to bring a new drug
to the market (Pammolli et al., 2019; Arrowsmith, 2011). To reduce the cost
associated with this process and to speed up the development of new drugs,
machine learning methods are often used to guide the discovery of new
compounds (Gilmer et al., 2017; Chen et al., 2018). A common task for these
models is to predict the activity of a compound on a given target exploiting
the compound’s structural or physio-chemical properties, referred to as
quantitative structure-activity relationship (QSAR) (Tropsha, 2010). QSAR
models can also be used as oracles to guide generative algorithms into
generating compounds while optimizing their properties (Bengio et al., 2021;
Blaschke et al., 2020; Gao et al., 2022).

Low data regime. The data exploited to train those models is scarce and costly
to generate, as it results from numerous wet lab experiments. In practice, the
size of the datasets corresponding to a novel target fluctuates along each
project. Often, only up to a few hundred molecules are available to train the
models. For this reason, various approaches for few-shot-learning in drug
discovery have been proposed, attempting to provide reliable predictions while
using a small amount of labelled data, where meta-learning seems to appear as
the default and best solution (Schimunek et al., 2023; Chen et al., 2023;
Altae-Tran et al., 2017).

Meta-learning. Meta-learning corresponds to a family of methods pre-trained to
’learn to learn’ from a small set of labelled examples (support set), directly
conditioning their predictions on this support set (Vinyals et al., 2016).

Hence, these algorithms must pretrain a model specific to few-shot learning
tasks. This implies that different pre-trained models must be maintained to
perform few-shot and standard classification. Furthermore, most meta-learning
algorithms use specific training procedures, as the training and test-time
scenarios must be similar. Controlling the training procedure of these methods
is hence necessary to use them. This can be problematic in a black-box
setting, where only the predictions of the model can be queried through an
API, and the model’s weights cannot be accessed (Ji et al., 2021; Ouali et
al., 2023). Using these methods in a black-box scenario would require imposing
the training procedure on the server, which is not always possible, or would
limit the number of models that could be used. Yet this black-box setting is
especially valuable in drug discovery, where data is highly sensitive, and
sharing a dataset or a pre-trained model on a private dataset is not always
possible.

Competitive fine-tuning methods. Fine-tuning methods correspond to using a
model pre-trained in a standard supervised setting and fine-tuning it for few-
shot classification. Furthermore, different backbones, pre-trained with
different pre-training objectives, can be adapted with such methods. Hence, a
wider range of models can be used. In other domains, such as computer vision,
meta-learning methods were shown to obtain only marginal improvements over
more straightforward fine-tuning methods, using models pre-trained in a
standard supervised setting (Chen et al., 2019; Luo et al., 2023). Some of
these results were made possible thanks to the availability of foundational
models pre-trained on large datasets available in these fields (Radford et
al., 2021; Brown et al., 2020). However, developing such foundational models
is a non-trivial task in the context of drug discovery, and is still an active
research field (Seidl et al., 2023; Ahmad et al., 2022). In drug discovery, it
was observed that fine-tuning approaches yielded subpar performances (Stanley
et al., 2021), a finding seemingly at odds with existing results in vision
applications.

### Contributions

We re-visit fine-tuning black-box models using a simple Multitask backbone to
perform QSAR few-shot learning. Our contributions could be summarized as
follows:

  * •

We evaluate the standard linear probe, a baseline often overlooked in the
molecular few-shot learning literature.

  * •

We generalize this standard cross-entropy baseline to a more expressive
quadratic-probe loss based on the Mahalanobis distance and the class
covariance matrices. We examine our loss from an optimization perspective,
observe degenerate solutions and mitigate the issue with a block-coordinate
descent and a dedicated regularization, which yields closed-form solutions at
each optimization step. Interestingly, our quadratic probe achieves highly
competitive performances as the size of the support set increases.

  * •

Finally, in addition to evaluations on the standard FS-mol benchmark, we built
two out-of-domain evaluation setups to evaluate the models’ robustness to
domain shifts. Our fine-tuning baselines obtain consistently better results
than meta-learning algorithms.

##  2 Related Work

Notations. Assume that we are given a base dataset
$\mathcal{D}_{\text{base}}=\\{\mathcal{D}_{\text{base}}^{t}\\}_{t\in\mathbf{T}_{\text{base}}}$,
where $\mathbf{T}_{\text{base}}$ is a set of tasks. For a task $t$, we have
$\mathcal{D}_{\text{base}}^{t}=\\{{\mathbf{x}}_{i},y_{i}\\}_{0\leq i\leq
N^{t}_{\text{base}}}$, where ${\mathbf{x}}_{i}$ is a molecule and
$y_{i}\in\\{0,1\\}$ denotes its label.

We then have access to a test dataset defined similarly:
$\mathcal{D}_{\text{test}}=\\{\mathcal{D}_{\text{test}}^{t}\\}_{t\in\mathbf{T}_{\text{novel}}}$
where $\mathbf{T}_{\text{novel}}$ is the set of new tasks
($\mathbf{T}_{\text{novel}}\cap\mathbf{T}_{\text{base}}=\emptyset$). Finally,
for a task $t$, $\mathcal{D}_{\text{test}}^{t}$ is divided into a set of
labelled data that we will use to adapt the model, i.e., support set
$\mathcal{S}_{t}=\\{{\mathbf{x}}_{i},y_{i}\\}_{0\leq i\leq
N_{\text{support}}}$, and an unlabelled set whose labels are to be predicted,
i.e., query set $\mathcal{Q}_{t}=\\{{\mathbf{x}}_{i},y_{i}\\}_{0\leq i\leq
N_{\text{query}}}$.

The goal of the few-shot algorithms we study is to leverage the supervision
information from the support samples to predict the labels within the query
set.

Meta-learning. Training a meta-learning model, referred to as meta-training,
differs from a classical supervised procedure by the fact that the training
set is divided into tasks. Each task is divided into a support set and a query
set that will be fed into the model to perform its training. This data
processing is specific to meta-learning.

The work of Stanley et al. (2021) introduced a benchmark for molecular few-
shot learning, FS-mol, and evaluated various methods. For instance, the
authors trained an adaptation of Protonet (Snell et al., 2017) with a graph
neural network (GNN). Besides, the authors pre-trained a graph neural network
(GNN) with the MAML algorithm (Finn et al., 2017), a model aiming at learning
a good initialization of the model’s weights, such that the model can be
quickly adapted to new tasks. Building on the MAML paradigm, the property-
aware relation network (PAR) (Wang et al., 2021) builds a similarity graph
between the molecules of the support set and the queried molecule and extracts
from this graph the molecule’s predicted class. In recent work, Chen et al.
(2023) proposed a method separating the parameters $\theta_{meta}$ that are
updated during meta-training, while the other parameters
$\theta_{\textrm{adapt}}$ are only updated during the adaptation. The back-
propagation on each task is done by using Cauchy’s implicit function theorem,
which requires estimating the hessian of the loss
$\frac{\partial^{2}\mathcal{L}_{\textrm{support}}}{\partial\theta_{\textrm{adapt}}^{2}}$
on the support set. Finally, some work evaluated the effect of leveraging a
large set of context molecules to construct similarity measures between the
support set’s molecules and the queried molecule using modern hopfield
networks (Schimunek et al., 2023).

Fine-tuning methods. Among the baselines presented by Stanley et al. (Stanley
et al., 2021), some of them used a backbone trained without relying on meta-
training. First, the authors fine-tuned a GNN trained with a multitask
objective on the support set of the novel task, consisting of a GNN common
trunk and a 2-layer perceptron task-specific head. At test time, both the
common-trunk and the task-specific parameters are fine-tuned on the support
set. A second approach used the molecular attention transformer (MAT)
(Maziarka et al., 2019) and fine-tuned it on the support set of the novel
task. Both methods achieved performances on FS-mol below the aforementioned
meta-learning methods. Furthermore, they need to fine-tune the whole model,
which can be computationally expensive and requires the user to access the
model’s weights at test time. Hence, these methods cannot be directly used in
a black-box setting. Finally, the work of Seidl et al. (2023) proposed a
multi-modal model leveraging both the molecular graph and a textual
representation of the assays to make predictions, enabling the creation of
zero-shot models. The authors evaluated the model on FS-mol with a linear
probe. However, to train their model, the support set of all tasks was
integrated into the pre-training set, which breaks the supposition
$\mathbf{T}_{novel}\cap\mathbf{T}_{base}=\emptyset$. This evaluation
corresponds to an assessment of the model’s ability to generalize on
previously seen tasks with few labels, not the model’s ability to adapt to
unseen tasks.

##  3 Methods

###  3.1 Model pre-training

Various methods exist to pre-train models for drug discovery (Hu* et al.,
2020), which can usually be divided into supervised and self-supervised
methods. Supervised pre-training refers to methods training a model on a
dataset of molecules relying on the association between the input and labels.
In contrast, self-supervised pre-training refers to methods training a model
on a dataset of molecules without using or knowing the labels. While self-
supervised pre-training enables the use of larger data sets, supervised pre-
training tends to be more effective when the labels used during training
correspond to assays similar to those of the downstream task (Sun et al.,
2022). In this work, we pursue the supervised setting via a multitask model
based on the training set of the FS-mol benchmark (Stanley et al., 2021). We
focus our work on QSAR modelling, evaluating the ability of the pre-training
backbone to adapt to new tasks.

The GNN backbone $f_{\theta}$ will take the molecular graph and the
fingerprints as input, based on a principal neighbourhood aggregation model
(Corso et al., 2020). We add a task-specific head $g_{\phi}$ to the backbone,
a one-layer binary classifier during the pre-training. Once the training is
finished, we only keep the parameters $\theta$ of the backbone. The molecular
embeddings passed to the few-shot classifier are obtained by

${\mathbf{z}}_{i}=\frac{f_{\theta}({\mathbf{x}}_{i})}{\lVert
f_{\theta}({\mathbf{x}}_{i})\rVert}=\frac{f_{\theta}({\mathcal{G}}_{i},{\mathbf{f}}_{i})}{\lVert
f_{\theta}({\mathcal{G}}_{i},{\mathbf{f}}_{i})\rVert},$

with ${\mathcal{G}}_{i}$ denoting the molecular graph, ${\mathbf{f}}_{i}$ the
molecular fingerprint.

###  3.2 Multitask Linear Probing

As explained in the section 2, multitask models were shown to be outperformed
by most other methods (Stanley et al., 2021). Still, we show that we can
obtain better classification capabilities by following the paradigm of linear
probing (Chen et al., 2019). Linear probing corresponds to a model training
the parameters ${\mathbf{w}}_{k},\lVert{\mathbf{w}}_{k}\rVert=1$ for each
class $k\in C=\\{0,\ldots,n_{\textrm{class}}\\}$. The predicted probability of
a molecule ${\mathbf{x}}_{i}$ to belong to the class $k\in C$ is then given
by:

$p_{i,k}=\frac{\exp{\left(\tau\langle{\mathbf{z}}_{i},{\mathbf{w}}_{k}\rangle+b_{k}\right)}}{\sum_{k^{\prime}\in
C}\exp{\left(\tau\langle{\mathbf{z}}_{i},{\mathbf{w}}_{k^{\prime}}\rangle+b_{k^{\prime}}\right)}}$
(1)

where $b_{k}$ is the model’s bias, $\tau$ is a temperature hyperparameter, and
for the clarity of the notation, we omitted the dependency on the task
${\mathbf{t}}$. The linear probe’s parameters are then optimized to minimize
the cross-entropy loss on the support set.

###  3.3 Quadratic probing

Motivation We propose an extension of the aforementioned linear probe, which
is more competitive as the support set grows. Instead of the cosine similarity
used in the linear probe, we propose to use the Mahalanobis distance between
the query points and the class prototypes:

$\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{k}\rVert^{2}_{\Sigma_{k}^{-1}}=\left({\mathbf{z}}_{i}-{\mathbf{w}}_{k}\right)^{T}\Sigma_{k}^{-1}\left({\mathbf{z}}_{i}-{\mathbf{w}}_{k}\right),$
(2)

where $\Sigma_{k}$ is a positive semi-definite (PSD) matrix. For simplicity,
we will refer to ${\mathbf{M}}_{k}\triangleq\Sigma_{k}^{-1}$ as the precision
matrix of the distribution to simplify the notations. The predicted
probability of a molecule ${\mathbf{x}}_{i}$ to belong to the class $k\in C$
can be then obtained as:

$\centering
p_{i,k}\triangleq\frac{\exp{\left(-\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{k}\rVert^{2}_{{\mathbf{M}}_{k}}\right)}}{\sum_{k^{\prime}\in
C}\exp{\left(-\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{k^{\prime}}\rVert^{2}_{{\mathbf{M}}_{k}}\right)}}.\@add@centering$
(3)

The Mahalanobis distance measures the distance between a point and a normal
distribution of mean ${\mathbf{w}}_{k}$ and covariance matrix $\Sigma_{k}$.
This distance is more expressive than the cosine similarity, as it considers
the covariance matrix of the class. In fact, one can see it as generalization
of the linear probe, which corresponds to choosing a covariance matrix
proportional to identity. Figure 2 depicts an illustration of the difference
between linear probe and our quadratic probe.

Figure 2:  Illustration of the difference between using the linear probe
approach (left) and classifiers based on a Mahalanobis distance (right).

In the next section, we show that directly minimizing the cross-entropy loss
over ${\mathbf{M}}_{k}$ yields degenerate solutions, in which the Frobenius
norms of the covariance matrices go to infinity. To mitigate this issue, we
propose a surrogate objective, which corresponds approximately to augmenting
the cross-entropy loss with a regularization term based on the Frobinous norms
of the covariance matrices, while yielding closed-form solutions at each
optimization step w.r.t ${\mathbf{M}}_{k}$. We proceed with block-coordinate
descent approach to optimize our regularized quadratic-probe loss, updating
the parameters $\\{{\mathbf{w}}_{k}\\}_{k\in\\{0,1\\}}$ and
$\\{{\mathbf{M}}_{k}\\}_{k\in\\{0,1\\}}$ separately.

Gradient descent. Optimizing the parameters of the quadratic probe is a
challenging task, as the estimator’s number of parameters is much larger than
the linear probe’s. The first greedy option would be to optimize
${\mathbf{M}}_{k}$ by gradient descent on the support set. We propose to
optimize $\mathbf{D}_{k}\triangleq{\mathbf{M}}_{k}^{-1/2}$
(${\mathbf{M}}_{k}=\mathbf{D}_{k}^{T}\mathbf{D}_{k}=\Sigma_{k}^{-1}$) as a
parameter of the model to ensure that ${\mathbf{M}}_{k}$ is PSD.

However, by following this procedure, the model has a high chance of over-
fitting. In particular, if the classes are linearly separable, the model will
find a minimum of the cross-entropy loss, pushing
$\lVert{\mathbf{M}}_{k}\rVert$ to become arbitrarily large.

######  Proposition 3.1.

Let $\Theta=\\{{\mathbf{w}}_{k},{\mathbf{M}}_{k}\\}_{k\in\\{0,1\\}}$, and we
will note $p_{\Theta}(k|{\mathbf{z}})=p_{i,k}$ as described in Equation 3 (to
highlight the dependency to the parameters $\Theta$).

If the samples from both classes are linearly separable, we can construct a
set of parameters,
$\Theta(\lambda)=\\{{\mathbf{w}}_{k},{\mathbf{M}}_{k}(\lambda)\\}_{k\in\\{0,1\\}}$
such that:

$\begin{cases}\forall
i\in\\{0,\ldots,N\\},\quad\lim_{\lambda\to+\infty}\mathcal{L}_{ce}({\mathbf{z}}_{i},y_{i},{\Theta(\lambda)})=0,\\\
\forall
k\in\\{0,1\\},\quad\lim_{\lambda\to+\infty}\lVert{\mathbf{M}}_{k}(\lambda)\rVert_{F}=+\infty,\end{cases}$

where $\mathcal{L}_{ce}({\mathbf{z}}_{i},y_{i},\Theta)=-\log
p_{\Theta}(y_{i}|{\mathbf{z}}_{i})$ is the point-wise cross-entropy loss w.r.t
our model.

The proof of proposition 3.1 is relegated to Appendix A.0.1. This proposition
relies on the assumption that the data points are linearly separable. In the
few-shot learning setting, this assumption is often verified. Indeed, as the
data dimension is much larger than the number of data points, Cover’s theorem
(Cover, 1965) states that the points are almost certainly linearly separable.
Another argument supporting the assumption of linear separability comes from
an approximation often made in high-dimensional spaces:

$\text{if }d\gg N,\quad\forall i,j\in\\{0,\ldots,N\\}^{2}\quad
z_{i}^{t}z_{j}=0.$

Then, the vector $v=\sum_{i=0}^{N}y_{i}z_{i}-(1-y_{i})z_{i}$ defines a
hyperplane separating data points in both classes.

Block coordinate optimization and modified loss.

Instead of optimizing the parameters of the quadratic probe by gradient
descent, we optimize the parameters of the model with a block coordinate
descent, updating the parameters $\\{{\mathbf{w}}_{k}\\}_{k\in\\{0,1\\}}$ and
$\\{{\mathbf{M}}_{k}\\}_{k\in\\{0,1\\}}$ separately. In particular, we will
perform one gradient descent step on $\\{{\mathbf{w}}_{k}\\}_{k\in\\{0,1\\}}$
using the cross-entropy loss and then minimize a modified loss w.r.t
$\\{{\mathbf{M}}_{k}\\}_{k\in\\{0,1\\}}$, that does not suffer from the issue
described in proposition 3.1. In the following section,
$\\{{\mathbf{w}}_{k}\\}_{k\in\\{0,1\\}}$ is considered fixed.

Decomposing the cross-entropy in two parts, we first identify the term pushing
the parameters ${\mathbf{M}}_{k}$ to be arbitrarily large. The cross-entropy
loss of the quadratic probe can be divided into two terms:

$\mathcal{L}_{ce}(\mathcal{S},\Theta)=\frac{1}{|\mathcal{S}|}\sum_{i\in\mathcal{S}}\left(f_{1}\left({\mathbf{z}}_{i},y_{i},\Theta\right)+f_{2}\left({\mathbf{z}}_{i},\Theta\right)\right),$
(4)

where

$\displaystyle
f_{1}\left({\mathbf{z}}_{i},y_{i},\Theta\right)\triangleq\lVert{\mathbf{z}}-{\mathbf{w}}_{y_{i}}\rVert^{2}_{{\mathbf{M}}_{y_{i}}},$
(5) $\displaystyle
f_{2}\left({\mathbf{z}}_{i},\Theta\right)\triangleq\log\left(\sum_{k\in\\{0,1\\}}\exp\left(-\lVert{\mathbf{z}}-{\mathbf{w}}_{k}\rVert^{2}_{{\mathbf{M}}_{k}}\right)\right).$

######  Remark 3.2.

For all $i\in\mathcal{S}$, $f_{1}\left({\mathbf{z}}_{i},y_{i},\Theta\right)$
is minimized when $\lVert{\mathbf{M}}_{k}\rVert_{F}\to 0$, and
$f_{2}\left({\mathbf{z}}_{i},y_{i},\Theta\right)$ as
$\lVert{\mathbf{M}}_{k}\rVert_{F}\to+\infty$.

The function $f_{2}$ is hence responsible for the degenerate solutions
obtained while minimizing the cross-entropy loss. We propose to replace
$f_{2}$ with a different function $\tilde{f}_{2}$, playing the same role
pushing $\lVert{\mathbf{M}}_{k}\rVert_{F}$ towards infinity, but with a less
aggressive behaviour. The new loss obtained $f_{1}+\tilde{f}_{2}$ admits a
closed-form solution on ${\mathbf{M}}_{k}$, for a fixed ${\mathbf{w}}_{k}$,
which corresponds to the estimator of the empirical covariance matrix’s
inverse. The expression of $\tilde{f}_{2}$ is then:

$\tilde{f}_{2}\left(\Theta\right)\triangleq-\sum_{k\in\\{0,1\\}}\frac{\lvert\mathcal{S}_{k}\rvert\log\det{({\mathbf{M}}_{k})}}{\lvert\mathcal{S}\rvert}.$

######  Proposition 3.3.

Both $f_{2}$ and $\tilde{f}_{2}$ are optimized when the norms of
$\\{{\mathbf{M}}_{k}\\}_{k\in\\{0,1\\}}$ approach infinity. When for all
$k\in\\{0,1\\}$, the eigenvalues of ${\mathbf{M}}_{k}$ grow to infinity:

$\displaystyle f_{2}\left({\mathbf{z}}_{i},\Theta\right)$
$\displaystyle=O(\sum_{k\in\\{0,1\\}}\lVert{\mathbf{M}}_{k}\rVert_{F}),$ (6)
$\displaystyle\text{and}\quad\tilde{f}_{2}\left({\mathbf{z}}_{i},\Theta\right)$
$\displaystyle=O(\sum_{k\in\\{0,1\\}}\log\lVert{\mathbf{M}}_{k}\rVert_{F}),$

And the resulting loss $f_{1}+\tilde{f}_{2}$ admits a closed-form solution,
for $k\in\\{0,1\\}$:

${\mathbf{M}}_{k}=\left(\frac{1}{\lvert\mathcal{S}_{y}\rvert}\sum_{i\in\mathcal{S}_{y}}\left({\mathbf{z}}_{i}-{\mathbf{w}}_{k}\right)\left({\mathbf{z}}_{i}-{\mathbf{w}}_{k}\right)^{T}\right)^{-1}.$
(7)

######  Remark 3.4.

Minimizing $f_{1}+\tilde{f}_{2}$ is equivalent to maximizing the log-
likelihood of a multivariate Gaussian distribution with mean
${\mathbf{w}}_{k}$ and covariance matrix $\Sigma_{k}$, explaining why the
closed form solution we obtain corresponds to the estimator of the empirical
covariance matrix’s inverse.

The function $\tilde{f}_{2}$ can hence be seen as an upper bound on $f_{2}$
when $\lVert{\mathbf{M}}_{k}\rVert_{F}$ becomes large, for $k=0,1$. Indeed,
$\tilde{f}_{2}$ exerts less pressure on expanding ${\mathbf{M}}_{k}$ to larger
matrices. This replacement can approximately be seen as adding the
regularization terms
$\alpha\lVert{\mathbf{M}}_{k}\rVert_{F}-\beta\log{\lVert{\mathbf{M}}_{k}\rVert_{F}}$
to the loss ($\alpha$ and $\beta$ are constants).

Finally, we train the quadratic probe by updating ${\mathbf{M}}_{k}$ using the
closed-form solution obtained above (Equation 7) and performing a gradient
step on ${\mathbf{w}}_{k}$ using the cross-entropy loss. Since the estimation
of the covariance matrix can be unstable, we use a shrinkage parameter
$\lambda\in[0,1]$ to stabilize the estimation of the covariance matrix
($\Sigma_{k}\triangleq(1-\lambda)\Sigma_{k,\text{empirical}}+\lambda{\mathbf{I}}$).
We consider $\lambda$ as a hyperparameter of the model to be optimized (see
Appendix A.1 for further details).

Table 1: Results on FS-mol ($\Delta AUCPR$) of each method for support set
sizes of 16, 32, 64 and 128 (averaged over 10 runs). The best models, or
models whose average performances are in the confidence interval of the best
model, are highlighted in bold.
Model-type | Model | $|\mathcal{S}|=16$ | $|\mathcal{S}|=32$ | $|\mathcal{S}|=64$ | $|\mathcal{S}|=128$
---|---|---|---|---|---
baselines | kNN |  0.051$\pm$ 0.005 |  0.076$\pm$ 0.006 |  0.102$\pm$ 0.007 |  0.139$\pm$ 0.007
GNN-ST |  0.021$\pm$ 0.005 |  0.027$\pm$ 0.005 |  0.031$\pm$ 0.005 |  0.052$\pm$ 0.006
RF |  0.093$\pm$ 0.007 |  0.125$\pm$ 0.008 |  0.163$\pm$ 0.009 |  0.213$\pm$ 0.009
SimSearch |  0.113$\pm$ 0.009 |  0.142$\pm$ 0.008 |  0.169$\pm$ 0.008 |  0.210$\pm$ 0.008
meta-learning |  PAR NeurIPS2021 |  0.130$\pm$ 0.009 |  0.140$\pm$ 0.009 |  0.147$\pm$ 0.009 |  0.161$\pm$ 0.009
GNN-MAML |  0.160$\pm$ 0.009 |  0.167$\pm$ 0.009 |  0.174$\pm$ 0.009 |  0.192$\pm$ 0.009
CNP |  0.187$\pm$ 0.011 |  0.192$\pm$ 0.011 |  0.195$\pm$ 0.011 |  0.208$\pm$ 0.011
Protonet NeurIPS2021 |  0.206$\pm$ 0.009 |  0.242$\pm$ 0.009 |  0.271$\pm$ 0.009 |  0.301$\pm$ 0.009
MHNFS ICLR 2023 |  0.205$\pm$ 0.010 |  0.215$\pm$ 0.009 |  0.219$\pm$ 0.009 |  0.231$\pm$ 0.009
ADKF-IFT ICLR 2023 | 0.231$\pm$ 0.009 | 0.263$\pm$ 0.010 | 0.287$\pm$ 0.010 | 0.318$\pm$ 0.010
fine-tuning | MAT |  0.052$\pm$ 0.005 |  0.069$\pm$ 0.006 |  0.092$\pm$ 0.007 |  0.136$\pm$ 0.009
GNN-MT |  0.112$\pm$ 0.006 |  0.144$\pm$ 0.007 |  0.177$\pm$ 0.008 |  0.223$\pm$ 0.009
CLAMP ICLR 2023 |  0.202$\pm$ 0.009 |  0.228$\pm$ 0.009 |  0.248$\pm$ 0.009 |  0.271$\pm$ 0.009
Linear Probe | 0.224$\pm$ 0.010 |  0.252$\pm$ 0.010 |  0.273$\pm$ 0.010 |  0.301$\pm$ 0.009
Quadratic Probe | 0.227$\pm$ 0.010 | 0.255$\pm$ 0.010 | 0.279$\pm$ 0.010 | 0.310$\pm$ 0.010

##  4 Experimental Results

###  4.1 FS-mol benchmark

Dataset. We evaluate our methods on the FS-mol benchmark (Stanley et al.,
2021). This benchmark aims to classify molecules into active/inactive binary
classes for a given therapeutic target, divided into 5000 tasks (40 in the
validation and 157 in the test sets). The tasks the models are evaluated on
are balanced, with on average 47% of active molecules, a value ranging from
30% to 50%.

Methods compared. We compare our methods to various meta-learning algorithms.
First, we compare our approach to the baselines presented in the paper of
Stanley et al. (2021). Their methods included: GNN-MAML (Finn et al., 2017),
and Protonet (Snell et al., 2017). We considered the methods presented in the
paper of Chen et al. (Chen et al., 2023), ADKT-IFT, the Property Aware
Relation network (PAR) (Wang et al., 2021), and conditional neural process
(CNP) on this benchmark (Garnelo et al., 2018). Finally, we evaluated MHNFS
(Schimunek et al., 2023) on this benchmark. We used the author’s scripts to
train the model while implementing the evaluation scripts, which were
unavailable on the author’s repository.

We report the results of the fine-tuning baselines already proposed by Stanley
et al.(2021), such as fully fine-tuning of a molecular attention transformer
(MAT) (Maziarka et al., 2019) and GNN-MT, their multitask fine-tuning
approach. We also re-evaluated the few-shot adaptation capabilities of CLAMP
(Seidl et al., 2023) on the FS-mol benchmark (without adding the support set
of each task to the model’s pre-training set).

We also consider simple baselines: the authors of FS-mol proposed to train a
random forest classifier and a kNN on the support set’s molecular fingerprint
(Stanley et al., 2021). The authors also propose to train a full single-task
GNN on the support set (GNN-ST). While proposing MHNFS, Schimuneck et al.
(2023) evaluated the Similarity Search, a standard chemoinformatic relying
only on the Tanimoto similarity between fingerprints (Bender et al., 2004;
Cereto-Massagué et al., 2015).

Evaluation procedure. To compare few-shot learning methods, all models were
pre-trained on FS-mol’s training set. Models are evaluated using the
improvement of the average precision compared to a random classifier ($\Delta
AUCPR$). Hyper-parameters are selected through a hyper-parameter search on the
validation set across all support set sizes. All hyperparameters are kept
constant over each support set size, except for the optimal numbers of epochs
for our fine-tuning baselines, which are fitted for each support set size on
the validation set. We present the results for support set sizes of 16, 32, 64
and 128 (larger support set sizes would discard some tasks from the test set).

Results  Table 1 presents the results of the different methods on the FS-mol
test set. First, our evaluation of CLAMP achieves results similar to the ones
reported in the original paper with 16 labelled points and obtains better
results with 64 and 128 labelled points. On the other hand, we did not succeed
in reproducing MHNFS’s results with our implementation.

Figure 3: Mean performances of each method on the FS-mol benchmark, for
support set sizes of 16, 32, 64 and 128.

The performances of both the linear and quadratic probes outperform the other
fine-tuning baselines. Using a basic pre-training procedure, our fine-tuning
methods also provide strong baselines for the few-shot adaptation of pre-
trained models, to estimate the relevance of a new pre-training procedure in
this setup.

As expected, the quadratic probe outperforms the linear probe as the support
set size increases. This result is expected since the quadratic probe is a
model fine-tuning more parameters than the linear probe. Additionally, the
quadratic probe consistently reaches performances that are competitive with
ADKT-IFT’s results at each support set size.

Figure 3 provides another visualization of the results, by using autorank
(Herbold, 2020) to rank the methods based on their performances. This metric
not only takes into account the mean performances of the methods, but also the
consistency of the results across the different models, where we can see how
fine-tuning methods achieve comparable results to meta-learning models while
being more straightforward and more convenient in a black-box setting.

###  4.2 Ablation on the Free optimisation of $\Sigma_{k}$

Figure 4: Evolution of the model’s performance on the validation set during
the few-shot adaptation. Free-opt refers to performing a gradient descent on
${\mathbf{M}}_{k}$, Free-opt-reg adds a regularisation on the norm of
${\mathbf{M}}_{k}$. ($|\mathcal{S}|=64$)

As explained in section 3, we optimize $\Sigma_{k}$ by taking the empirical
covariance matrix of the embeddings, considering $\textbf{w}_{k}$ is the mean
of the distribution. Figure 4 highlights the difference between optimizing
$\Sigma_{k}$ with our method or using gradient descent steps to minimize the
cross-entropy loss (Free-opt). Our quadratic probe (q-probe) outperforms the
Free-Optimisation approach, and is less sensitive to the number of epochs the
model should be optimized with. This sensitivity to the number of epochs
becomes especially problematic under domain shifts, as this hyperparameter is
fitted on a validation set that is not representative of the test set.
Finally, by adding a regularization on the norm of ${\mathbf{M}}_{k}$ to the
Free-Opt approach, the model becomes less sensitive to the number of epochs
but achieves lower performances than our proposed optimization approach.

###  4.3 Impact of Domain Shifts

To further investigate the robustness of the different methods, we wish to
evaluate the models’ performances when the tasks they are assessed on differ
from their training set. We designed scenarios extracted from the therapeutic
data commons platform (Huang et al., 2021) and the LIT-PCBA dataset (Tran-
Nguyen et al., 2020), in which the tasks yield a domain shift compared to the
base dataset.

First, the FS-mol dataset mainly comprises balanced tasks, as explained in the
previous section, while the tasks we will consider in this assessment are more
imbalanced. The robustness of those different methods to class imbalance is
essential, as molecules are often inactive in practice. We also propose to
assess the ability of these few-shot models to retrieve active compounds from
a large molecular library (library screening) while having access to a few
labelled examples. The ability to rank and identify the molecules based on
their chances of being active can also be helpful while using generative
models, generating large batches of molecules, to identify the most promising
ones.

In this section, we focus our work on the top-performing models on FS-mol, the
linear and quadratic probes (l-probe, q-probe), protonet, adkt-ift, clamp, and
the standard chemo-informatic method: simsearch.

####  4.3.1 QSAR modelling with imbalanced class distribution

Datasets. In this section, we will consider the Drug-Target-Interaction
datasets available on the Therapeutic Data Common (Huang et al., 2021)
platform. Drug Target interaction refers to a family of tasks that aim at
predicting the affinity of a molecule to a given target using the molecular
graph of the molecule, and the sequence of the target (which we discarded in
our experiments)

The datasets chosen to evaluate the models are the KIBA (Tang et al., 2014),
DAVIS (Davis et al., 2011), and BindingDB_Kd (Liu et al., 2006) datasets.
These datasets are mainly composed of kinases, the most common target type in
the FS-mol benchmark. Hence, this section will help evaluate the robustness of
the different methods to a prior shift (difference in the label’s marginal
distribution), keeping the target of the task in-domain. The mean class
imbalance, measured as the absolute deviation of the proportion of positive
examples to a perfectly balanced scenario, is the lowest in the KIBA dataset
(0.12), while the BindingDB_Kd and DAVIS datasets possess more imbalanced
tasks (0.19 and 0.21, respectively).

Evaluation procedure. As some tasks are composed of a few molecules (all tasks
in the DAVIS dataset are composed of 68 molecules), we only considered support
sets of 16 molecules in this evaluation. All models were trained on the FS-mol
benchmark, and their hyperparameters were also chosen on this benchmark. We
used the $\Delta AUCPR$ metric to evaluate our models.

Figure 5: Performances of various methods on the DTI tasks. The fine-tuning
baselines outperform the meta-learning methods when the tasks become more
imbalanced.

Results. The results of the different methods on those datasets are presented
in Figure 5. While the state-of-the-art meta-learning methods achieve
competitive results on the KIBA dataset, our baselines outperform all other
methods when the distribution of the labels becomes more imbalanced. The
performances of the linear and quadratic probes are similar, which is expected
as we performed this evaluation on a support set size of 16, when both methods
achieved similar results on the main benchmark. These results seem to indicate
that the fine-tuning baselines are more robust to prior shifts than the meta-
learning methods. This can be explained by the fact that the base dataset is
divided into tasks during meta-training, giving the model information on the
expected marginal distributions of positive and negative classes.

On the other hand, experiments on the BindingDB_Ki and IC50 datasets are
presented in the Appendix A.2.1, where our fine-tuning baselines do not
outperform meta-learning models, but where no model significantly outperforms
the similarity search baseline.

####  4.3.2 Library screening

Datasets. Finding potential hits (active molecules) in a large library of
compounds, referred to as library screening, is a common step in drug
discovery, where scientists can then buy the most promising compounds from a
commercial molecular database (Gentile et al., 2022). In these tasks, the
proportion of $"$hits$"$ (active compounds) is usually very low (1̃%),
yielding a much more significant prior shift than in the previous section.
Besides, on the opposite of the section above, the targets considered in the
datasets we will use are not kinases (see Appendix A.2.2), including another
type of domain shift. To perform this evaluation, we will use 6 HTS datasets
from the therapeutic data commons platform (Huang et al., 2021) and ten
datasets from the LIT-PCBA database (Tran-Nguyen et al., 2020); details on the
task choice can be found in the Appendix A.2.2.

Evaluation procedure. The experiments were run by considering support sets of
16, 32, 64 and 128 molecules, with a proportion of hits of both 5% and 10% in
the support set, leaving all remaining hits in the query set. We considered
the top-k% hitrate (proportion of true positive in the predicted positive when
selecting k-% of the library) as a metric. Taking the average hitrate over all
datasets can be problematic, as the average performance of the models largely
varies from one dataset to the other. We hence decided to report the average
rank of each model using autorank (Herbold, 2020), over 20 runs.

Figure 6:  Average ranking performances of each method on the HTS tasks
according to the percentage of the dataset selected. The quadratic probe
obtains the best results when the support set’s size is small, while the
similarity search is the best model with larger support sets.

Results. Figure 6 presents the average rank of each model on the HTS tasks
over 20 runs, for the different support set sizes, and proportion of molecules
selected.

First, with a limited number of known labels (support set size of 16 or 32),
our method consistently outperforms all other baselines, a trend observed
across varying proportions of selected molecules. CLAMP also proves to be
competitive in this low-data regime, surpassing the performance of meta-
learning methods and the linear probe with larger support sets. Indeed, the
linear probe exhibits strong performance with a support set size of 16, but
its performance significantly diminishes as the support set size increases.

As the support set size increases, meta-learning methods demonstrate
comparable results to the quadratic probe and CLAMP, until protonet
outperforms the quadratic probe with a support set size of 128. Notably, the
similarity search, employed as our baseline for these experiments, is
outperformed by most other methods when the support set comprises only a few
molecules. However, it becomes the best model when the support set is
extensive, exhibiting superior performances, especially when few molecules are
selected in each dataset (1 to 5%).

The robustness of the similarity search in these tasks can be attributed to
the fact that it does not rely on any pre-training, and is hence not affected
by the domain shift between the HTS and FS-mol datasets.

##  5 Summary and Concluding Remarks

While meta-learning methods consistently yield highly competitive results in
molecular few-shot learning, it is essential to acknowledge that fine-tuning
baselines have the potential to achieve comparable outcomes. In particular, we
explored the intricacies of optimizing an extension of the linear probe, a
quadratic probe. This extension is capable of competing with state-of-the-art
meta-learning methods while relying on a model pre-trained with a standard
multitask objective.

Moreover, our findings demonstrate that these fine-tuning baselines exhibit
heightened robustness when confronted with a shift in the label distribution.
This resilience further enhances their practical utility in diverse scenarios.
Regrettably, these baseline approaches are often overlooked in the existing
literature in this field. Notably, these methods possess distinctive
advantages, particularly in a black-box setting, where they can be readily
benchmarked upon the availability of a pre-trained model.

The anonymized codebase used in this paper is available at:
https://github.com/Fransou/is-meta-learning-necessary

##  6 Impact Statements

This paper presents work that aims to advance the field of Machine Learning.
Our work has many potential societal consequences. We hope our work can be
used to help drug discovery pipelines, help discover treatments, and
accelerate the process (which would limit the amount of animal testing, for
instance).

## References

  * Ahmad et al. (2022) Ahmad, W., Simon, E., Chithrananda, S., Grand, G., and Ramsundar, B.  Chemberta-2: Towards chemical foundation models, 2022.
  * Altae-Tran et al. (2017) Altae-Tran, H., Ramsundar, B., Pappu, A. S., and Pande, V.  Low data drug discovery with one-shot learning.  _ACS Central Science_ , 3(4):283–293, 2017.  doi: 10.1021/acscentsci.6b00367.  URL https://doi.org/10.1021/acscentsci.6b00367.  PMID: 28470045.
  * Arrowsmith (2011) Arrowsmith, J.  Phase II failures: 2008–2010.  _Nature Reviews Drug Discovery_ , 10(5):328–329, May 2011.  ISSN 1474-1784.  doi: 10.1038/nrd3439.
  * Bender et al. (2004) Bender, A., Mussa, H. Y., Glen, R. C., and Reiling, S.  Similarity Searching of Chemical Databases Using Atom Environment Descriptors (MOLPRINT 2D): Evaluation of Performance.  _Journal of Chemical Information and Computer Sciences_ , 44(5):1708–1718, September 2004.  ISSN 0095-2338.  doi: 10.1021/ci0498719.  URL https://doi.org/10.1021/ci0498719.  Publisher: American Chemical Society.
  * Bengio et al. (2021) Bengio, E., Jain, M., Korablyov, M., Precup, D., and Bengio, Y.  Flow network based generative models for non-iterative diverse candidate generation.  In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), _Advances in Neural Information Processing Systems_ , volume 34, pp. 27381–27394. Curran Associates, Inc., 2021.
  * Blaschke et al. (2020) Blaschke, T., Arús-Pous, J., Chen, H., Margreitter, C., Tyrchan, C., Engkvist, O., Papadopoulos, K., and Patronov, A.  Reinvent 2.0: An ai tool for de novo drug design.  _Journal of Chemical Information and Modeling_ , 60(12):5918–5922, 2020.  doi: 10.1021/acs.jcim.0c00915.
  * Brown et al. (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D.  Language Models are Few-Shot Learners.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), _Advances in Neural Information Processing Systems_ , volume 33, pp. 1877–1901. Curran Associates, Inc., 2020.
  * Butkiewicz et al. (2013) Butkiewicz, M., Lowe, E. W., Mueller, R., Mendenhall, J. L., Teixeira, P. L., Weaver, C. D., and Meiler, J.  Benchmarking Ligand-Based Virtual High-Throughput Screening with the PubChem Database.  _Molecules_ , 18(1), January 2013.  ISSN 1420-3049.  doi: 10.3390/molecules18010735.  URL https://www.mdpi.com/1420-3049/18/1/735.
  * Cereto-Massagué et al. (2015) Cereto-Massagué, A., Ojeda, M. J., Valls, C., Mulero, M., Garcia-Vallvé, S., and Pujadas, G.  Molecular fingerprint similarity search in virtual screening.  _Methods_ , 71:58–63, 2015.  ISSN 1046-2023.  doi: https://doi.org/10.1016/j.ymeth.2014.08.005.  URL https://www.sciencedirect.com/science/article/pii/S1046202314002631.  Virtual Screening.
  * Chen et al. (2018) Chen, H., Engkvist, O., Wang, Y., Olivecrona, M., and Blaschke, T.  The rise of deep learning in drug discovery.  _Drug Discovery Today_ , 23(6):1241–1250, 2018.  ISSN 1359-6446.  doi: https://doi.org/10.1016/j.drudis.2018.01.039.  URL https://www.sciencedirect.com/science/article/pii/S1359644617303598.
  * Chen et al. (2023) Chen, W., Tripp, A., and Hernández-Lobato, J. M.  Meta-learning adaptive deep kernel gaussian processes for molecular property prediction.  In _International Conference on Learning Representations_ , 2023.
  * Chen et al. (2019) Chen, W.-Y., Liu, Y.-C., Kira, Z., Wang, Y.-C. F., and Huang, J.-B.  A closer look at few-shot classification.  In _International Conference on Learning Representations_ , 2019.
  * Corso et al. (2020) Corso, G., Cavalleri, L., Beaini, D., Liò, P., and Veličković, P.  Principal neighbourhood aggregation for graph nets.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Cover (1965) Cover, T. M.  _Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition_.  1965\.
  * Davis et al. (2011) Davis, M. I., Hunt, J. P., Herrgard, S., Ciceri, P., Wodicka, L. M., Pallares, G., Hocker, M., Treiber, D. K., and Zarrinkar, P. P.  Comprehensive analysis of kinase inhibitor selectivity.  _Nature Biotechnology_ , 29(11):1046–1051, November 2011.  ISSN 1546-1696.  doi: 10.1038/nbt.1990.  Number: 11 Publisher: Nature Publishing Group.
  * Eckert & Bajorath (2007) Eckert, H. and Bajorath, J.  Molecular similarity analysis in virtual screening: foundations, limitations and novel approaches.  _Drug Discovery Today_ , 12(5):225–233, 2007.  ISSN 1359-6446.  doi: https://doi.org/10.1016/j.drudis.2007.01.011.  URL https://www.sciencedirect.com/science/article/pii/S1359644607000529.
  * Finn et al. (2017) Finn, C., Abbeel, P., and Levine, S.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ , ICML’17, pp. 1126–1135. JMLR.org, 2017.
  * Gao et al. (2022) Gao, W., Fu, T., Sun, J., and Coley, C. W.  Sample efficiency matters: A benchmark for practical molecular optimization.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022.  URL https://openreview.net/forum?id=yCZRdI0Y7G.
  * Garnelo et al. (2018) Garnelo, M., Rosenbaum, D., Maddison, C., Ramalho, T., Saxton, D., Shanahan, M., Teh, Y. W., Rezende, D., and Eslami, S. M. A.  Conditional neural processes.  In Dy, J. and Krause, A. (eds.), _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pp. 1704–1713. PMLR, 10–15 Jul 2018.  URL https://proceedings.mlr.press/v80/garnelo18a.html.
  * Gentile et al. (2022) Gentile, F., Yaacoub, J. C., Gleave, J., Fernandez, M., Ton, A.-T., Ban, F., Stern, A., and Cherkasov, A.  Artificial intelligence–enabled virtual screening of ultra-large chemical libraries with deep docking.  _Nature Protocols_ , 17(3):672–697, March 2022.  ISSN 1750-2799.  doi: 10.1038/s41596-021-00659-2.  URL https://www.nature.com/articles/s41596-021-00659-2.  Number: 3 Publisher: Nature Publishing Group.
  * Gilmer et al. (2017) Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E.  Neural message passing for quantum chemistry.  _CoRR_ , 2017.
  * Herbold (2020) Herbold, S.  Autorank: A python package for automated ranking of classifiers.  _Journal of Open Source Software_ , 5(48):2173, 2020.  doi: 10.21105/joss.02173.  URL https://doi.org/10.21105/joss.02173.
  * Hu* et al. (2020) Hu*, W., Liu*, B., Gomes, J., Zitnik, M., Liang, P., Pande, V., and Leskovec, J.  Strategies for pre-training graph neural networks.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=HJlWWJSFDH.
  * Huang et al. (2021) Huang, K., Fu, T., Gao, W., Zhao, Y., Roohani, Y. H., Leskovec, J., Coley, C. W., Xiao, C., Sun, J., and Zitnik, M.  Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_ , 2021.
  * Huang et al. (2016) Huang, R., Xia, M., Nguyen, D.-T., Zhao, T., Sakamuru, S., Zhao, J., Shahane, S. A., Rossoshek, A., and Simeonov, A.  Tox21challenge to build predictive models of nuclear receptor and stress response pathways as mediated by exposure to environmental chemicals and drugs.  _Frontiers in Environmental Science_ , 3, 2016.  ISSN 2296-665X.  doi: 10.3389/fenvs.2015.00085.  URL https://www.frontiersin.org/articles/10.3389/fenvs.2015.00085.
  * Ji et al. (2021) Ji, D., Logan, R. L., Smyth, P., and Steyvers, M.  Active bayesian assessment of black-box classifiers.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(9):7935–7944, May 2021.  doi: 10.1609/aaai.v35i9.16968.  URL https://ojs.aaai.org/index.php/AAAI/article/view/16968.
  * Liu et al. (2006) Liu, T., Lin, Y., Wen, X., Jorissen, R. N., and Gilson, M. K.  Bindingdb: a web-accessible database of experimentally determined protein–ligand binding affinities.  _Nucleic Acids Research_ , 35:D198–D201, 12 2006.  ISSN 0305-1048.  doi: 10.1093/nar/gkl999.
  * Luo et al. (2023) Luo, X., Wu, H., Zhang, J., Gao, L., Xu, J., and Song, J.  A closer look at few-shot classification again.  In _International Conference on Machine Learning_ , 2023.
  * Maziarka et al. (2019) Maziarka, L., Danel, T., Mucha, S., Rataj, K., Tabor, J., and Jastrzebski, S.  Molecule attention transformer.  In _Graph Representation Learning Workshop at NeurIPS 2019_ , 2019.
  * Ouali et al. (2023) Ouali, Y., Bulat, A., Matinez, B., and Tzimiropoulos, G.  Black box few-shot adaptation for vision-language models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 15534–15546, October 2023.
  * Pammolli et al. (2019) Pammolli, F., Righetto, L., Abrignani, S., Pani, L., Pelicci, P., and Rabosio, E.  The endless frontier? the recent upsurge of r&d productivity in pharmaceuticals, 06 2019.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I.  Learning transferable visual models from natural language supervision, 2021.
  * Schimunek et al. (2023) Schimunek, J., Seidl, P., Friedrich, L., Kuhn, D., Rippmann, F., Hochreiter, S., and Klambauer, G.  Context-enriched molecule representations improve few-shot drug discovery.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Seidl et al. (2023) Seidl, P., Vall, A., Hochreiter, S., and Klambauer, G.  Enhancing activity prediction models in drug discovery with the ability to understand human language.  _Proceedings of the 40th International Conference on Machine Learning (ICML)_ , July 2023.
  * Snell et al. (2017) Snell, J., Swersky, K., and Zemel, R.  Prototypical networks for few-shot learning.  In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.
  * Stanley et al. (2021) Stanley, M., Bronskill, J. F., Maziarz, K., Misztela, H., Lanini, J., Segler, M., Schneider, N., and Brockschmidt, M.  FS-mol: A few-shot learning dataset of molecules.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.
  * Sun et al. (2022) Sun, R., Dai, H., and Yu, A. W.  Does gnn pretraining help molecular representation?  In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), _Advances in Neural Information Processing Systems_ , volume 35, pp. 12096–12109. Curran Associates, Inc., 2022.
  * Tang et al. (2014) Tang, J., Szwajda, A., Shakyawar, S., Xu, T., Hintsanen, P., Wennerberg, K., and Aittokallio, T.  Making sense of large-scale kinase inhibitor bioactivity data sets: A comparative and integrative analysis.  _Journal of Chemical Information and Modeling_ , 54(3):735–743, 2014.  doi: 10.1021/ci400709d.
  * Touret et al. (2020) Touret, F., Gilles, M., Barral, K., Nougairède, A., van Helden, J., Decroly, E., de Lamballerie, X., and Coutard, B.  In vitro screening of a FDA approved chemical library reveals potential inhibitors of SARS-CoV-2 replication.  _Scientific Reports_ , 10(1):13093, August 2020.  ISSN 2045-2322.  doi: 10.1038/s41598-020-70143-6.
  * Tran-Nguyen et al. (2020) Tran-Nguyen, V.-K., Jacquemard, C., and Rognan, D.  Lit-pcba: An unbiased data set for machine learning and virtual screening.  _Journal of Chemical Information and Modeling_ , 60(9):4263–4273, 2020.  doi: 10.1021/acs.jcim.0c00155.
  * Tropsha (2010) Tropsha, A.  Best practices for qsar model development, validation, and exploitation.  _Molecular Informatics_ , 29(6-7):476–488, 2010.  doi: https://doi.org/10.1002/minf.201000061.  URL https://onlinelibrary.wiley.com/doi/abs/10.1002/minf.201000061.
  * Vinyals et al. (2016) Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al.  Matching networks for one shot learning.  In _Advances in Neural Information Processing Systems_ , pp. 3630–3638, 2016.
  * Wang et al. (2016) Wang, Y., Bryant, S. H., Cheng, T., Wang, J., Gindulyte, A., Shoemaker, B. A., Thiessen, P. A., He, S., and Zhang, J.  PubChem BioAssay: 2017 update.  _Nucleic Acids Research_ , 45(D1):D955–D963, 11 2016.  ISSN 0305-1048.  doi: 10.1093/nar/gkw1118.  URL https://doi.org/10.1093/nar/gkw1118.
  * Wang et al. (2021) Wang, Y., ABUDUWEILI, A., quanming yao, and Dou, D.  Property-aware relation networks for few-shot molecular property prediction.  In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), _Advances in Neural Information Processing Systems_ , 2021.
  * Wu et al. (2018) Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Geniesse, C., Pappu, A. S., Leswing, K., and Pande, V.  MoleculeNet: A Benchmark for Molecular Machine Learning.  October 2018.  doi: 10.48550/arXiv.1703.00564.  URL http://arxiv.org/abs/1703.00564.
  * Xu & Hagler (2002) Xu, J. and Hagler, A.  Chemoinformatics and drug discovery.  _Molecules_ , 7(8):566–600, 2002.  ISSN 1420-3049.  doi: 10.3390/70800566.  URL https://www.mdpi.com/1420-3049/7/8/566.

##  Appendix A Appendix

####  A.0.1 Proofs

Proof of Proposition 3.1

###### Proof.

The samples from both classes are linearly separable. As such, there exists a
vector ${\mathbf{v}}$ such that

$\forall
i\in\\{0,\ldots,N\\},\begin{cases}{\mathbf{v}}^{T}{\mathbf{z}}_{i}\textgreater
b&\text{ if }y_{i}=1\\\ {\mathbf{v}}^{T}{\mathbf{z}}_{i}\textless b&\text{ if
}y_{i}=0,\\\ \end{cases}$ (8)

where $b$ is a constant, and we take ${\mathbf{v}}$ such that
$\lVert{\mathbf{v}}\rVert=1$.

First, by dividing the expression of $p_{\Theta}(Y=y|Z={\mathbf{z}})$ by its
numerator:

$\displaystyle\mathcal{L}_{ce}({\mathbf{z}}_{i},y_{i},{\Theta})=\log(1+e^{-(\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{1-y_{i}}\rVert^{2}_{{\mathbf{M}}_{1-y_{i}}}-\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{y_{i}}\rVert^{2}_{{\mathbf{M}}_{y_{i}}})})\geq
0.$ (9)

Our goal, is to build a set of parameters $\Theta(\lambda)$ such that
$\lim_{\lambda\to+\infty}\mathcal{L}_{ce}({\mathbf{z}}_{i},y_{i},{\Theta(\lambda)})=0$,
and $\lambda$ is an eigenvalue of ${\mathbf{M}}_{k}$ for all $k\in\\{0,1\\}$.

Since $\Sigma_{k}$ is PSD, we can decompose it as
${\mathbf{M}}_{k}={\mathbf{U}}_{k}^{t}\Lambda_{k}{\mathbf{U}}_{k}$, with
${\mathbf{U}}_{k}$ an orthogonal matrix, and
$\Lambda_{k}=\text{diag}(\lambda_{k,0},...\lambda_{k,d})$ a diagonal matrix
containing the eigenvalues of ${\mathbf{M}}_{k}$. The matrix
${\mathbf{U}}_{k}$ corresponds to a rotation of the vector space to a new
basis corresponding to the eigenvectors of ${\mathbf{M}}_{k}$ (the rows of
${\mathbf{U}}_{k}$).

We choose ${\mathbf{M}}_{k}(\lambda)$ so that its first eigenvector is
${\mathbf{v}}$ with an eigenvalue of $\lambda_{k,0}=\lambda$, and we keep all
other eigenvalues and eigenvector constant. We set ${\mathbf{w}}_{k}$ as:

${\mathbf{w}}_{k}=\begin{cases}(b-1){\mathbf{v}}\quad\text{if }k=0\\\
(b+1){\mathbf{v}}\quad\text{if }k=1.\\\ \end{cases}$

First, let’s find an equivalent of the Mahalanobis distance when
$\lambda\to+\infty$:

$\displaystyle\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{k}\rVert^{2}_{{\mathbf{M}}_{k}}$
$\displaystyle=\frac{1}{2}\left({\mathbf{z}}_{i}-{\mathbf{w}}_{k}\right)^{T}{\mathbf{U}}^{T}\Lambda_{k}{\mathbf{U}}\left({\mathbf{z}}_{i}-{\mathbf{w}}_{k}\right)$
$\displaystyle\sim_{\lambda\to+\infty}\frac{\lambda}{2}\left({\mathbf{z}}_{i}-{\mathbf{w}}_{k}\right)^{T}{\mathbf{v}}{\mathbf{v}}^{T}\left({\mathbf{z}}_{i}-{\mathbf{w}}_{k}\right)$
$\displaystyle\sim_{\lambda\to+\infty}\frac{\lambda}{2}\left({\mathbf{v}}^{T}{\mathbf{z}}_{i}-{\mathbf{v}}^{T}{\mathbf{w}}_{k}\right)^{2}.$

Let’s start with the case $y_{i}=1$:

$\displaystyle\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{1-y_{i}}\rVert^{2}_{{\mathbf{M}}_{1-y_{i}}}$
$\displaystyle\sim_{\lambda\to+\infty}\frac{\lambda}{2}\left(1+{\mathbf{v}}^{T}{\mathbf{z}}_{i}-b\right)^{2}$
$\displaystyle\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{y_{i}}\rVert^{2}_{{\mathbf{M}}_{y_{i}}}$
$\displaystyle\sim_{\lambda\to+\infty}\frac{\lambda}{2}\left(1-\left({\mathbf{v}}^{T}{\mathbf{z}}_{i}-b\right)\right)^{2}.$

Since we chose $y_{i}=1$, the assumption of linear separability in the
equation 8 states that ${\mathbf{v}}^{T}{\mathbf{z}}_{i}-b\textgreater 0$. As
a result,
$\left(1+{\mathbf{v}}^{T}{\mathbf{z}}_{i}-b\right)^{2}\textgreater\left(1-\left({\mathbf{v}}^{T}{\mathbf{z}}_{i}-b\right)\right)^{2}$.
Hence, for all $i$ such that $y_{i}=1$,

$\left({\mathbf{v}}^{T}\left({\mathbf{z}}_{i}-{\mathbf{w}}_{1-y_{i}}\right)\right)^{2}\textgreater\left({\mathbf{v}}^{T}\left({\mathbf{z}}_{i}-{\mathbf{w}}_{y_{i}}\right)\right)^{2}.$
(10)

The same reasoning applies for $y_{i}=0$, and we finally obtain for all
$i\in\\{0,\ldots,N\\}$:

$\displaystyle\lim_{\lambda\to+\infty}\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{1-y_{i}}\rVert^{2}_{{\mathbf{M}}_{1-y_{i}}}-\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{y_{i}}\rVert^{2}_{{\mathbf{M}}_{y_{i}}}$
$\displaystyle=\lim_{\lambda\to+\infty}\frac{\lambda}{2}\left(\left({\mathbf{v}}^{T}\left({\mathbf{z}}_{i}-{\mathbf{w}}_{1-y_{i}}\right)\right)^{2}-\left({\mathbf{v}}^{T}\left({\mathbf{z}}_{i}-{\mathbf{w}}_{y_{i}}\right)\right)^{2}\right)$
(11) $\displaystyle=+\infty.$

Finally, using the equation 11, and the equation 9, we obtain:

$\lim_{\lambda\to+\infty}\mathcal{L}_{ce}({\mathbf{z}}_{i},y_{i},{\Theta(\lambda)})=0.$
(12)

Note that this proof holds because we could choose a direction (an eigenvector
of ${\mathbf{M}}_{k}$) common to all data points, obtained from the linear
separability assumption on the data. ∎

Proof of Remark 3.2

###### Proof.

It is straightforward to see that for a given data point ${\mathbf{z}}_{i}$,
$\mathcal{L}_{ce}({\mathbf{z}}_{i},y_{i},\Theta)=f_{1}\left({\mathbf{z}}_{i},y_{i},\Theta\right)+f_{2}\left({\mathbf{z}}_{i},\Theta\right)$.

Let’s first consider $f_{1}$. Since ${\mathbf{M}}_{k}$ is PSD,

$0\leq\lVert z_{i}-{\mathbf{w}}_{k}\rVert^{2}_{{\mathbf{M}}_{k}}=\lVert
D_{k}\left(z_{i}-{\mathbf{w}}_{k}\right)\rVert_{2}^{2}\leq\lVert
D_{k}\rVert_{F}^{2}\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{k}\rVert_{2}^{2}.$
(13)

Hence when $\lVert{\mathbf{M}}_{k}\rVert_{F}\to 0$, $f_{1}\to 0$.

Then, let’s consider $f_{2}$.

$\displaystyle\underset{{\mathbf{M}}_{k},k\in\\{0,1\\}}{\arg\min}f_{2}\left({\mathbf{z}}_{i},\Theta\right)$
$\displaystyle=\underset{{\mathbf{M}}_{k},k\in\\{0,1\\}}{\arg\min}\log\left(\sum_{k\in\\{0,1\\}}\exp\left(-\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{k}\rVert^{2}_{{\mathbf{M}}_{k}}\right)\right)$
$\displaystyle=\underset{{\mathbf{M}}_{k},k\in\\{0,1\\}}{\arg\min}\sum_{k\in\\{0,1\\}}\exp\left(-\lVert{\mathbf{z}}_{i}-{\textnormal{w}}_{k}\rVert^{2}_{{\mathbf{M}}_{k}}\right)$
$\displaystyle=\left\\{\underset{{\mathbf{M}}_{k}}{\arg\min}\exp\left(-\lVert{\mathbf{z}}_{i}-{\textnormal{w}}_{k}\rVert^{2}_{{\mathbf{M}}_{k}}\right)\right\\}_{k\in\\{0,1\\}}.$

Minimizing $f_{2}$ is then equivalent to maximizing
$\lVert{\mathbf{z}}_{i}-{\textnormal{w}}_{k}\rVert^{2}_{{\mathbf{M}}_{k}}$ for
all $k\in\\{0,1\\}$, and:

$\lVert
z_{i}-{\mathbf{w}}_{k}\rVert^{2}_{{\mathbf{M}}_{k}}=({\mathbf{z}}_{i}-{\mathbf{w}}_{k})^{T}{\mathbf{M}}_{k}({\mathbf{z}}_{i}-{\mathbf{w}}_{k})\geq\sigma_{k,min}\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{k}\rVert^{2}_{2},$
(14)

where $\sigma_{k,min}$ is the minimal eigenvalue of ${\mathbf{M}}_{k}$.

As a result, $f_{2}$ is minimized when for all $k\in\\{0,1\\}$
$\sigma_{k,min}\to+\infty$, and hence when
$\lVert{\mathbf{M}}_{k}\rVert_{F}^{2}\to+\infty$.

This decomposition illustrates the role of each term in the cross-entropy
loss, the first trying to reduce the distance between each point belonging to
the same class, the other increasing the distance between all points. ∎

Proof of Proposition 3.3

###### Proof.

For a molecular embedding ${\mathbf{z}}_{i}$, the expression of
$f_{2}\left({\mathbf{z}}_{i},\Theta\right)$ is:

$f_{2}\left({\mathbf{z}}_{i},\Theta\right)=\log\left(\sum_{k\in\\{0,1\\}}\exp\left(-\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{k}\rVert^{2}_{{\mathbf{M}}_{k}}\right)\right).$

We use the following property of the log-sum-exp function:

$\max\left\\{-\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{k}\rVert^{2}_{{\mathbf{M}}_{k}}\right\\}_{k\in\\{0,1\\}}\leq\log\left(\sum_{k\in\\{0,1\\}}\exp\left(-\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{k}\rVert^{2}_{{\mathbf{M}}_{k}}\right)\right)\leq\max\left\\{-\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{k}\rVert^{2}_{{\mathbf{M}}_{k}}\right\\}_{k\in\\{0,1\\}}+\log
2.$

We can then majorize the maximum function:

$|\max\left\\{-\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{k}\rVert^{2}_{{\mathbf{M}}_{k}}\right\\}_{k\in\\{0,1\\}}|=\min\left\\{\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{k}\rVert^{2}_{{\mathbf{M}}_{k}}\right\\}_{k\in\\{0,1\\}}\leq\frac{1}{2}\sum_{k\in\\{0,1\\}}\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{k}\rVert^{2}_{{\mathbf{M}}_{k}}.$

Furthermore,
$\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{k}\rVert^{2}_{{\mathbf{M}}_{k}}=\lVert
D_{k}\left({\mathbf{z}}_{i}-{\mathbf{w}}_{k}\right)\rVert^{2}_{2}$.

And $\lVert
D_{k}\left({\mathbf{z}}_{i}-{\mathbf{w}}_{k}\right)\rVert^{2}_{2}\leq\lVert
D_{k}\rVert_{F}^{2}\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{k}\rVert^{2}_{2}$,
where $\lVert{\mathbf{z}}_{i}-{\mathbf{w}}_{k}\rVert^{2}_{2}$ is constant
w.r.t ${\mathbf{M}}_{k}$,

$f_{2}\left({\mathbf{z}}_{i},\Theta\right)=O\left(\sum_{k\in\\{0,1\\}}\lVert
D_{k}\rVert_{F}^{2}\right)=O\left(\sum_{k\in\\{0,1\\}}\lVert{\mathbf{M}}_{k}\rVert_{F}\right).$

Let’s study the asymptotic behavior of
$\tilde{f}_{2}\left({\mathbf{z}}_{i},\Theta\right)$.

$\displaystyle|\tilde{f}_{2}\left({\mathbf{z}}_{i},\Theta\right)|$
$\displaystyle=|\sum_{k\in\\{0,1\\}}\frac{\lvert\mathcal{S}_{k}\rvert}{\lvert\mathcal{S}\rvert}\log\det{{\mathbf{M}}_{k}}|$
$\displaystyle=|\sum_{k\in\\{0,1\\}}\frac{\lvert\mathcal{S}_{k}\rvert}{\lvert\mathcal{S}\rvert}\sum_{j\in\\{1\ldots
d\\}}\log{\sigma_{k,j}}|$ $\displaystyle\leq
d\sum_{k\in\\{0,1\\}}\frac{\lvert\mathcal{S}_{k}\rvert}{\lvert\mathcal{S}\rvert}\max\left\\{|\log{\sigma_{k,j}}|\right\\}_{j\in\\{1\ldots
d\\}}.$

Where d is the dimension of the data, and $\sigma_{k,j}$ is the $j$-th
eigenvalue of ${\mathbf{M}}_{k}$.

Since all eigenvalues of ${\mathbf{M}}_{k}$ for $k\in\\{0,1\\}$ grow to
infinity, $\max\left\\{|\log{\sigma_{k,j}}|\right\\}_{j\in\\{1\ldots
d\\}}=O(\log{\max\left\\{\sigma_{k,j}\right\\}_{j\in\\{1\ldots d\\}}})$.

Finally, all precision matrices are PSD, hence for $k\in\\{0,1\\}$:
$\sqrt{\sum_{j\in\\{1\ldots
d\\}}\sigma_{k,j}^{2}}=\lVert{\mathbf{M}}_{k}\rVert_{F}\geq\max\left\\{\sigma_{k,j}\right\\}_{j\in\\{1\ldots
d\\}}$.

As a result
$\tilde{f}_{2}\left({\mathbf{z}}_{i},\Theta\right)=O\left(\sum_{k\in\\{0,1\\}}\log{\lVert{\mathbf{M}}_{k}\rVert_{F}}\right)$.

∎

Proof of Remark 3.4

###### Proof.

First the log-likelihood of a gaussian distribution of mean ${\mathbf{w}}_{k}$
and of precision matrix ${\mathbf{M}}_{k}$, evaluated on ${\mathbf{z}}$ is:

$\mathcal{LL}_{{\mathbf{w}}_{k},{\mathbf{M}}_{k}}\left({\mathbf{z}}\right)=-\frac{1}{2}\left(\log\det{{\mathbf{M}}_{k}}-\lVert{\mathbf{z}}-{\mathbf{w}}_{k}\rVert_{{\mathbf{M}}_{k}}^{2}\right)+c,$

where c is a constant.

For a given support set $\mathcal{S}$, let’s compute the modified loss:

$\displaystyle\frac{1}{|\mathcal{S}|}\sum_{i\in\mathcal{S}}\left(f_{1}({\mathbf{z}}_{i},y_{i},\Theta)+\tilde{f}_{2}(\Theta)\right)$
$\displaystyle=\frac{1}{|\mathcal{S}|}\sum_{i\in\mathcal{S}}\left(\lVert{\mathbf{z}}-{\mathbf{w}}_{y_{i}}\rVert_{{\mathbf{M}}_{y_{i}}}^{2}-\sum_{k\in\\{0,1\\}}\frac{\lvert\mathcal{S}_{k}\rvert\log\det{({\mathbf{M}}_{k})}}{\lvert\mathcal{S}\rvert}\right)$
$\displaystyle=\frac{1}{|\mathcal{S}|}\sum_{i\in\mathcal{S}}\lVert{\mathbf{z}}-{\mathbf{w}}_{y_{i}}\rVert_{{\mathbf{M}}_{y_{i}}}^{2}-\frac{1}{|\mathcal{S}|}\sum_{k\in\\{0,1\\}}\lvert\mathcal{S}_{k}\rvert\log\det{({\mathbf{M}}_{k})}$
$\displaystyle=\frac{1}{|\mathcal{S}|}\sum_{k\in\\{0,1\\}}\sum_{i\in\mathcal{S}_{k}}\left(\lVert{\mathbf{z}}-{\mathbf{w}}_{k}\rVert_{{\mathbf{M}}_{k}}^{2}-\log\det{({\mathbf{M}}_{k})}\right).$

For a class $k\in\\{0,1\\}$, we are trying to minimize the negative log-
likelihood of a Gaussian distribution evaluated on the support set
$\mathcal{S}_{k}$, with mean ${\mathbf{w}}_{k}$ and precision matrix
${\mathbf{M}}_{k}$.

∎

###  A.1 Effect of $\lambda$ on the quadratic probe’s performances

Figure 7: Optimal values of $\lambda_{maha}$ found during the hyperparameter
search on FS-mol’s validation set. (computing
$\Sigma_{k}\triangleq(1-\lambda)\Sigma_{k,\text{empirical}}+\lambda{\mathbf{I}}$)

Estimating the covariance matrix of the data is challenging in the few-shot
setting. To regularize the empirical covariance matrix, we use a diagonal
shift to make the inverse of the covariance matrix well-defined and more
stable. The optimal value of $\lambda$ varies with the size of the support
set, as shown in the Figure 7. We see that the optimal value of $\lambda$
decreases with the size of the support set, which is expected as the empirical
covariance matrix becomes more stable with more samples.

All hyperparameters of the meta-learning methods are constant over the support
set sizes. In particular, all meta-learning methods are trained with a fixed
support set size (64 for Protonet, for instance) and evaluated on all support
set sizes. We chose similarly to use a single value of $\lambda=0.2$ across
all support set sizes. However, while modifying a hyperparameter in a meta-
learning method usually requires retraining the model, only the few-shot
adaptation step needs to be re-run when changing $\lambda$ in the quadratic
probe, for instance.

###  A.2 Out-of-domain experiments details

Some few-shot learning methods for drug discovery proposed out-of-domain
scenarios to evaluate their models. Still, the domain shifts involved are
often unrealistic, such as evaluating a model on Tox21(Huang et al., 2016), a
dataset containing chemicals that are not drug-like (pesticides, food
additives, …), or evaluating models’ performances on material design tasks.
However, models trained and designed for activity prediction would marginally
be used in such drastically different scenarios, giving limited insights into
the model’s performances. To propose more realistic domain-shift experiments,
we decided to consider scenarios using molecular inputs that do not
drastically differ from the ones seen in training, as well as considering
tasks related to activity prediction.

####  A.2.1 Drug-target-interaction details

Creation of Few-shot tasks

Figure 8: Imbalance of the DTI datasets’ task classes and distribution of the
thresholds used to obtain the classification labels in each dataset.

As explained in Section 4.3.1, we introduce various new datasets on which we
compared the various methods. To adapt the various datasets to the few-shot
learning scenario, we followed five steps:

  1. 1.

Identifying tasks already in the FS-mol training set is not trivial, as the
FS-mol dataset is large. To do so, we removed every task in the DTI datasets
that contained more than 30% of molecules already in the FS-mol training set.

  2. 2.

We then removed every SMILES that cannot be processed by the RDKit library or
ended up with a molecule containing an atom with a valence greater than 6.

  3. 3.

A molecule in a given task can be associated with several different activity
values on the same target. We decided to remove any molecule associated with
multiple measurements in a task.

  4. 4.

We separated the molecules into active and inactive classes by computing the
median of the activity values in the task, clipped between a maximum pXC50 of
9 and a minimum of pXC50 of 5, or 6 for BindingDB_Ki and BindingDB_IC50. We
then removed molecules whose activity was equal to the threshold chosen in the
previous step.

  5. 5.

Finally, we dropped all tasks with less than 60 molecules, or more than 5000,
and removed all tasks whose label distribution was too imbalanced (a positive
proportion outside of the $[0.1,0.6]$ range).

The resulting datasets provide different class imbalances, illustrated in the
Figure 8, inferred by the choice of the thresholds on the activity numeric
values for each task.

Complementary results and limitations

Figure 9: Performances of the considered methods on the BindingDB_Ki and
BindingDB_IC50 datasets. Few-shot models do not significantly outperform the
similarity search, and CLAMP is the best fine-tuning method.

We provided in the paper the results on DAVIS, KIBA and BindingDB_Kd, and we
provide supplementary results on BindingDB_IC50 and BindingDB_Ki in the Figure
9. We see that the fine-tuning methods we presented do not obtain competitive
results, which implies that our pretraining procedure resulted in a model that
is still sensible to shifts in the target type. Finally, CLAMP’s fine-tuning
outperforms our methods, which rely on a standard multi-task model. This can
be explained by the fact that CLAMP is trained on textual descriptions of the
assays. Obtaining textual information about the assays allows the model to
assimilate knowledge that is shareable and applicable across various target
types, potentially leading to better robustness to this domain shift.

ADKT-IFT obtains competitive results, showing strong robustness to the type of
target considered, but none of the methods achieved significantly better
results than the similarity search. This underscores the significant challenge
for the presented few-shot methods in effectively adapting to new targets.

####  A.2.2 Library screening details

Creation of Few-shot tasks

This section provides more details on adapting the HTS datasets we used in our
experiments. HTS datasets correspond to assays usually evaluated early in the
drug discovery process. These tasks often comprise many compounds with very
few active molecules to identify. We considered datasets available through the
Therapeutic Data Commons (TDC) (Huang et al., 2021) plateform (Touret et al.,
2020; Wu et al., 2018; Butkiewicz et al., 2013), and various tasks from the
LIT-PCBA dataset (Tran-Nguyen et al., 2020), a database built on 149
dose–response PubChem bioassays (Wang et al., 2016).

Running these experiments on such a large number of molecules would be too
computationally expensive. As a result, we down-sampled all datasets to 30000
molecules.

We processed each dataset as follows:

  1. 1.

Only compounds whose SMILES can be processed by the RDKit library are kept,
and we removed all molecules whose SMILES ended up with a molecule containing
an atom with a valence greater than 6.

  2. 2.

To create tasks of 30000 molecules, we first kept all active molecules if the
resulting proportion of active molecules remained under 7%. Otherwise, we
sampled the active molecules to keep the same proportion of active molecules
as in the original dataset.

  3. 3.

We then randomly sampled inactive molecules to obtain at most 30k molecules.

Table 2: Description of the datasets used for the library screening task. Benchmark | Dataset name name | Target type if known | Hit proportion | Dataset size
---|---|---|---|---
TDC | SARSCoV2 Vitro Touret | not known | 5.9% | 1500
| HIV | not known | 4.8% | 30k
| Orexin1 Receptor | GPCR | 0.77% | 30k
| M1 Musarinic Receptor agonist | GPCR | 0.62% | 30k
| M1 Musarinic Receptor antagonist | GPCR | 1.2% | 30k
| CAV3 T-type Calcium channels | Ion Channel | 4.8% | 30k
LIT-PCBA | ALDH1 | oxidoreductase | 4.9% | 30k
ESR1 antagonist | Nuclear receptor | 1.9% | 5k
GBA | hydrolase | 0.54% | 30k
MAPK1 | kinase | 1.0% | 30k
MTORC1 | protein complex | 1.9% | 30k
OPRK1 | GPCR | 0.066% | 30k
| PKM2 | kinase | 1.8% | 30k
| PPARG | nuclear receptor | 0.44% | 5k
| TP53 | antigen | 1.8% | 4k
| VDR | nuclear receptor | 2.9% | 30k

We removed all datasets initially composed of more than 300k compounds. We
also did not include the dataset SARSCoV2_3CLPro_Diamond dataset, as it
corresponds to a fragment screen, using a different input type than the drug-
like compounds we considered so far. The Table 2 describes the datasets used
for the library screening task. The datasets are diverse, with few datasets
containing about 5% of hits (SARSCOV2_Vitro_Touret, HIV, CAV3_T-
type_Calcium_channels, ALDH1), and the others having less than 2% of hits in
general.

Figure 10:  Distribution of the Tanimoto similarity between an active compound
and its nearest active neighbour in the HTS datasets (left). Similarity
matrices between the active compounds of the HTS datasets (right).

The Figure 10 reveals how different the molecular structure between hits can
be in these datasets. In particular, we computed the Tanimoto similarity
between all active compounds in each dataset. Also, we measure the maximum
similarity between an active compound and the other hits in the task. In
particular, the hits are structurally different in tasks such as PPARG or
Orexin1 receptor. As a result, retrieving active molecules using the
information of the hits available in the support set can be challenging. In
some other tasks, such as OPRK1 or HIV, some hits share similar structures.
The heterogeneity of the tasks motivates the choice of using rankings as a
metric, as the models’ performances on each task can be drastically different.

Complete results

Results on all datasets are presented in the figure 11 and 12, as a line plot
displaying the evolution of the hitrate when the number of molecules selected
increases. The datasets are ordered according to the value 75% quantile of the
maximum Tanimoto similarity between an active compound and the other hits in
the task. This can be interpreted as a mild approximation of the task’s
difficulty (as it does not consider similarity to inactive compounds or the
proportion of hits).

Interestingly, in the estimated ’hardest’ tasks, when the support set size
increases, the performances of some models (such as the linear probe) do not
always increase (such as on the SARSCOV2_Vitro_Touret or the ALDH1 datasets).
This can be explained by the fact that the hits are structurally different.
Indeed, adding a new hit to the support set can be seen as adding noise to the
support set, adding a molecule whose structure does not help to find a common
representation for all hits. This is especially the case on the
SARSCOV2_Vitro_Touret dataset.

Figure 11:  Evolution of the hitrate with the percentage of the dataset
selected, on each dataset for different support set sizes, when 5% of the
support set are hits.  Figure 12:  Evolution of the hitrate with the
percentage of the dataset selected, on each dataset for different support set
sizes, when 10% of the support set are hits.


capbtabboxtable[][]

11institutetext: FIZ Karlsruhe – Leibniz Institute for Information
Infrastructure, Germany
22institutetext: Karlsruhe Institute of Technology, Institute AIFB, Germany
22email<EMAIL_ADDRESS>
22email: {russa.biswas, mehwish.alam<EMAIL_ADDRESS>
# Entity Type Prediction in Knowledge Graphs using Embeddings

Russa Biswas 1122 Radina Sofronova 22 Mehwish Alam 1122 Harald Sack 1122

###### Abstract

Open Knowledge Graphs (such as DBpedia, Wikidata, YAGO) have been recognized
as the backbone of diverse applications in the field of data mining and
information retrieval. Hence, the completeness and correctness of the
Knowledge Graphs (KGs) is vital. Most of these KGs are mostly created either
via an automated information extraction from Wikipedia snapshots or
information accumulation provided by the users or using heuristics. However,
it has been observed that the type information of these KGs is often noisy,
incomplete and incorrect. To deal with this problem a multi-label
classification approach is proposed in this work for entity typing using KG
embeddings. We compare our approach with the current state-of-the-art type
prediction method and report on experiments with the KGs.

###### Keywords:

Type PredictionKnowledge Graph Embeddings Knowledge Graph Completion.

##  1 Introduction

Open Knowledge Graphs (KGs) such as DBpedia, Wikidata, YAGO, etc. have been
recognized as the foundations for diverse KG based applications including
Natural Language Processing, data mining and Information Retrieval. Most of
these KGs are created either via automated information extraction from
Wikipedia snapshots, information accumulation provided by the users or by
using heuristics. However, each KG follows a different knowledge organization
and is based on differently structured ontologies. Moreover, it has been
observed that type information are often noisy or incomplete. On the other
hand, these KGs contain huge amount of data which makes it difficult to be
used by the applications. Therefore, recent years have witnessed an extensive
research on the latent representation of the KGs in a low dimensional vector
space. In this work, the proposed method addresses the entity typing problem
in DBpedia using the embeddings as a multi-label classification problem.

Entity typing is the process of assigning a type to an entity and is a
fundamental task in KG completion. For example, the triple
<dbr:Albert_Einstein, rdf:type, dbo:Scientist> states that Albert Einstein is
assigned to the type class Scientist. The type information in DBpedia is
derived directly by an external extraction framework from the Wikipedia
infobox types. Since, the Wikipedia is a crowd sourced encyclopedia, hence
this type information is often incomplete. Therefore, a huge number of
entities in DBpedia are assigned to a coarse grained rdf:type. Table 2
provides the distribution of entities of five types. For e.g., class
dbo:SportsTeam has 14 subclasses in DBpedia and 352006 entities, out of which
only 8.9% are assigned to its subclasses. Hence, there arouses a necessity to
have fine grained types for the entities in the KGs.

On the other hand, most of the existing state-of-the-art KG embedding
approaches such as translational approaches such as TransE [5], TransR [8],
etc. exploit only the structure of the KG.However, besides the structural
information, implicit textual semantic information is also stored in the KGs
as illustrated in Figure 2. This subgraph depicts, the birthplace of Albert
Einstein is Ulm, which is located in the country Germany. The labels of the
triples in the subgraph, such as birthplace, country, Ulm, etc. contains
implicit textual information in the graph, that is not captured in
translational embedding models.

In this paper, a multi-label classification approach is proposed for fine
grained entity typing. To do so, the model uses different existing word
embedding models such as Word2Vec [10], GloVe [12], and FastText [4] to learn
the KG embeddings capturing the graph structure as well as the implicit
textual information available. The main contributions of this paper are:

  * •

Vector representation of entities and relations in DBpedia using the existing
word embedding models.

  * •

A multi-label classification based approach for fine grained entity typing.

  * •

An analysis and comparison of the aforementioned word embedding models for the
task of entity type prediction.

The rest of the paper is structured as follows. To begin with, a review of the
related work is provided in Section 2. Section 3 accommodates the detailed
description of the approach followed by experimental setup and report on the
results in Section 4. Finally, an outlook of future work is provided in
Section 5.

Figure 1: Exempt of a subgraph from DBpedia

Class |  #Entities |  #Coarse-Grained entities
---|---|---
SportsTeam |  352006 |  320835 (91.1%)
Company |  70208 |  55524 (79.1%)
Settlement |  478906 |  246163 (51.4%)
Activity |  19464 |  8824 (45.3%)
Event |  76029 |  19418 (25.5%)

Figure 2: Distribution of entities in subclasses

##  2 Related Work

This section presents the prior related works on entity typing considering
both Wikipedia Infobox type prediction as well as RDF type prediction.

Wikipedia Infobox Type Prediction. One of the initial works in this domain was
proposed by Wu et al.[14]. To do so, KYLIN considers pages having similar
infoboxes, determines the common attributes in them to learn a CRF extractor.
Sultana et al.[13] focuses on automated approach by training a SVM classifier
on the feature set of TF-IDF on the first k sentences of an article as well as
on categories and Named Entity mentions. Biswas et al.[3, 2] provides a neural
network based approach for infobox prediction using word embeddings on
abstract, table of contents, and categories of Wikipedia articles.

RDF Type Prediction. A statistical heuristic link based type prediction
mechanism, SDTyped, has been proposed by Paulheim et al. and was evaluated on
DBpedia and OpenCyc [11]. Another RDF type prediction of KGs has been studied
by Melo et al. [9], where the type prediction of the KGs is performed via the
hierarchical SLCN algorithm using a set of incoming and outgoing relations as
features for classification. In [7], the authors propose a supervised
hierarchical SVM classification approach for DBpedia by exploiting the
contents of Wikipedia articles. However, none of these methods exploit
embeddings to perform the type prediction. In this work, different word
embedding algorithms will be exploited on the KGs for the task of entity
typing.

##  3 Entity Typing using Embeddings

The task of entity type prediction is multi-label classification problem
considering the entity type information as classes which is discussed in this
section.

###  3.1 Word Embeddings on KGs

Each triple or fact in the KG is considered as a sentence where the relation
serves as a verb and the two entities are considered as the subject and the
object of this relation in the sentence. For e.g.,
$<dbr:Albert\\_Einstein,dbo:birthplace,dbr:Ulm>$ is considered as a sentence
These sentences are then used as a corpus for all the three word embeddings.
The URIs are considered for training. The dimension of the vectors for each of
the embedding models is 100 and the embeddings from all the models for DBpedia
available in our GitHub [1]

Word2Vec. It aims to learn the distributed representation for words reducing
the high dimensional word representations in large corpus. It comprises of two
model architectures, Continuous Bag of Words (CBOW) and Skip-gram. In CBOW
approach, the model predicts the current word from a window of context words.
On the other hand, the skip-gram model tries to predict the context words
based on the current word. In this work, the CBOW approach of Word2Vec model
has been used to learn the vector representation of the entities and relations
in the KG based on the context entity or relation.

FastText. FastText is an extension of the word2vec model, which follows both
CBOW and Skip-gram architectures. The main difference with the Word2Vec is
that it learns the representation of each word in the corpus as n-gram
characters. This benefits in capturing representations for shorter or rare
words which can be obtained by breaking down words into n-grams to get its
embeddings. Therefore, it would help in having embeddings for unseen facts in
KGs.

GloVe. GloVe is another word embedding model which exploits the global word-
word co-occurrence statistics in the corpus. The model is essentially a log-
bilinear model with a weighted least-squares objective. The main underlying
intuition is that ratios of word-word co-occurrence probabilities have the
potential for encoding some form of meaning. The co-occurrence of the entities
and the properties is important in learning the latent representation of KGs.

###  3.2 Entity Typing

Two approaches have been used to determine the entity types in this work, (i)
a supervised Convolutional Neural Network (CNN) based approach and (ii) vector
similarity.

Convolutional Neural Network. The entity typing problem is converted to a
classification problem with the rdf:type as classes in which, a 1D CNN model
[6] built on top of the embedding models. The model takes into account the
vectors of the entities generated from the embedding models and predicts its
type. The model consists of a convolutional layer which involves a feature
detector followed by a global max pool layer. The activation function ReLu has
been used in the convolutional layer. The output of the pooling layer is used
which is then passed through a fully connected final layer, in which the
sigmoid function calculates the probabilities of an entity belonging to
different classes. The filter size taken is 128, with kernel sizes ${3,4,6}$,
are chosen for the model.

Vector Similarity. In order to assign fine-grained type to an entity with an
already assigned coarse-grained type, class hierarchy in DBpedia has been
exploited. For e.g., in DBpedia, for the entity dbr:Baker&McKenzie, the
rdf:type class is dbo:LawFirm. Next, class hierarchy of dbo:LawFirm is
traversed to find the highest level parent class dbo:Organisation after
dbo:Agent. Now, all the subclasses of dbo:Organisation in the hierarchy is
extracted and the cosine similarity between all the subclasses and the entity
dbr:Baker&McKenzie has been calculated. Since the entities of a class
represent the characteristic features of the class, the average vector of the
entity vectors belonging to a certain class has been chosen as the class
vector.

##  4 Experiments and Results

This section contains description of the experiments and analysis of the
results.

Dataset. In order to have fine-grained type prediction of the entities which
are already coarse-grained typed in DBpedia
2016-10111https://wiki.dbpedia.org/downloads-2016-10, 3 datasets have been
generated to evaluate the method. To determine the robustness of the method,
the datasets comprise of classes with less number of entities as well as the
ones with large entity count. The statistics of the dataset is provided in
Table 1.

Results.  The vector similarity approach is considered as the baseline model
in this work. The CNN model is evaluated on 80%-20% of training and test split
of each of the dataset as depicted in Table 1. It is trained with a batch size
of 32, 125 hidden layers and 1000 epochs.

Datasets | Models (Results in Accuracy)
---|---
Word2Vec | FastText | GloVe
| Vector
---
Similarity
CNN |  | Vector
---
Similarity
CNN |  | Vector
---
Similarity
CNN
Hits@3 | Hits@1 | Hits@3 | Hits@1 | Hits@3 | Hits@1
| 59 classes,
---
500 entities/class
47.83% | 28.46% | 56% | 29.81% | 17.44% | 54% | 7.07% | 3.54% | 53.7%
| 86 classes,
---
2k entities/class
58% | 39.4% | 58.4% | 43.81% | 31.16% | 56% | 15.9% | 8.2% | 55%
| 81 classes,
---
4k entities/class
58% | 39.7% | 62% | 44.3% | 31.4% | 59% | 16.2% | 8.4% | 55.8%
Table 1: Experimental Results

It has been observed from the results that the CNN built on the top of the
embedding models achieved better results in the entity typing task. However,
the vector similarity results with Hits@3 for the word2vec vectors is
comparable to CNN for the 86 classes dataset. The results of vector similarity
depict that the vectors generated by the GloVe model is not so similar to each
other, even then the CNN predicts the correct type with a much better
accuracy. Also, 81 classes is a subset of 86 classes, with more number of
entities per class which strengthens the fact the neural network models work
better with more data. For the dataset with 4000 entities per class, the CNN
works the best for all the embedding models as compared to the other methods.

Also, the method has been compared with the available SDTyped
dataset222http://downloads.dbpedia.org/2016-10/core-i18n/en/instance_types_sdtyped_dbo_en.ttl.bz2.
This dataset consists of the entity types predicted by SDType method. It is to
be noted that only a small fraction of entities are common between the SDType
dataset and our dataset as depicted in column 3 of Table 2. The count of the
entities in SDTypes whose type information matches the ground truth is
provided in the last column of same table. Due to huge differences in the
datasets, a direct comparison of the models with this dataset is not possible.
However, an analysis based only on the overlapping entities is available in
the GitHub [1].

Datasets |  | #Entities in
---
our dataset (E)
| #$E\cap\\#E_{SDType}$
---
| #$E_{SDType}$
---
$=GroundTruth$
| 59 classes,
---
500 entities/class
28106 | 7425 (26%) | 6115
| 86 classes,
---
2k entities/class
172000 | 57467 (33%) | 46222
| 81 classes,
---
4k entities/class
324000 | 109948 (34%) | 89301
Table 2: Analysis with the SDType data

##  5 Conclusion and Future Work

In this paper, different word embeddings approaches for entity typing in a KG
have been analyzed. The achieved results demonstrate that vectors coupled with
CNN works better for the task. On the other hand, set theory concept333A set
is represented by its members, which exhibit the same properties. when applied
to generate the class vectors from the entity vectors proved to be beneficial.
In future, these embedding models would be used for other KG completion tasks
such as link prediction, triple classification, etc. Also, for the entity
typing task, more information to be included in this embedding space such the
DBpedia categories to improve the results.

## References

  * [1] DBpedia Embeddings. https://github.com/ISE-FIZKarlsruhe/Entity-Typing-with-Word-Embeddings
  * [2] Biswas, R., Koutraki, M., Sack, H.: Predicting wikipedia infobox type information using word embeddings on categories. In: EKAW (Posters & Demos) (2018)
  * [3] Biswas, R., Türker, R., Moghaddam, F.B., Koutraki, M., Sack, H.: Wikipedia infobox type prediction using embeddings. In: DL4KGS@ ESWC (2018)
  * [4] Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. TACL (2017)
  * [5] Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. In: NIPS (2013)
  * [6] Kim, Y.: Convolutional neural networks for sentence classification. In: EMNLP (2014)
  * [7] Kliegr, T., Zamazal, O.: LHD 2.0: A Text Mining Approach to Typing Entities in Knowledge Graphs. J. Web Sem. (2016)
  * [8] Lin, Y., Liu, Z., Sun, M., Liu, Y., Zhu, X.: Learning entity and relation embeddings for knowledge graph completion. In: Twenty-ninth AAAI conference on artificial intelligence (2015)
  * [9] Melo, A., Paulheim, H., Völker, J.: Type Prediction in RDF Knowledge Bases Using Hierarchical Multilabel Classification. In: WIMS (2016)
  * [10] Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient Estimation of Word Representations in Vector Space. CoRR (2013)
  * [11] Paulheim, H., Bizer, C.: Type Inference on Noisy RDF Data. In: ISWC (2013)
  * [12] Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: EMNLP (2014)
  * [13] Sultana, A., Hasan, Q.M., Biswas, A.K., Das, S., Rahman, H., Ding, C.H.Q., Li, C.: Infobox Suggestion for Wikipedia Entities. In: CIKM (2012)
  * [14] Wu, F., Weld, D.S.: Autonomously Semantifying Wikipedia. In: CIKM (2007)