#  SPOT: Sequential Predictive Modeling of Clinical Trial Outcome with Meta-
Learning

Zifeng Wang  UIUCUrbanaILUSA<EMAIL_ADDRESS>,  Cao Xiao
RelativityChicagoILUSA<EMAIL_ADDRESS>and  Jimeng Sun
UIUCUrbanaILUSA<EMAIL_ADDRESS>
###### Abstract.

Clinical trials are essential to drug development but time-consuming, costly,
and prone to failure. Accurate trial outcome prediction based on historical
trial data promises better trial investment decisions and more trial success.
Existing trial outcome prediction models were not designed to model the
relations among similar trials, capture the progression of features and
designs of similar trials, or address the skewness of trial data which causes
inferior performance for less common trials.

To fill the gap and provide accurate trial outcome prediction, we propose
Sequential Predictive mOdeling of clinical Trial outcome (SPOT) that first
identifies trial topics to cluster the multi-sourced trial data into relevant
trial topics. It then generates trial embeddings and organizes them by topic
and time to create clinical trial sequences. With the consideration of each
trial sequence as a task, it uses a meta-learning strategy to achieve a point
where the model can rapidly adapt to new tasks with minimal updates. In
particular, the topic discovery module enables a deeper understanding of the
underlying structure of the data, while sequential learning captures the
evolution of trial designs and outcomes. This results in predictions that are
not only more accurate but also more interpretable, taking into account the
temporal patterns and unique characteristics of each trial topic. We
demonstrate that SPOT wins over the prior methods by a significant margin on
trial outcome benchmark data: with a 21.5% lift on phase I, an 8.9% lift on
phase II, and a 5.5% lift on phase III trials in the metric of the area under
precision-recall curve (PR-AUC).

##  1\. Introduction

Clinical trials are essential for developing new treatments, where drugs must
pass the safety and efficacy thresholds across at least three trial phases
before they can get approval for manufacturing. Running the trials is time-
consuming and costly. On average, it takes about $10\sim 15$ years and costs
$2.87 billion for a drug to complete all trial phases, and many drugs fail in
clinical trials (Blass, 2015). As clinical trial records and their outcomes
have been collected and digitized during the past decade, which opens the door
for providing more data-driven insights that can ensure more trial success.

Figure 1. Our formulation differs from all existing works: existing works
predict trial outcomes for each trial separately; while SPOT aggregates trials
of the same topic to a sequence based on their timestamps, thus utilizing
information of similar trials and their progression in predictions. It also
considers each trial sequence as a task and mitigates the trial data imbalance
problem using a meta-learning strategy.

Figure 2.  Trial outcome distributions of phase I, II, and III trials from the
TOP dataset (Fu et al., 2022). We plot target conditions of trials on $x$-axis
and the average trial success rates on $y$-axis. It can be seen that trials on
different topics have distinct success rates.

Among others, clinical trial outcome prediction is one critical task. Earlier
attempts were often made to predict individual components in clinical trials
to improve the trial results for each trials (Gayvert et al., 2016; Qi and
Tang, 2019). Recently, researchers started to propose general methods for
trial outcome predictions. For example, Lo et al. (2019) predicted drug
approvals for 15 disease groups based on drug and clinical trial features
using classical machine learning methods. More recently, Fu et al. (2022)
proposed leveraging data from multiple sources (e.g., molecule information,
trial documents, disease knowledge graph, etc.) with an interaction network to
capture their correlations to support trial outcome predictions. Despite these
initial successes, they still present the following limitations.

  * •

Lack of alleviating heterogeneity in trial patterns. Clinical trials targeting
different diseases and phases could exhibit heterogeneous patterns. A
preferred strategy to resolve this heterogeneity is to cluster trials into
more homogeneous groups before prediction. Existing works do not consider the
grouping. While simply group clinical trials based on targeting diseases or
phases can be inadequate as they may yield too broad categories, where trials
under the same branch can vary dramatically in their semantics, or too fine-
grained categories, where there are limited numbers of trials in many
categories, making it difficult to extract useful patterns.

  * •

Lack of modeling the progression of trial design for similar trials. For
similar trials (e.g., targeting disease, drug structure, etc.), the
progression of trial designs encodes the knowledge from clinical experts.
Modeling this progression pattern could benefit predicting outcomes for new
trials. Existing works failed to consider the progression of trial design in
their trial outcome prediction.

  * •

Lack of handling of the imbalance data . Clinical trial data can be highly
imbalanced, with specific subgroups or treatments having a small number of
trials and diverse average success rates, as illustrated by Fig. 2. This
presents machine learning challenge for trial embedding and outcome
prediction. Existing methods for outcome prediction struggle to accurately
predict outcomes for these minority trials. This is a significant limitation,
as it can result in inaccurate predictions and a lack of understanding of the
outcomes for these subgroups.

To fill the gap and provide accurate trial outcome prediction for all trials,
we propose Sequential Predictive mOdeling of clinical Trial outcome (SPOT).
SPOT is enabled by three main components:

  * •

Topic discovery to alleviate heterogeneous trial patterns. SPOT clusters
trials into topics where trials of the same topic are more likely to share
more homogeneous patterns. This allows for a more nuanced representation of
the trials, reducing the noise from dissimilar trials and capturing subtle
semantic similarities that pre-defined categories may not capture.

  * •

Sequential modeling to capture the progression of trial design. SPOT
aggregates trials of the same topic to a sequence based on their timestamps
and learns to model the temporal patterns of the trial sequence. This allows
for extracting knowledge underlying the progression of trial designs and their
outcomes, thus enhancing the trial embedding and outcome prediction.

  * •

Meta learning to model the imbalanced data. To alleviate the imbalanced data
challenge, SPOT considers each trial sequence as a task and uses a meta-
learning strategy that can generalize well across broad clinical trial
distributions. It allows SPOT to provide a customized model for each task to
maximize the utility.

##  2\. Related Work

Machine Learning for Clinical Trials. Machine learning has been used in
advancing the clinical trial modeling across a diverse set of tasks (Wang et
al., 2022), including clinical trial patient-level outcome prediction (Wang
and Sun, 2022a), clinical trial referential search (Marshall et al., 2017;
Wang and Sun, 2022b), and clinical trial outcome prediction (Gayvert et al.,
2016; Siah et al., 2021; Fu et al., 2022). For the task of trial outcome
prediction, Gayvert et al. (2016) fused drug chemical structures and
properties to predict drug toxicity based on random forests; Qi and Tang
(2019) predicted the pharmacokinetics at phase III using records obtained from
phase II trials using RNNs; Siah et al. (2021) predicted drug approvals using
a combination of drug and trial features; Lo et al. (2019) tried to predict
the drug approval with the outcomes of phase II trials or phase III trials
using statistical machine learning models like random forests.

Among them, Fu et al. (2022) is most relevant to our work. It aggregates trial
data from multiple sources and utilizes a graph neural network on an
interaction graph with trial components to fuse multi-modal input data for
outcome prediction. However, it does not account for the relations among the
trials. Also, the application of average empirical risk minimization urges the
model to serve the majority group while neglecting the utility of the minority
group. This issue was demonstrated by Fu et al. (2022) that although the model
achieves 0.723 of AUC for all phase III trials on average, it only gets 0.595
of AUC for trials targeting Neoplasm. This paper accounts for the imbalanced
model performance across trial groups by aggregating trials by topics and
leveraging meta-learning. In addition, we consider the progression of trial
outcomes within each topic by sequential modeling.

Meta-Learning. Meta-learning is performed on a set of tasks and leverages
prior experiences when tackling a new task. The research on meta-learning
could be traced back to decades ago (Schmidhuber, 1987; Thrun and Pratt,
1998). A resurgence of interest in meta-learning raised in recent years with
the development of machine learning and deep learning, which includes meta-
learning for few-shot learning (Ravi and Larochelle, 2016; Gidaris and
Komodakis, 2018), long-tailed learning (Kang et al., 2019), reinforcement
learning (Duan et al., 2016; Gupta et al., 2018), neural architecture search
(Liu et al., 2018; Brock et al., 2018), etc. In particular, we consider taking
the optimization-based meta-learning (Finn et al., 2017; Li et al., 2017;
Nichol et al., 2018). One of the most well-known methods is MAML (Finn et al.,
2017), which makes iterative two-round updates for local and global
parameters, respectively. Through this process, it learns a model that can
quickly adapt to new tasks, e.g., the classification of new classes of images,
by direct task loss minimization. Our proposed SPOT utilizes this technique to
provide task-specific prediction models for clinical trial outcomes,
accounting for the discrepancy of optimal weights across trial tasks.

Figure 3.  The illustration of SPOT workflow. The SPOT model takes raw
clinical trial data with multi-sourced features as input. It first clusters
these data into relevant topics using the topic discovery module. The trial
embeddings are then generated to prepare data for topic-specific clinical
trial sequences. The output trial embeddings are organized based on their
topics and arranged chronologically to form clinical trial sequences. We then
train a meta-learner based on the sequences, allowing for rapid adaptation to
each task with several updates.

##  3\. The SPOT Method

The SPOT model processes raw clinical trial data with multi-sourced features
as input then leverages a topic discovery module to cluster the data into
relevant topics (Sec 3.2). It then generates trial embeddings and organizes
them by topic and time to create topic-specific clinical trial sequences (Sec
3.3 and 3.4). Through learning a meta-learner based on these sequences, our
model can rapidly adapt to the outcome prediction of incoming sequences of
clinical trials with minimal updates (Sec 3.5). In the following, we start by
presenting the problem formulation and then go with the details of each
component one by one.

###  3.1. Method Overview

We illustrate the workflow of SPOT in Fig. 3. The raw training set
$\mathcal{X}=\\{x_{1},\dots,x_{N}\\}$ is a collection of $N$ clinical trials
with corresponding labels $\mathcal{Y}=\\{y_{1},\dots,y_{N}\\}$ (i.e., whether
a trial reaches its desirable end point). Each trial $x_{n}$ is represented by
three main features as
$x_{n}=\\{\mathbf{d}_{n},\mathbf{t}_{n},\mathbf{c}_{n}\\}$, where
$\mathbf{d}_{n}=\\{d_{1},\dots,d_{N_{n}^{d}}\\}$ is a set of target diseases
of the $n$-th trial; $\mathbf{t}_{n}=\\{t_{1},\dots,t_{N_{n}^{t}}\\}$ is a set
of treatments used in the $n$-th trial;
$\mathbf{c}_{n}=\\{c_{1},\dots,c_{N_{n}^{c}}\\}$ is a set of inclusion and
exclusion eligibility criteria of the $n$-th trial. $N_{n}^{d}$, $N_{n}^{t}$,
$N_{n}^{c}$ are the total numbers of diseases, treatments, and eligibility
criteria in trial $n$, respectively. The following are the two main steps of
SPOT.

Topic Discovery. The first step is to execute topic discovery on $\mathcal{X}$
such that we split all trials into $K$ mutually exclusive clusters, as
$\mathcal{X}=\\{\mathcal{X}_{1},\dots,\mathcal{X}_{K}\\}$ and
$\mathcal{Y}=\\{\mathcal{Y}_{1},\dots,\mathcal{Y}_{K}\\}$. For each cluster
$\mathcal{X}_{k}$, we rearrange all trials based on their timestamps
$t\in\\{1,\dots,T_{k}\\}$. Dependening on the time granularity (e.g., by
year), there could be multiple concurrent trials with the same timestamp $t$
denoted by the concurrent trial set in topic cluster $k$:
$\mathbf{x}_{k}^{(t)}=\\{x_{k,1}^{(t)},\dots,x_{k,M_{k}^{t}}^{(t)}\\}$, where
$M_{k}^{t}$ is the total number of trials at the timestamp $t$ in the cluster
$k$. We thus obtain a sequence of clinical trials based on $\mathcal{X}_{k}$,
which we denote by $\mathbf{X}_{k}$ as

(1)
$\mathbf{X}_{k}=[\mathbf{x}_{k}^{(1)},\mathbf{x}_{k}^{(2)},\dots,\mathbf{x}_{k}^{(T_{k})}].$

We then model the progression of clinical trial designs and outcomes based on
sequences $\mathbf{X}_{k}$ for $k=1,\dots,K$.

Trial Outcome Prediction. The ultimate goal of our method is to predict the
outcomes of clinical trials as $\mathcal{Y}=\\{y_{1},\dots,y_{N}\\}$, where
each $y_{n}\in\\{0,1\\}$ indicating the failure ($y_{n}=0$) or success
($y_{n}=1$) of the trial. Here, trial success is defined by whether the trial
is completed and meets its primary endpoint.

To predict trial $x_{n}$ from topic $k$ and timestamp $t$, we first put trial
$x_{n}$ into the proper position in the trial sequence and then predict using
the trials before timestamp $t$. Formally we consider $x_{n}$ be in
$\mathbf{x}_{k}^{(t)}=\\{x_{n},x_{k,2}^{(t)},\dots,x_{k,M_{k}^{t}}^{(t)}\\}$.
Therefore, the prediction model $f(\cdot)$ leverages both the features of
$x_{n}$ as $x_{n}=\\{\mathbf{d}_{n},\mathbf{t}_{n},\mathbf{c}_{n}\\}$ and the
precedent trials in the same sequence $\mathbf{X}_{k}$, as

(2)
$\hat{y}_{n}=f\left(\\{\mathbf{d}_{n},\mathbf{t}_{n},\mathbf{c}_{n}\\},\mathbf{X}_{k}^{1:t}\right)\in[0,1],$

where $\mathbf{X}_{k}^{1:t}$ are all the elements of $\mathbf{X}_{k}$ that are
before the timestamp $t$, as

(3) $\mathbf{X}_{k}^{1:t}=[\mathbf{x}_{k}^{(1)},\dots,\mathbf{x}_{k}^{(t)}].$

The objective function is hence the binary cross entropy leveraging $y$ as the
supervision to $\hat{y}$ for updating $f(\cdot)$.

###  3.2. Trial Topic Discovery

As illustrated by Fig. 1, we solicit the latent topic structures in the trial
dataset to establish the trial topics, as the trials of the same topic should
be similar, considering multiple aspects such as their targeting diseases,
treatments, criteria design, etc. We further define the predictive modeling of
outcomes for different topics of trials are distinct tasks, which comprise a
sequence of trials belonging to the same topic.

To cluster trials into coherent topics, we utilize a pretrained language model
Trial2Vec that generates dense trial embeddings with rich semantics (Wang and
Sun, 2022b). Formally, we extract the most prominent attributes from the raw
trial document, such as the title, disease, treatment, eligibility criteria,
and outcome measure, as $x^{\text{attr}}$; and the context information such as
the trial description, as $x^{\text{ctx}}$. We first encode all elements in
$x^{\text{attr}}$ to dense embeddings as in Eq. (4),

(4)
$\mathbf{z}^{\text{attr}}=\\{\mathbf{z}^{\text{title}},\mathbf{z}^{\text{disease}},\mathbf{z}^{\text{treatment}},\mathbf{z}^{\text{criteria}},\mathbf{z}^{\text{outcome}}\\}.$

We concatenate all elements in the context $x^{\text{ctx}}$ and encode it to
$\mathbf{z}^{\text{ctx}}$. Combining $\mathbf{z}^{\text{attr}}$ and
$\mathbf{z}^{\text{ctx}}$ using multi-head attention leads to a semantically
meaningful embedding $\mathbf{z}$ for similar trial searching, as given by Eq.
(5),

(5)
$\mathbf{z}=\text{MultiHeadAttn}(\mathbf{z}^{\text{ctx}},\mathbf{z}^{\text{attr}};\mathbf{W}),$

where $\mathbf{W}$ indicates the parameters of the attention layer.

We apply k-means clustering to the trial embeddings
$\\{\mathbf{z}_{1},\dots,\mathbf{z}_{N}\\}$, hence each trial is assigned a
topic label $k\in\\{1,\dots,K\\}$, where $K$ is the specified number of
topics. For each topic $\mathcal{X}_{k}$, the trials are ordered by the
starting year, so as to build a sequence of trials, as $\mathbf{X}_{k}$
mentioned in Eq. (1). To evaluate the quality of the clustering results and
pick the best $K$, we can leverage internal clustering metrics such as Sum of
Squared Distance and Average Silhouette Coefficient.

###  3.3. Static Trial Embedding

While learning trial topics, we also prepare clinical trial data for trial
sequences by first transforming them into static trial embedding.
Specifically, given a trial
$x_{n}=\\{\mathbf{d}_{n},\mathbf{t}_{n},\mathbf{c}_{n}\\}$ which has three
main components: disease, treatment, and criteria, we apply three different
encoders to each of these components.

Disease Embedding. The disease set
$\mathbf{d}_{n}=\\{d_{1},\dots,d_{N_{n}^{d}}\\}$ contains the ICD codes that
can be mapped back to a hierarchy of conceptions. We adopt a graph-based
attention model named GRAM (Choi et al., 2017) to obtain the code embeddings
considering their positions in the knowledge graph as in Eq. (6),

(6)
$\mathbf{h}_{n}^{d}=\frac{1}{N_{n}^{d}}\sum_{i}^{N_{n}^{d}}\text{GRAM}(d_{i})\in\mathbb{R}^{p},$

where we take a mean-pooling of all code embeddings to get the disease
embedding $\mathbf{h}_{n}^{d}$.

Treatment Embedding. We adopt a graph message passing neural network (MPNN)
(Gilmer et al., 2017) to encode the treatment set that consists of drug
molecules represented by SMILES strings, then also take a mean-pooling to
obtain the treatment embedding in Eq. (7),

(7)
$\mathbf{h}_{n}^{t}=\frac{1}{N_{n}^{t}}\sum_{i}^{N_{n}^{t}}\text{MPNN}(t_{i})\in\mathbb{R}^{p}.$

In this work, we focus on small molecule treatments and plan to explore other
treatment embedding for other treatment types such as biologics and medical
devices as future work.

Criteria Embedding. We leverage the pretrained Trial2Vec (Wang and Sun, 2022b)
to encode criteria text $\mathbf{c}_{n}=[c_{1},\dots,c_{N_{n}^{c}}]$ as shown
in Eq. (8),

(8)
$\mathbf{h}_{n}^{c}=\frac{1}{N_{n}^{c}}\sum_{i}^{N_{n}^{c}}\text{Trial2Vec}(c_{i})\in\mathbb{R}^{p},$

which we also take mean-pooling.

We also encode the trial’s starting year by
$\mathbf{h}_{y}=\text{year}*\mathbf{h}_{ts}$, where $\mathbf{h}_{ts}$ is a
trainable numerical embedding. The year embedding $\mathbf{h}_{y}$ acts as a
representation of the trend of trial outcomes and advances over time because
scientific advancements can vary from year to year. Thus, the trial embedding
is produced by a highway neural network (Srivastava et al., 2015) that
adaptively adjusts the importance of different components through gating as in
Eq. (9),

(9)
$\mathbf{h}_{n}=\frac{1}{4}\sum\text{Highway}([\mathbf{h}_{n}^{d}\oplus\mathbf{h}_{n}^{t}\oplus\mathbf{h}_{n}^{c}\oplus\mathbf{h}_{n}^{y}])\in\mathbb{R}^{p}.$

Here, $\oplus$ indicates the concatenation of the input embeddings; we take an
average pooling of the concatenated embeddings which are with a shape of
$4\times p$ to obtain an overall trial representation
$\mathbf{h}_{n}\in\mathbb{R}^{p}$.

###  3.4. Sequential Trial Embedding

With the trial topics and the static trial embedding, we are now ready to
construct the sequential trial embeddings. As illustrated by the left part of
Fig. 3, a collection of clinical trials belonging to the same topic are placed
in the temporal order, as
$\mathbf{X}_{k}=[\mathbf{x}_{k}^{(1)},\dots,\mathbf{x}_{k}^{(T_{k})}]$. At
timestep $t$,
$\mathbf{x}_{k}^{(t)}=\\{x_{k,1}^{(t)},\dots,x_{k,M_{k}^{t}}^{(t)}\\}$ is a
set of trials occurred concurrently. From Eq. (9), we encode trials in
$\mathbf{x}_{k}^{(t)}$ to get the static embeddings as
$\mathbf{H}^{(t)}_{k}=[\mathbf{h}^{(t)}_{k,1},\dots,\mathbf{h}^{(t)}_{k,M_{k}^{t}}]$
for all $t\in\\{1,\dots,T\\}$.

As the sequential embedding was calculated within a topic, we omit the cluster
id $k$ to avoid clutter from now on. The static encoding of a topic
$\mathbf{X}$ can be written by
$\mathcal{H}=[\mathbf{H}^{(1)},\dots,\mathbf{H}^{(T)}]$. We propagate the
historical trial information along the timesteps through a recurrent neural
network (RNN) as given by Eq. (10),

(10) $\mathbf{h}^{(t)}=\text{RNN}(\mathbf{h}^{(t-1)},\dots,\mathbf{h}^{(1)}),$

where $\mathbf{h}^{(t)}$ is initialized by the mean-pooling of all trial
embeddings $\mathbf{H}^{(t)}$ at the timestamp.

In this way, $\mathbf{h}^{(t)}$ encodes the temporal trial information and can
contribute to a spatial interaction with the static embeddings $\mathbf{h}$
obtained by Eq. (9), as

(11)
$[\widetilde{\mathbf{h}}^{(t)}\oplus\widetilde{\mathbf{H}}^{(t)}]=\text{Interaction}([\mathbf{h}^{(t)}\oplus\mathbf{H}^{(t)}])\in\mathbb{R}^{(M^{t}+1)\times
d}.$

Here, $\text{Interaction}(\cdot)$ is a multi-head attention layer that
dynamically fuses the temporal information to different trial embeddings
$\widetilde{\mathbf{H}}^{(t)}$.

We predict trial outcomes via another function
$\text{NN}_{\text{pred}}(\cdot)$ that is a one-layer fully-connected neural
network:

(12)
$\hat{\mathbf{y}}^{(t)}=\text{NN}_{\text{pred}}(\widetilde{\mathbf{H}}^{(t)})=[\hat{y}_{1}^{(t)},\dots,\hat{y}_{M^{t}}^{(t)}]\in[0,1]^{M^{t}},$

which takes the concatenation of the adjusted embeddings and the static
embeddings. $M$ is the number of trials of the topic. Applying the prediction
model to the whole topic $\mathbf{X}$ yields predictions as

(13) $\hat{\mathbf{Y}}=[\hat{\mathbf{y}}^{(1)},\dots,\hat{\mathbf{y}}^{(T)}],$

which corresponds to trials
$\mathbf{X}=[\mathbf{x}^{(1)},\dots,\mathbf{x}^{(T)}]$.

###  3.5. Meta-Learning Task Embedding

Inspired by the concept of meta-learning and the distinction across trial
topics, we put model-agnostic meta-learning (MAML) (Finn et al., 2017) to the
SPOT framework to learn task-specific models. Concretely, we assume that
modeling on each trial topic is a separate task with two sets of model
parameters: global parameters $\\{\theta_{1},\theta_{2}\\}$ and task-specific
parameters $[\theta_{2}^{1},\dots,\theta_{2}^{K}]$. All tasks share the same
$\theta_{1}$ was determined in the static trial embedding modules described in
§3.3, e.g., the disease, treatment, and criteria encoder. $\theta_{2}$
corresponds to the parameters in the RNN and sequential prediction modules. To
reflect the progression patterns of topics, SPOT updates $\theta_{2}^{k}$ when
adapted to the $k$-th task. Formally, the algorithm first samples a batch of
tasks $\mathcal{X}$, then it undergoes two steps:

Step 1: Conduct local updates for each task $\mathbf{X}_{k}\sim\mathcal{X}$
via stochastic gradient descent as given by Eq. (14),

(14)
$\theta_{2}^{k}\leftarrow\theta_{2}^{k}-\alpha\nabla_{\theta_{2}^{k}}\mathcal{L}(\hat{\mathbf{Y}}_{k},\mathbf{Y}_{k}),$

where $\theta_{2}^{k}$ is initialized using $\theta_{2}$;
$\hat{\mathbf{Y}}_{k}$ is predicted by the task-specific model
$f_{\theta_{1},\theta_{2}^{k}}$; the loss $\mathcal{L}(\cdot)$ is binary cross
entropy given by Eq. (15),

(15)
$\mathcal{L}(\hat{\mathbf{Y}},\mathbf{Y})=-\mathbf{Y}\log\hat{\mathbf{Y}}-(1-\mathbf{Y})\log(1-\hat{\mathbf{Y}}),$

where $\mathbf{Y}\in\\{0,1\\}$: $\mathbf{Y}=0$ trial failed and $\mathbf{Y}=1$
trial succeeded.

Step 2: Conduct global updates of $\theta_{1}$ and $\theta_{2}$ based on the
task-specific parameters as shown in Eq. (16),

(16) $\displaystyle\theta_{1}$
$\displaystyle\leftarrow\theta_{1}-\beta\sum_{\mathbf{X}_{k}\in\mathcal{X}}\nabla_{\theta_{1}}\mathcal{L}(\hat{\mathbf{Y}}_{k},\mathbf{Y}),$
$\displaystyle\theta_{2}$
$\displaystyle\leftarrow\theta_{2}-\beta\sum_{\mathbf{X}_{k}\in\mathcal{X}}\nabla_{\theta_{2}}\mathcal{L}(\hat{\mathbf{Y}}_{k},\mathbf{Y}).$

Here, $\hat{\mathbf{Y}}_{k}$ is predicted by $f_{\theta_{1},\theta_{2}^{k}}$.
We take an average of all task-specific parameter gradients to update the
global parameter $\theta_{2}$.

The above steps repeat until both $\theta_{1}$ and $\theta_{2}$ converges.
Therefore, we can freeze $\theta_{1}$ and start from $\theta_{2}$ to quickly
adapt the model to every task to obtain the task-specific model. It is based
on the intuition that the basic trial property representations should keep
stable while the topic style may vary. This method also enjoys fast adaption
to a new task with a small query set and a few updates.

##  4\. Experiment

###  4.1. Experimental Setup

Dataset. We utilize the TOP clinical trial outcome prediction benchmark from
Fu et al. (2022). The dataset includes information on drug, disease, and
eligibility criteria for a total of 17,538 clinical trials. The trials are
divided into three phases: Phase I (1,787 trials), Phase II (6,102 trials),
and Phase III (4,576 trials). Success rates vary by phase, with 56.3% of phase
I trials, 49.8% of phase II trials, and 67.8% of phase III trials resulting in
success. The distribution of the targeting diseases is available at Table 6.
We conduct experiments on different phases of trials separately.

Table 1. Dataset statistics. # is short for the number of. # Trials are shown
by the split of train/validation/test sets.

| # Trials | # Drugs | # Diseases | # Success | # Failure
---|---|---|---|---|---
Phase I | 1,044/116/627 | 2,020 | 1,392 | 1,006 | 781 [t]
Phase II | 4,004/445/1,653 | 5,610 | 2,824 | 3,039 | 3,063
Phase III | 3,092/344/1,140 | 4,727 | 1,619 | 3,104 | 1,472 [b]

Metrics. We consider the following performance metrics.

  * •

AUROC: the area under the Receiver Operating Characteristic curve. It
summarizes the trade-off between the true positive rate (TPR) and the false
positive rate (FPR) with the varying threshold of FPR. In theory, it is
equivalent to calculating the ranking quality by the model predictions to
identify the true positive samples. However, better AUROC does not necessarily
indicate better outputting of well-calibrated probability predictions.

  * •

PRAUC: the area under the Precision-Recall curve. It summarizes the trade-off
between the precision (PPV) and the recall (TPR) with the varying threshold of
recall. It is equivalent to the average of precision scores calculated for
each recall threshold and is more sensitive to the detection quality of true
positives from the data, e.g., identifying which trial is going to succeed.

  * •

F1-score: the harmonic mean of precision and recall. It cares if the predicted
probabilities are well-calibrated as exact as PRAUC.

Baseline. We compare SPOT against the following machine learning and deep
learning models, mostly following the setups in (Fu et al., 2022).

  * •

LR (Lo et al., 2019): logistic regression with the default hyperparameters
implemented by scikit-learn (Pedregosa et al., 2011).

  * •

RF (Lo et al., 2019): random forests implemented by scikit-learn (Pedregosa et
al., 2011).

  * •

XGBoost (Siah et al., 2021): a gradient-boosted decision tree method.

  * •

AdaBoost (Fan et al., 2020): an adaptive boosting-based decision tree method
implemented by scikit-learn (Pedregosa et al., 2011).

  * •

kNN+RF (Lo et al., 2019): handling missing data by imputation with the k
nearest neighbors and predict using random forests.

  * •

FFNN (Tranchevent et al., 2019): a feedforward neural network that uses the
same features as in (Fu et al., 2022). It has 3 fully-connected layers with
the dimensions of dim-of-input-feature, 500, and 100, and ReLU activations.

  * •

DeepEnroll (Zhang et al., 2020): it was adapted here for encoding clinical
trial eligibility criteria: a hierarchical embedding network to encode disease
ontology, and an alignment model to capture interactions between eligibility
criteria and disease information. We concatenate the criteria embedding with
the molecule embedding obtained by MPNN for adapting it to trial outcome
prediction.

  * •

COMPOSE (Gao et al., 2020): it was originally used for patient-trial matching
based on a convolutional neural network and a memory network to encode
eligibility criteria and diseases, respectively. We also concatenate its
embedding with a molecule embedding by MPNN for trial outcome prediction.

  * •

HINT (Fu et al., 2022): it is the state-of-the-art trial outcome prediction
model that incorporates (1) a drug molecule encoder based on MPNN; (2) a
disease ontology encoder based on GRAM; (3) a trial eligibility criteria
encoder based on BERT; (4) a drug molecule pharmacokinetic encoder; (5) graph
neural network for feature interactions. Then, it feeds the interacted
embeddings to a prediction model for outcome predictions.

To summarize, embeddings from HINT are used in all statistical machine
learning models (LR, RF, XGBoost, ADAboost, and kNN+RF) as the input for
predicting trial outcomes, while the deep learning models are added with an
additional molecule encoder.

###  4.2. Results

We conduct experiments to evaluate SPOT in the following aspects:

  * •

EXP 1. The overall performance of outcome predictions for phase I, II, and III
trials, respectively, compared with other baselines.

  * •

EXP 2. Analysis of the disparity of prediction performances across different
trial groups.

  * •

EXP 3. The in-depth analysis of SPOT in terms of its sensitivity and its
sequential modeling ability.

  * •

EXP 4. The ablation of the different components of SPOT.

### Exp 1. Performance of Trial Outcome Predictions

We show the trial outcome prediction performances of all compared methods in
Tables 2, 3, 4, for phase I, II, and III trials, respectively. The results
observed show that SPOT, demonstrates superior performance over all the
baselines across all metrics. Specifically, SPOT yields a significant
performance lift among phase I trials over baselines. In particular,
preventing a failing phase I trial can have a substantial impact on the entire
drug development process. Accurate prediction of Phase I trial outcomes can
allow us to identify more promising clinical trials and improve the chance of
discovering new drugs.

Table 2. Trial outcome prediction results compared between SPOT and baselines
for phase I trials.

Phase I Trials [t]
---
# train: 1,044; # valid: 116; # test: 627; # patients per trial: 45 [b]
Method | PR-AUC | F1 |  ROC-AUC [t]
LR | 0.500 ± 0.005 | 0.604 ± 0.005 | 0.520 ± 0.006
RF | 0.518 ± 0.005 | 0.621 ± 0.005 | 0.525 ± 0.006
XGBoost | 0.513 ± 0.06 | 0.621 ± 0.007 | 0.518 ± 0.006
AdaBoost | 0.519 ± 0.005 | 0.622 ± 0.007 | 0.526 ± 0.006
kNN+RF | 0.531 ± 0.006 | 0.625 ± 0.007 | 0.538 ± 0.005
FFNN | 0.547 ± 0.010 | 0.634 ± 0.015 | 0.550 ± 0.010
DeepEnroll | 0.568 ± 0.007 | 0.648 ± 0.011 | 0.575 ± 0.013
COMPOSE | 0.564 ± 0.007 | 0.658 ± 0.009 | 0.571 ± 0.011
HINT | 0.567 ± 0.010 | 0.665 ± 0.010 | 0.576 ± 0.008
SPOT | 0.689 ± 0.009 | 0.714 ± 0.011 | 0.660 ± 0.008 [b]

Table 3. Trial outcome prediction results compared between SPOT and baselines
for phase II trials.

Phase II Trials [t]
---
# train: 4,004; # valid: 445; # test: 1,653; # patients per trial: 183 [b]
Method | PR-AUC | F1 |  ROC-AUC [t]
LR | 0.565 ± 0.005 | 0.555 ± 0.006 | 0.587 ± 0.009
RF | 0.578 ± 0.008 | 0.563 ± 0.009 | 0.588 ± 0.009
XGBoost | 0.586 ± 0.006 | 0.570 ± 0.009 | 0.600 ± 0.007
AdaBoost | 0.586 ± 0.009 | 0.583 ± 0.008 | 0.603 ± 0.007
kNN+RF | 0.594 ± 0.008 | 0.590 ± 0.006 | 0.597 ± 0.008
FFNN | 0.604 ± 0.010 | 0.599 ± 0.012 | 0.611 ± 0.011
DeepEnroll | 0.600 ± 0.010 | 0.598 ± 0.007 | 0.625 ± 0.008
COMPOSE | 0.604 ± 0.007 | 0.597 ± 0.006 | 0.628 ± 0.009
HINT | 0.629 ± 0.009 | 0.620 ± 0.008 | 0.645 ± 0.006
SPOT | 0.685 ± 0.010 | 0.656 ± 0.009 | 0.630 ± 0.007 [b]

Table 4. Trial outcome prediction results compared between SPOT and baselines
for phase III trials.

Phase III Trials [t]
---
# train: 3,092; # valid: 344; # test: 1,140; # patients per trial: 1,418 [b]
Method | PR-AUC | F1 |  ROC-AUC [t]
LR | 0.687 ± 0.005 | 0.698 ± 0.005 | 0.650 ± 0.007
RF | 0.692 ± 0.004 | 0.686 ± 0.010 | 0.663 ± 0.007
XGBoost | 0.697 ± 0.007 | 0.696 ± 0.005 | 0.667 ± 0.005
AdaBoost | 0.701 ± 0.005 | 0.695 ± 0.005 | 0.670 ± 0.004
kNN+RF | 0.707 ± 0.007 | 0.698 ± 0.008 | 0.678 ± 0.010
FFNN | 0.747 ± 0.011 | 0.748 ± 0.009 | 0.681 ± 0.008
DeepEnroll | 0.777 ± 0.008 | 0.786 ± 0.007 | 0.699 ± 0.008
COMPOSE | 0.782 ± 0.008 | 0.792 ± 0.007 | 0.700 ± 0.007
HINT | 0.811 ± 0.007 | 0.847 ± 0.009 | 0.723 ± 0.006
SPOT | 0.856 ± 0.008 | 0.857 ± 0.008 | 0.711 ± 0.005 [b]

Figure 4. The calibration plots of our method SPOT and the best baseline HINT.
We split the raw trials into a series of subgroups and calculate the average
ground truth and predicted success rate of these groups. The red dotted line
is the theoretical upper bound of the performance where the prediction is
perfectly calibrated, such that the closer to it the better the prediction.

Figure 5. Predicted probability distributions of SPOT and HINT. The x-axis
shows topic indices determined by SPOT. The red line indicates the actual
average success rates of different topics with associated confidence
intervals.

In addition, from Tables 2, 3, 4, we also observe the superior performance of
SPOT in terms of PR-AUC: it obtains 8.9% lift in phase II trials and 5.5% lift
in phase III trials, respectively, which indicates that SPOT has the strength
in making well-calibrated probability predictions. Nonetheless, it is found
that SPOT is slightly weaker than HINT in ROC-AUC for these trials, which
implies it’s a potential drawback of ranking the success probabilities across
trials. We dive deep to investigate the cause, plotting the comparison between
the probability distributions of SPOT and HINT in Fig. 5. In particular, we
divide the trials by their topics that are assigned by the topic discovery
module of SPOT. We also plot the ground truth average success rates of each
topic in red dotted lines as a reference.

Fig. 5 shows the predictions generated by SPOT exhibit a higher degree of
concentration (i.e., narrow ranges across different topics) around the ground
truth average success rates (red dash lines), compared to the predictions
generated by HINT (with wider ranges across topics and not calibrated with
ground truth). This confirmed the benefit of the task-specific modeling
employed by SPOT, which utilizes a distinct set of parameters for each task.
This feature enables SPOT to more accurately capture the actual success
probabilities of trials, taking into account the specific trial topic.

Figure 6. The analysis of SPOT on the performance varies according to the
adjusted number of topics.

### Exp 2. Performance Across Trial Groups

To assess if the predicted trial outcomes are well calibrated with the true
probabilities, we report the calibration plot of our method SPOT and HINT in
Fig. 4, for phases I, II, and III trial data, respectively. The calibration
plot is often used in ML applications such as classification, where the aim is
to predict the probability of an event, such as success or failure. In a well-
calibrated model, the predicted probabilities should align with the actual
outcomes, with high predicted probabilities indicating high actual
probabilities, and low predicted probabilities indicating low actual
probabilities.

In Fig. 4, it is witnessed that SPOT (blue line) is in general closer to the
perfect calibration line (the 45-degree diagonal red dotted line), especially
for phase II and III trials. That means, the SPOT’s predicted probability can
be a better estimate of the true trial success probability. Specifically, we
find HINT is prone to overestimating the success probability for those success
trials while underestimating the probability for those failure trials. For
instance, in the phase I calibration plot (left in Fig. 4), in the region
where $x\in[0,0.5]$, the HINT line is above the perfect line, meaning the
predicted trials with around 35% success rates have an actual success rate of
around 50%; while in the region $x\in[0.6,0.8]$, the HINT line is below the
perfect line, meaning the predicted trials with around 70% success rates only
have an actual success rate of around 58%.

In comparison, SPOT typically aligns closely with the perfect line in the
range $x\in[0.5,1.0]$ for all three phases and delivers a more accurate
prediction quality for the failure trials. For failure trials, SPOT tends to
slightly overestimate in phase I while underestimating in phases II and III,
reflecting the challenges in predicting these outcomes due to the
unpredictable nature of complex factors that cause the failure of trials.

We also report the relative success and failure proportion divided by the
predicted success probability on the test set in Fig. 8 for further
visualization of the predictions made by HINT and SPOT. To be specific, our
method is more accurate in identifying failed trials, as they constitute a
larger proportion of the group with the lowest predicted success probability.
This feature can provide valuable insight for investigators to optimize their
trial design and reduce the likelihood of executing failing trials. On the
other hand, HINT has been exhibited to be less accurate in identifying failed
trials. This is attributed to the fact that failed trials do not constitute a
majority of the group with the lowest predicted success probability by HINT.
Additionally, the success-failure ratio across groups illustrates that the
predictions made by SPOT deviate less from the ground truth ratio. It is worth
noting that, generally, the identification of failed trials is a more
challenging task than the identification of successful trials for both.
Despite it, the superior performance of SPOT in identifying failed trials
renders it a valuable tool for investigators to improve the design and
execution of clinical trials.

### Exp 3. In-depth Analysis

Sensitivity Analysis. A key insight of SPOT is to build the topic-specific
models because the trial outcomes often depend on their trial topics. The
number of topics $K$ becomes one critical hyperparameter that needs to be
properly chosen. To test the sensitivity of our method with respect to $K$, we
train SPOT with varying $K$ on three phases of trial data separately. We set
$K\in\\{1,3,5,10,30\\}$ for phase I trials, $K\in\\{10,30,50,100,200\\}$ for
phase II and III trials, respectively. The results are reported in Fig. 6. The
ranges of $K$ for phase II and III are larger than that of phase I because
there are more trials from phase II and III in the training data.

As shown in Fig 6, SPOT is reasonably robust to different topic numbers $K$ as
SPOT almost always outperforms HINT with all $K$ selections. This is
particularly useful as the optimal number of topics can vary depending on the
size and complexity of the data. In particular, the optimal topic number for
phase I trials is 3 while is 50 for phases II and III in our experiments. The
training data of phase II and III trials are both around three times bigger
than phase I and the primary measures for the later stage trials are much more
diverse than the early ones. Most phase I trials focus on evaluating the
treatment safety while the phase II and III trials will also consider the
efficacy considering distinct diseases. Moreover, phases II and III trials
typically involve a larger number of patients and a wider range of diseases,
which can result in a more diverse set of outcomes. This increased diversity
may require a more granular topic-aware modeling approach to capture the
nuances of the data.

Figure 7. The analysis of SPOT on modeling the progression of clinical trial
outcomes along the timesteps within each topic. The results are obtained on
the test set of phase I trials.

Trial Outcome Progression. The ability of SPOT to model the sequential
progression of trial outcomes within each trial topic is a key feature that
sets it apart from other methods. This is because it allows for a deeper
understanding of the temporal patterns of trial outcomes and how they are
affected by technological advancement. In Fig. 7, we present several examples
of the progression modeling results in the topics by our method. These
examples are specifically chosen from the test set of phase I trials, and they
show the average ground truth success rates of the trials, as well as the
average predicted probabilities, along the timestep in the topic.

The result illustrates that SPOT is able to accurately capture the temporal
progression of trial outcomes as the predicted probabilities closely match the
ground truth success rates. The ability to track the success rate over time
allows an understanding of how the technology and the treatments evolve. It
can also identify the points where the success rate is increasing or
decreasing and by how much. This can provide valuable insights for researchers
to track the progress of drug development and identify the trends over time,
which can inform future research and development.

### Exp 4. Ablation Study

Our method constitutes three main components including the topic discovery
module, the sequential modeling module, and the meta-learning module. We
conduct ablations on them to estimate the effects of each component
contributed to the final performance. We test the variants that remove topic
discovery (w/o Topic), and remove sequential modeling (w/o Sequence), and
remove meta-learning (w/o Meta-Learning). Note that the w/o Topic variant also
does not include meta-learning because we only have one task in this case. We
report the results in Table 5.

The results reveal that the topic discovery module has a significant effect on
the overall performance. Specifically, when evaluating the results of phase II
and III trials, it is evident that the w/o Topic variant performs
significantly worse, with a low ROC-AUC score. The reason for this poor
performance is that aggregating all trials together and ordering them in
chronological order creates a random temporal pattern, resulting in trivial
predictions. On the other hand, incorporating sequential modeling improves the
performance of phase I trials significantly. But its impact is less
significant in Phase II and III trials, where the temporal pattern is more
complex.

Table 5. Ablation analysis of SPOT when each module is removed. w/o Topic:
without topic discovery; w/o Sequence: without sequential modeling; w/o Meta-
Learning: without meta-learning.

Method | PR-AUC | F1 |  ROC-AUC [b]
---|---|---|---
Phase I Trials |  |  |  [t]
SPOT | 0.689 | 0.714 | 0.660 [t]
w/o Topic | 0.604 | 0.713 | 0.569
w/o Sequence | 0.657 | 0.713 | 0.630
w/o Meta-Learning | 0.641 | 0.710 | 0.582 [b]
Phase II Trials |  |  |  [t]
SPOT | 0.685 | 0.656 | 0.630 [t]
w/o Topic | 0.573 | 0.521 | 0.516
w/o Sequence | 0.678 | 0.649 | 0.621
w/o Meta-Learning | 0.663 | 0.646 | 0.615 [b]
Phase III Trials |  |  |  [t]
SPOT | 0.856 | 0.857 | 0.711 [t]
w/o Topic | 0.769 | 0.813 | 0.518
w/o Sequence | 0.850 | 0.857 | 0.699
w/o Meta-Learning | 0.799 | 0.812 | 0.685 [b]

##  5\. Conclusion

In conclusion, the accurate prediction of clinical trial outcomes is crucial
for saving time and funding for pharmaceutical companies, as it allows for the
early detection of trial failure. Previous approaches, however, suffer from
poor performance for trials belonging to less common topics. In this work, we
propose a meta-learning approach, called SPOT, which models a topic of
clinical trials in sequence instead of modeling each trial independently. By
using trial topic discovery to define the topic, our approach can take into
account the relations among trials and the distinction between groups of
trials. Our results show that SPOT outperforms previous methods by a
significant margin.

Future work for this approach could include incorporating additional sources
of information, such as patient data or genetic information, to improve the
performance of the model. More research could also be done on the
interpretability of the model, such as identifying which factors have the most
impact on trial outcomes. Additionally, it may be of interest to develop more
sophisticated topic discovery method thus enhancing the task-aware modeling
for clinical trial outcome predictions.

## References

  * (1)
  * Blass (2015) Benjamin E Blass. 2015\.  _Basic principles of drug discovery and development_.  Elsevier.
  * Brock et al. (2018) Andrew Brock, Theo Lim, JM Ritchie, and Nick Weston. 2018\.  SMASH: One-Shot Model Architecture Search through HyperNetworks. In _International Conference on Learning Representations_.
  * Choi et al. (2017) Edward Choi, Mohammad Taha Bahadori, Le Song, Walter F Stewart, and Jimeng Sun. 2017.  GRAM: graph-based attention model for healthcare representation learning. In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. 787–795.
  * Duan et al. (2016) Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. 2016\.  RL^ 2: Fast Reinforcement Learning via Slow Reinforcement Learning. In _ICLR_.
  * Fan et al. (2020) Zhao Fan, Fanyu Xu, Cai Li, and Lili Yao. 2020\.  Application of KPCA and AdaBoost algorithm in classification of functional magnetic resonance imaging of Alzheimer’s disease.  _Neural Computing and Applications_ 32, 10 (2020), 5329–5338.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017\.  Model-agnostic meta-learning for fast adaptation of deep networks. In _International Conference on Machine Learning_. PMLR, 1126–1135.
  * Fu et al. (2022) Tianfan Fu, Kexin Huang, Cao Xiao, Lucas M Glass, and Jimeng Sun. 2022.  HINT: Hierarchical interaction network for clinical-trial-outcome predictions.  _Patterns_ 3, 4 (2022), 100445.
  * Gao et al. (2020) Junyi Gao, Cao Xiao, Lucas M Glass, and Jimeng Sun. 2020\.  COMPOSE: cross-modal pseudo-siamese network for patient trial matching. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 803–812.
  * Gayvert et al. (2016) Kaitlyn M Gayvert, Neel S Madhukar, and Olivier Elemento. 2016\.  A data-driven approach to predicting successes and failures of clinical trials.  _Cell chemical biology_ 23, 10 (2016), 1294–1301.
  * Gidaris and Komodakis (2018) Spyros Gidaris and Nikos Komodakis. 2018.  Dynamic few-shot visual learning without forgetting. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4367–4375.
  * Gilmer et al. (2017) Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017\.  Neural message passing for quantum chemistry. In _International Conference on Machine Learning_. PMLR, 1263–1272.
  * Gupta et al. (2018) Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. 2018.  Meta-reinforcement learning of structured exploration strategies.  _Advances in Neural Information Processing Systems_ 31 (2018).
  * Kang et al. (2019) Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. 2019.  Decoupling Representation and Classifier for Long-Tailed Recognition. In _International Conference on Learning Representations_.
  * Li et al. (2017) Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. 2017\.  Meta-SGD: Learning to learn quickly for few-shot learning.  _arXiv preprint arXiv:1707.09835_ (2017).
  * Liu et al. (2018) Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2018\.  DARTS: Differentiable Architecture Search. In _International Conference on Learning Representations_.
  * Lo et al. (2019) Andrew W Lo, Kien Wei Siah, and Chi Heem Wong. 2019\.  Machine learning with statistical imputation for predicting drug approval.  _Harvard Data Science Review_ (June 2019).
  * Marshall et al. (2017) Iain J Marshall, Joël Kuiper, Edward Banner, and Byron C Wallace. 2017.  Automating biomedical evidence synthesis: RobotReviewer. In _Proceedings of the conference. Association for Computational Linguistics. Meeting_ , Vol. 2017. NIH Public Access, 7.
  * Nichol et al. (2018) Alex Nichol, Joshua Achiam, and John Schulman. 2018\.  On first-order meta-learning algorithms.  _arXiv preprint arXiv:1803.02999_ (2018).
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine Learning in Python.  _Journal of Machine Learning Research_ 12 (2011), 2825–2830.
  * Qi and Tang (2019) Youran Qi and Qi Tang. 2019\.  Predicting phase 3 clinical trial results by modeling phase 2 clinical trial subject level data using deep learning. In _Machine Learning for Healthcare Conference_. PMLR, 288–303.
  * Ravi and Larochelle (2016) Sachin Ravi and Hugo Larochelle. 2016.  Optimization as a model for few-shot learning. In _ICLR_.
  * Schmidhuber (1987) Jürgen Schmidhuber. 1987\.  _Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-… hook_.  Ph. D. Dissertation. Technische Universität München.
  * Siah et al. (2021) Kien Wei Siah, Nicholas W Kelley, Steffen Ballerstedt, Björn Holzhauer, Tianmeng Lyu, David Mettler, Sophie Sun, Simon Wandel, Yang Zhong, Bin Zhou, et al. 2021\.  Predicting drug approvals: the novartis data science and artificial intelligence challenge.  _Patterns_ 2, 8 (2021), 100312.
  * Srivastava et al. (2015) Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015.  Highway networks.  _arXiv preprint arXiv:1505.00387_ (2015).
  * Thrun and Pratt (1998) Sebastian Thrun and Lorien Pratt. 1998.  Learning to learn: Introduction and overview.  In _Learning to learn_. Springer, 3–17.
  * Tranchevent et al. (2019) Léon-Charles Tranchevent, Francisco Azuaje, and Jagath C Rajapakse. 2019.  A deep neural network approach to predicting clinical outcomes of neuroblastoma patients.  _BMC medical genomics_ 12, 8 (2019), 1–11.
  * Wang et al. (2022) Zifeng Wang, Chufan Gao, Lucas M Glass, and Jimeng Sun. 2022\.  Artificial Intelligence for In Silico Clinical Trials: A Review.  _arXiv preprint arXiv:2209.09023_ (2022).
  * Wang and Sun (2022a) Zifeng Wang and Jimeng Sun. 2022a.  TransTab: Learning Transferable Tabular Transformers Across Tables. In _Advances in Neural Information Processing Systems_.
  * Wang and Sun (2022b) Zifeng Wang and Jimeng Sun. 2022b.  Trial2Vec: Zero-Shot Clinical Trial Document Similarity Search using Self-Supervision. In _Findings of EMNLP_.
  * Zhang et al. (2020) Xingyao Zhang, Cao Xiao, Lucas M Glass, and Jimeng Sun. 2020\.  DeepEnroll: patient-trial matching with deep embedding and entailment prediction. In _Proceedings of The Web Conference 2020_. 1029–1037.

Table 6. The dataset statistics of the trial’s target disease groups, in phase
I, II, and III clinical trials.

Phase I |  Phase II |  Phase III [t]
---|---|---
Disease group |  Num |  Disease group |  Num |  Disease group |  Num [b]
Neoplasms | 552 |  Neoplasms | 1931 |  Neoplasms | 637 [t]
Factors Influencing Health Status and Contact with Health Services | 149 |  Factors Influencing Health Status and Contact with Health Services | 405 |  Endocrine, Nutritional and Metabolic Diseases | 447
Endocrine, Nutritional and Metabolic Diseases | 99 |  Injury, Poisoning and Certain Other Consequences of External Causes | 288 |  Diseases of the Nervous System | 392
Diseases of the Nervous System | 71 |  Diseases of the Digestive System | 284 |  Diseases of the Circulatory System | 361
Certain Infectious and Parasitic Diseases | 70 |  Diseases of the Nervous System | 276 |  Factors Influencing Health Status and Contact with Health Services | 351
Injury, Poisoning and Certain Other Consequences of External Causes | 70 |  Diseases of the Circulatory System | 267 |  Mental, Behavioral and Neurodevelopmental Disorders | 329
Diseases of the Eye and Adnexa | 69 |  Endocrine, Nutritional and Metabolic Diseases | 258 |  Certain Infectious and Parasitic Diseases | 290
Diseases of the Digestive System | 54 |  Certain Infectious and Parasitic Diseases | 254 |  Diseases of the Digestive System | 284
Diseases of the Circulatory System | 52 |  Diseases of the Respiratory System | 242 |  Diseases of the Respiratory System | 272
Diseases of the Genitourinary System | 47 |  Diseases of the Genitourinary System | 230 |  Diseases of the Genitourinary System | 271
Certain Conditions Originating in the Perinatal Period | 32 |  Diseases of the Musculoskeletal System and Connective Tissue | 214 |  Certain Conditions Originating in the Perinatal Period | 247
Symptoms, Signs and Abnormal Clinical and Laboratory Findings, Not Elsewhere Classified | 28 |  Mental, Behavioral and Neurodevelopmental Disorders | 194 |  Diseases of the Musculoskeletal System and Connective Tissue | 227
Diseases of the Musculoskeletal System and Connective Tissue | 28 |  Certain Conditions Originating in the Perinatal Period | 133 |  Diseases of the Eye and Adnexa | 189
Diseases of the Respiratory System | 27 |  Diseases of the Eye and Adnexa | 103 |  Injury, Poisoning and Certain Other Consequences of External Causes | 114
Mental, Behavioral and Neurodevelopmental Disorders | 21 |  Congenital Malformations, Deformations and Chromosomal Abnormalities | 96 |  Symptoms, Signs and Abnormal Clinical and Laboratory Findings, Not Elsewhere Classified | 88
Congenital Malformations, Deformations and Chromosomal Abnormalities | 19 |  Diseases of the Blood and Blood Forming Organs and Certain Disorders Involving the Immune Mechanism | 83 |  Diseases of the Blood and Blood Forming Organs and Certain Disorders Involving the Immune Mechanism | 84
Diseases of the Blood and Blood Forming Organs and Certain Disorders Involving the Immune Mechanism | 18 |  Diseases of the Skin and Subcutaneous Tissue | 80 |  Congenital Malformations, Deformations and Chromosomal Abnormalities | 77
Diseases of the Skin and Subcutaneous Tissue | 11 |  Symptoms, Signs and Abnormal Clinical and Laboratory Findings, Not Elsewhere Classified | 51 |  Pregnancy, Childbirth and the Puerperium | 44
External Causes of Morbidity | 11 |  Pregnancy, Childbirth and the Puerperium | 25 |  Diseases of the Skin and Subcutaneous Tissue | 40
Pregnancy, Childbirth and the Puerperium | 9 |  External Causes of Morbidity | 22 |  External Causes of Morbidity | 19
Diseases of the Ear and Mastoid Process | 5 |  Diseases of the Ear and Mastoid Process | 5 |  Diseases of the Ear and Mastoid Process | 5 [b]

Figure 8. The predicted success probability versus relative success/failure
proportion of our method SPOT (top) and the baseline HINT (bottom). The
predicted probabilities of SPOT reflect the actual distribution of success and
failure trials, where failure trials are mostly found in the low predicted
probability group. On the other hand, the baseline method HINT shows a less
distinct division between success and failure trials in its predicted
probabilities.