# Using LLMs to label medical papers according to the CIViC evidence model

Markus Hisch, Xing David Wang
Humboldt-Universität zu Berlin
<EMAIL_ADDRESS>
###### Abstract

We introduce the sequence classification problem CIViC Evidence to the field
of medical NLP. CIViC Evidence denotes the multi-label classification problem
of assigning labels of clinical evidence to abstracts of scientific papers
which have examined various combinations of genomic variants, cancer types,
and treatment approaches.

We approach CIViC Evidence using different language models: We fine-tune
pretrained checkpoints of BERT and RoBERTa on the CIViC Evidence dataset and
challenge their performance with models of the same architecture which have
been pretrained on domain-specific text. In this context, we find that
BiomedBERT and BioLinkBERT can outperform BERT on CIViC Evidence (+0.8% and
+0.9% absolute improvement in class-support weighted F1 score). All
transformer-based models show a clear performance edge when compared to a
logistic regression trained on bigram tf-idf scores (+1.5 – 2.7% improved F1
score).

We compare the aforementioned BERT-like models to OpenAI’s GPT-4 in a few-shot
setting (on a small subset of our original test dataset), demonstrating that,
without additional prompt-engineering or fine-tuning, GPT-4 performs worse on
CIViC Evidence than our six fine-tuned models (66.1% weighted F1 score
compared to 71.8% for the best fine-tuned model). However, performance gets
reasonably close to the benchmark of a logistic regression model trained on
bigram tf-idf scores (67.7% weighted F1 score).

Using LLMs to label medical papers according to the CIViC evidence model

Markus Hisch, Xing David Wang Humboldt-Universität zu Berlin
<EMAIL_ADDRESS>

##  1 Introduction

Rare malignancies continue to pose unique challenges in cancer treatment
([1]). Addressing this phenomenon, tools like next-generation sequencing (NGS)
have become essential for identifying specific genomic variants that inform
targeted therapies ([19]).

As NGS has permeated clinical practice, Molecular Tumor Boards (MTBs) have
emerged as collaborative forums, uniting multidisciplinary teams of
oncologists, pathologists, bioinformaticians, and other researchers. These
boards are dedicated to analyzing and discussing cases where standard
treatment options like chemotherapy have proven ineffective. The central
objective of MTBs is to develop highly personalized treatment strategies based
on thorough analysis of a patient’s genomic variants ([25]).

Hence, access to relevant data relating genomic variants and tumor
characteristics with druggability is paramount for creating effective
treatment plans. Such data is usually collected by consulting medical
databases combined with extensive literature research ([37]).

#### The sequence classification problem CIViC Evidence

CIViC is an open database curated by experts, facilitating clinical
interpretations of variants in cancer. It includes a table (Evidence Table)
mapping tuples of genetic variants, concrete cancer manifestations, and
treatment approaches to labels indicating different levels of clinical
evidence. The label of clinical evidence is inferred from peer-reviewed papers
that have investigated underlying variants and their clinical response to
treatments ([9]).

However, this process of data labeling still relies on manual expertise. Given
an ever-growing corpus of relevant research literature, this approach faces
scalability challenges.

Addressing this issue, Machine Learning Systems, particularly Large Language
Models (LLMs), have shown promising results across a variety of clinical
language understanding tasks, including sequence classification, albeit with
limitations ([40]).

We propose and compare different Natural Language Processing (NLP) models, all
based on the transformer architecture ([39]), to predict these evidence levels
using the abstract of the paper associated to it in the CIViC Evidence Table.
We will refer to this dataset and to the corresponding classification problem
as CIViC Evidence. Trained models can then be leveraged to automatically infer
labels of clinical evidence from medical publications and abstracts, thus
reducing the need for manual data labeling.

#### Research questions

In the context of the sequence classification problem CIViC Evidence, we seek
to answer the following questions:

  1. 1.

Can transformer-based language models outperform competitive models which are
not based on transformers?

  2. 2.

Do different transformer-based models benefit from domain-specific
pretraining?

  3. 3.

Do BERT-like models benefit from increasing context-width (see Biomed-RoBERTa-
Long in section 4.2) given that a significant portions of abstracts in CIViC
Evidence are longer than the 512 token window of BERT or RoBERTa?

  4. 4.

Can very large language models like GPT-4 outperform fine-tuned BERT-based
language models in a few-shot setting (section 2.4)?

#### Main contributions

We show that various transformer-based models can significantly outperform tf-
idf score based logistic regressions on CIViC Evidence. Furthermore, we
demonstrate that domain-specific pretraining can further increase performance
of BERT and RoBERTa on CIViC Evidence. We propose a new RoBERTa-based
transformer (Biomed-RoBERTa-Long) with a context width of 1024 tokens
(compared to 512 tokens for Biomed-RoBERTa). We perform domain-specific
pretraining of this model on long PubMed abstracts and show that this
increases downstream performance in the context of CIViC Evidence. Lastly, we
demonstrate that GPT-4, in a few-shot setting without additional prompt-
engineering or fine-tuning, performs significantly worse on CIViC Evidence
than our fine-tuned models based on BERT or RoBERTa. Our code is made
accessible as a Github repository at
https://github.com/GMahlerTheTragic/civic.

##  2 Background

###  2.1 The role of automatic evidence classification in clinical practice

In this section, our aim is to underscore the importance of automated
inference of evidence labels from pertinent research. Fundamentally, cancer is
a disease caused by genetic alterations, which contribute to uncontrolled
proliferation of malignant cells. However, cancer is seldomly caused by a
single genetic mutation, but by a series of mutations affecting multiple
genes, combined with failed DNA repair and apoptosis mechanisms. Proto-
oncogenes and tumor suppressor genes are the primary targets of carcinogenic
processes. As they are essential for regulating cell proliferation and
differentiation, alterations in their genetic structure can lead to faulty
gene products and unbalanced gene expression, eventually resulting in cell
degeneration ([14]).

Modern research in cancer and its causes has identified correlations between
the presence of various genetic alterations and the occurrences of particular
kinds of cancer. Precision oncology tries to exploit these links by applying
treatments targeting these molecular causes of cancer. To facilitate this
approach, NGS tools usually sequence a patient’s entire genome aiming at
identifying the unique combination of genetic alterations associated with
their disease ([36]).

As a next step, groups of specialists (Molecular Tumor Boards) can perform
extensive literature research with the goal of identifying promising treatment
approaches for the patient’s illness ([25]).

Practitioners are confronted with the dual task of handling the large variety
of genomic alterations linked to a particular disease, while also staying
abreast of the ever-growing body of research linked to these alterations. In
fact, a simple PubMed keyword search relating Pancreatic Cancer with different
molecular profiles yields tens of thousands of potentially relevant research
publications (Fig. 1).

Figure 1: PubMed keyword search matches for Pancreatic Cancer and various
molecular profiles

Adding the different medication options to this analysis further complicates
the picture: Treating physicians are confronted with a plethora of research on
hundreds of combinations of different molecular profiles and treatment
approaches. Moreover, these research corpora have often been growing at an
exponential rate (Fig. 2).

Figure 2: PubMed keyword search matches for Pancreatic Cancer and BRCA1
Mutation over time

In practice, only few of these publications are relevant for an individual
patient’s treatment. Consequently, it is imperative to pinpoint research
pertinent to the specific case at hand, particularly research that advocates
treatment approaches supported by robust clinical evidence.

Precision oncology information systems ([37]) aim to streamline this process
by automatically identifying and filtering research with significant clinical
implications, thereby enhancing targeted patient care. Achieving this goal can
be accomplished through a two-step approach. Initially, text-mining software
can scan research articles, focusing on the intersection of malignant
diseases, genomic alterations, and treatment strategies. Subsequently, a
language model can assign levels of clinical evidence to these articles. Once
labeled, these articles form the foundation of a curated database, accessible
to healthcare providers. While the creation of such databases traditionally
relies on manual curation, integrating automatic evidence classification can
potentially improve both, the quality and economic feasibility, of compiling
these resources ([12]).

###  2.2 Natural Language Processing using Transformers

Since the groundbreaking introduction of the transformer architecture in
([39]), the field of Natural Language Processing (NLP) has experienced
significant and continuous progress.

Notably, Google’s BERT ([5]) model and OpenAI’s GPT model family ([31, 32, 4,
29]) can be seen as key milestones, culminating in the recent achievements of
GPT-3.5 and GPT-4.

Transformers can be roughly classified into encoder-only transformers (BERT-
like models, as in ([5])), decoder-only transformers (GPT-like models, as in
([31])), and encoder-decoder transformers (the original transformer, as
introduced in ([39])). Encoder-only transformers have traditionally excelled
in language comprehension tasks such as sequence classification, while
decoder-only architectures have been used (and continue to be used) for
generative language tasks. Encoder-decoder transformers can be used for
sequence-to-sequence tasks, such as machine translation ([22]).

Fig. 3 shows a prototypical transformer architecture for either an encoder-
only or decoder-only architecture (depending on the attention pattern used).
Encoder-only architectures generally make use of unmasked self-attention,
whereas decoder-only architectures require the use of masked self-attention.

Figure 3: Possible architecture of encoder-only and decoder-only transformers

We kindly refer the reader to relevant literature discussing particular
transformer flavours in greater detail (e.g. ([23])). Furthermore, it seems
prudent to mention that transformers have found applications in other fields
of machine learning beyond NLP. This includes the field of Computer Vision
([7]). Some of the sections which follow (e.g. Tokenization) may not be
applicable to these kinds of transformers.

###  2.3 Task-specific prediction heads: Pretraining & Fine-tuning

Large Language Models are typically pretrained on vast corpora of mixed-domain
text, enabling them to develop a comprehensive understanding of language. One
major advantage of this approach is the possibility to leverage unsupervised
training on text data by means of e.g. masked language modeling as in ([5]),
eliminating the need for large labeled datasets.

#### Pretraining by means of masked language modeling (MLM)

The MLM objective involves randomly masking a certain percentage of the input
tokens in a sequence and tasking the model with predicting the masked tokens
based on the context provided by the surrounding words. This self-supervised
learning approach allows language models to learn rich, contextualized
representations that capture syntactic, semantic, and contextual nuances in
language. For example, the training procedure using MLM for BERT ([5])
involves the following steps:

  1. 1.

Masking Tokens: Randomly, a certain percentage (15% in ([5])) of tokens in the
sequence are masked, meaning they are replaced by a special token <MASK>.

  2. 2.

All masked tokens are set to a random token from the vocabulary with a
probability of 10% and reverted to the original unmasked token with a
probability of 10%.

  3. 3.

The token sequence is processed by an encoder-only transformer and the model
is trained to predict the original token from the encodings of the masked
tokens. Here, the corresponding cross-entropy loss function only takes into
account the positions of masked tokens (which is different from e.g. auto-
encoders).

This learning objective allows for the use of unmasked self-attention without
information leak. Architecture-wise this training procedure is realized using
a linear layer which realizes the following transformation:

$\displaystyle\text{Linear(X)}=XW$ (1)

Let $X\in\mathbb{R}^{n\times e}$ denote the encodings of the input sequence
after all transformer blocks have been applied. The application of
$W\in\mathbb{R}^{e\times v}$ allows for the computation of a logit matrix. The
maxima over the rows denote the predicted token for each input token. Here,
$n$, $e$, $v$ denote context width, embedding dimension (assumed to be
constant for all transformer blocks) and vocabulary size, respectively.

#### Domain-specific pretraining

Additionally, in domains like the biomedical field with ample text data
available, models have been exclusively pretrained on domain-specific text
corpora (for example using only papers from PubMed). Examples include
BiomedBERT ([10]), LinkBERT ([42]), ClinicalLongformer ([21]), and BioMedLM
([3]). These models have shown elevated performance on various biomedical
benchmarks when compared to their mixed-domain counterparts.

#### Fine-tuning for sequence classification

To apply pretrained models to specific tasks like sequence classification,
transfer learning in the form of fine-tuning can be employed. This involves
updating model weights through gradient descent and adding custom model layers
(e.g. Classification Heads), learning from labeled datasets curated for the
concrete task at hand.

For a multi-label classification task (i.e. multiple labels can be assigned to
the same item - labels are not mutually exclusive) comprised of $l$ different
labels the classification-specific model head needs to output a logit vector
of dimension $l$.

In BERT-like models (including RoBERTa) this is achieved by first pooling the
final encodings and then applying a last linear transformation. One option to
perform this, is to only keep the encoding of the <BOS> token (<CLS> in
([5])). The final logit computation is then performed as:

$\displaystyle\text{logits}=e_{\text{cls}}W\in\mathbb{R}^{l}$ (2)

where $e_{\text{cls}}\in\mathbb{R}^{1\times e}$ denotes the encoding of the
<CLS> token and $W\in\mathbb{R}^{e\times l}$. To perform the actual label
prediction, logit entries for every class can be classified as positive if the
value is greater than a certain class-specific threshold (e.g. zero).
Otherwise, they are classified as negative.

###  2.4 Few-shot Learning

As models have grown larger, the new learning paradigm of In-Context Learning
([4]) has grown in significance, presenting an alternative to domain- and
task-specific fine-tuning. In such settings, models are presented with prompts
including examples of a context (task description) and the correct
output/label. Models are then expected to solve further instances of the task.
As opposed to model fine-tuning, no gradient updates are performed and example
datasets are usually much smaller.

In this setting, the term N-shot learning refers to the number of examples
used to condition the model on the task it is expected to solve.

Special cases include zero-shot and one-shot learning. Models like GPT-3.5 and
GPT-4 have shown promising results in biomedical language-processing tasks
despite being trained on general-domain data ([40]) hinting at strong In-
Context Learning capabilities.

###  2.5 Explainable AI (XAI) by means of Integrated Gradients

As machine learning systems become more integrated into various aspects of
life, understanding why and how these systems reach specific conclusions is
imperative for fostering trust, accountability, and ethical use.

Explainable AI (XAI) methods formally try to solve the problem of assigning
attributions towards an output to inputs of machine learning methods.
Integrated Gradients, as introduced in ([38]), is an attribution algorithm
which addresses certain shortcomings of previous methods. That is, many
attribution algorithms do not fulfill three fundamental axioms which are
identified as desirable for any attribution algorithms designed for deep
neural networks:

######  Axiom (Sensitivity(a)).

An attribution method satisfies Sensitivity(a), if, for every input and
baseline that differ in one feature but have different predictions, the
differing feature is given a non-zero attribution.

######  Axiom (Sensitivity(b)).

If the function implemented by the deep network does not depend
(mathematically) on some variable, then the attribution to that variable is
always zero.

######  Axiom (Implementation Invariance).

Attribution methods should be implementation invariant, i.e., the attributions
are always identical for two functionally equivalent networks.

Integrated Gradients attributes a model prediction to its features by
integrating the gradients of the model output with respect to the input
features along a straight path from a baseline to the input. Formally,

$\displaystyle\text{ig}_{i}(x)=(x_{i}-x^{\prime}_{i})\int_{0}^{1}{\frac{\partial
F(x^{\prime}+\alpha(x-x^{\prime}))}{\partial x_{i}}\text{d}\alpha}$

Here, $x$ is a vector of input features and $x^{\prime}$ is a corresponding
vector of baseline features. In the context of linear models, this integration
simply corresponds to the product of inputs and coefficients (if we assume the
zero-vector as baseline).

Choosing a non-informative baseline provides a mechanism of translating these
attributions to absolute attributions for the input vector $x$. For image
models, a simple black image is a frequent choice. For text models, choosing
an all-zero embedding vector provides an uninformative choice, since it will
lead to an uniform distribution at the final softmax layer. Another choice for
text models is to choose the baseline to be a sequence comprised of only
padding tokens wrapped with <BOS> and <EOS> tokens.

In the context of transformer architectures, the integrals shown above can be
computed with respect to each entry of the matrix resulting from embedding and
positional encoding. They can then be summed over the embedding dimension $e$
to provide attributions on the token level.

##  3 Related work

The biomedical language domain has been a prominent target of inference in
Natural Language Processing (NLP) for the last 40 years ([8]). Adding to its
allure is the potential for a multitude of real-world applications. These
applications encompass diagnostic assistance, automated knowledge extraction
from textual sources, and decision-making support in treatment scenarios
([40]).

While the field of biomedical NLP has benefited from recent advances in
general-domain NLP, substantial amounts of digitized in-domain text has fueled
research on the topic of domain-specific adaptions of language models. Notable
examples of this approach include BiomedBERT ([10]) and BioLinkBERT ([42]).
([10]) trained the original BERT-architecture, as in ([5]), exclusively on
abstracts from PubMed. The resulting model has been shown to outperform BERT
on various benchmarks of clinical language understanding. BioLinkBERT follows
a similar idea, additionally exploiting the rich citation (link) structure of
PubMed papers.

One striking benefit of domain-specific pretraining which the authors outline
in ([10]) is the possibility to train domain-specific tokenizers: The
tokenizer underlying BiomedBERT recognizes the word "Imatinib" (a drug for
treating leukemia ([15])) as a whole token, whereas the BERT tokenizer would
produce the three tokens "im atin ib".
These encoder-only models have to be fine-tuned on concrete clinical language
understanding tasks such as medical multiple choice tasks. They are also
natural candidates to be fine-tuned on text classification tasks akin to CIViC
Evidence.

With the advent of very large closed weight language models such as GPT-3.5
([4]) and GPT-4 ([29]), fine-tuning has become less practical. This is due to
the fact that exact model snapshots and architectures may not be available and
classical fine-tuning for all sorts of downstream applications like CIViC
Evidence may be prohibitively expensive. On the other hand, these models have
shown impressive few-shot performance across a wide range of biomedical
language understanding tasks without further fine-tuning ([40]).

Regarding the classification problem CIViC Evidence, we believe that we are
the first to explore this exact problem. However, similar studies related to
the prediction of clinical evidence levels from medical papers have been
performed. The methods used reflect the state of the art in medical NLP at the
time.

In ([33]), the authors use a rule-based information extraction model to
determine whether biomedical papers from a corpus dealing with the genetics
and molecular biology of Drosophila (fruit flies) contain experimental
evidence for certain gene products. This model was the winner of the 2002 KDD
cup, achieving an F1 score of 67% on the Gene Product task.

([17]) use conditional random fields to first extract relevant sentences from
biomedical paper abstracts and then classify these into labels roughly
corresponding to the PICO (Population Intervention Comparison Outcome) schema
commonly used in evidence based medicine. The authors report F1 scores between
66.9% and 80.9% for their classification problems.

In ([27]), a kernel-based Bayesian Network is used to predict if sentences
extracted from a collection of medical papers are clinically useful. The
authors propose an intricate feature engineering approach using various
biomedical NLP tools like MedTagger111https://github.com/medtagger/MedTagger
and SemRep ([16]). These features serve as input to the downstream Bayesian
Network which achieves a performance of 78% F1 score.

The Evidence Inference task as established in ([18]) and expanded to abstract-
only based prediction in ([6]) can serve as a proxy to CIViC Evidence and
provide guidance on which performance levels (the authors report 0.776
macro-F1 score) can be achieved by various model architectures (including but
not limited to transformer based models).

In ([20]), the authors propose a text mining strategy to extract papers which
fit the CIViC evidence model from a large corpus of papers from the biomedical
domain. This includes categorizing papers into the four CIViC evidence
types222https://civic.readthedocs.io/en/latest/model/evidence/type.html#understanding-
evidence-types (note that these denote a concept distinct from the evidence
levels) using comparatively simple NLP models. A notable limitation of the
approach presented in ([20]) is that the classification relies on a single
extracted sentence, which must encompass all the relevant information
necessary for evidence type annotation.

##  4 Methodology

In this section, we present our methodology for CIViC Evidence in more detail.
This includes providing an examination of the dataset, outlining any
subsequent processing steps. Additionally, a comprehensive description of the
machine learning architectures employed to address the classification problem
is presented.

###  4.1 The Dataset CIViC Evidence

Sample entries from this Evidence Table of the CIViC
database333https://civicdb.org/evidence/home are shown in Tab. 1:

Table 1: Sample entries from the Evidence table of the CIViC database Id |  Molecular Profile |  Disease |  Therapy |  Evidence Level |  PubMed Id
---|---|---|---|---|---
EID238 |  EGFR T790M |  Non-small Cell Lung
Carcinoma |  Erlotinib |  A \- Validated
Association |  25668228
EID1246 |  FGFR1 N546K |  Ewing
Sarcoma
Of Bone |  Ponatinib |  E \- Inferential
Association |  26179511
EID7033 |  ABL1 BCR::ABL V379I |  Chronic Myeloid Leukemia |  Imatinib Mesylate |  C \- Case Study |  17264298

Evidence IDs are always linked to one relevant publication (usually
retrievable from the online library PubMed444https://pubmed.ncbi.nlm.nih.gov/)
from which the evidence levels have been extracted by expert judgement. The
official CIViC
documentation555https://civic.readthedocs.io/en/latest/model/evidence/level.html#understanding-
evidence-levels (see Understanding Evidence Levels) defines five evidence
levels as shown in Tab. 2.

Table 2: The five different levels of clinical evidence as used in the CIViC project Level | Name |  Definition |  Description
---|---|---|---
A | Validated association |  Proven/consensus association in human medicine |  Validated associations are often in routine clinical practice already or are the subject of major clinical trial efforts
B | Clinical evidence |  Clinical trial or other primary patient data supports association |  The evidence should be supported by observations in multiple patients. Additional support from functional data is desirable but not required
C | Case study |  Individual case reports from clinical journals |  The study may have involved a large number of patients, but the statement was supported by only a single patient. In some cases, observations from just a handful of patients (e.g. 2–3) or a single family may also be considered a case study/report
D | Preclinical evidence |  In vivo or in vitro models support association |  The study may have involved some patient data, but support for this statement was limited to in vivo or in vitro models (e.g. mouse studies, cell lines, molecular assays, etc.)
E | Inferential association |  Indirect evidence |  The assertion is at least one step removed from a direct association between a molecular profile ( variant) and clinical relevance

It is worth mentioning that one abstract can be associated with multiple
evidence items. This may be the case when an abstract deals with multiple
therapies, molecular profile, or cancer manifestations, at once.

For model training and evaluation we use a self-compiled version of CIViC
Evidence retrieved via the CIViC GraphQL-
API666https://civicdb.org/api/graphiql. We include evidence items with
curation status Accepted, as well as those with status Under Review to not
reduce the amount of available training data. From 10109 items retrieved via
the API, we remove all items which meet one or more of the following criteria:

  * •

Abstract text or evidence level are not populated.

  * •

The combination of abstract, disease name, significance column, molecular
profile name, and therapy names is non-unique.

  * •

Either of disease name, significance column, molecular profile name, or
therapy names is not populated.

As a next step, we compile the data for multi label classification: For every
abstract, we compile boolean variables for all evidence levels, indicating
whether the abstract is associated with the evidence level or not. We base
evidence level predictions solely on the abstract of the paper associated to
it in the CIViC Evidence table. As of 2024/02/04, this yields a dataset
containing 3369 items.

The occurrences of each of the five evidence levels in this version of CIViC
Evidence are highly biased towards the three classes in the middle, as shown
in Tab. 3. To take into account this class imbalance, we make use of a
80%-10%-10% train-validation-test split which tries to maintain overall class
probabilities.

Table 3: Number of class occurrences in our version of CIViC Evidence | A | B | C | D | E
---|---|---|---|---|---
Overall | 150 (4.5%) | 1363 (40.5 %) | 1135 (33.7%) | 948 (28.1%) | 44 (1.3 %)
Train | 115 (4.3%) | 1091 (40.5 %) | 908 (33.7%) | 759 (28.1%) | 36 (1.3 %)
Validation | 18 (5.3%) | 136 (40.4 %) | 113 (33.5%) | 95 (28.2%) | 4 (1.2 %)
Test | 17 (5.0%) | 136 (40.4 %) | 114 (33.8%) | 94 (27.9%) | 4 (1.2 %)

The length of the token sequence which transformer based language models can
process is limited (e.g. BERT can process no more than 512 tokens in one
forward pass). A total of 207 abstracts (6.1 %) in CIViC Evidence are longer
than 512 tokens using the BiomedBERT tokenization strategy as in ([10]). We
call these abstracts long abstracts in the context of CIViC Evidence.

It is worth mentioning, that both, disease name, and molecular profile, carry
discriminative information about the evidence class. To illustrate this
phenomenon, we show examples in Fig. 4. As a consequence, we decide to not use
any further metadata for evidence level prediction. We test abilities of
language models to infer the label purely from related abstracts.

Figure 4: Evidence level distribution given two distinct disease names and
molecular profiles

###  4.2 Machine Learning Models

We outline our approach to the sequence classification problem CIViC Evidence
in Fig. 5:

Figure 5: Predicting labels of clinical evidence from abstracts of medical
publications. Shown abstract from ([41])

PubMed paper abstract are fed into an encoder-only transformer based on BERT
or RoBERTa. Input tokens are encoded and solely the encoding of the <CLS>
token is used in a final linear prediction layer (as is common practice in the
literature ([5])).

The classification head outputs logits for each evidence level which can be
turned into class probabilities if desired by applying a sigmoid function. If
the predicted probability is larger than a predefined threshold, the
prediction is that an abstract is related to the evidence level in question.

The following paragraph summarizes the models we compare regarding their
performance on CIViC Evidence.

#### BERT & RoBERTa pretrained on mixed-domain text data

We leverage pretrained versions of the BERT777We use the following snapshot:
https://huggingface.co/bert-base-uncased and RoBERTa888We use the following
snapshot: https://huggingface.co/roberta-base models. These models have been
pretrained on text data pulled from the web, covering potentially a wide
variety of domains.

We fine-tune these models to solve CIViC Evidence. BERT is an encoder-only
transformer with $N=12$ transformer blocks with $e=768$, $h=768$, which uses
the GELU activation function ([5]). It was pretrained using masked-language-
modeling as described above. In addition, pretraining included an objective
called Next Sentence Prediction. This objective aims to train a model to
predict whether a given pair of sentences appears consecutively or not within
a text corpus, hoping to enhance the model’s understanding of context and
coherence in language ([5]).

The authors of ([24]) introduce RoBERTa, improving the original BERT model by
extending the training duration on a larger dataset, employing smaller batch
sizes. The architecture of both models, however, is nearly identical. Both
models have a maximum context length of $512$ tokens. Input strings which are
longer than 512 tokens are truncated.

#### BiomedBERT, BioLinkBERT & Biomed-RoBERTa pretrained on biomedical text

We compare the performance of BiomedBERT999We use the following snapshot:
https://huggingface.co/microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract
([10]), BioLinkBERT101010We use the following snapshot:
https://huggingface.co/michiyasunaga/BioLinkBERT-base ([42]), and Biomed-
RoBERTa111111We use the following snapshot:
https://huggingface.co/allenai/biomed_roberta_base ([11]) to their respected
mixed-domain variant.

Architecture-wise these models are identical to the mixed-domain versions
discussed above. However, they have been pretrained on texts from the
biomedical domain. In the case of Biomed-RoBERTa, pretraining is first
performed on general text data and then continued on biomedical text data.
This is a key difference to BiomedBERT and BioLinkBERT which have exclusively
been pretrained on biomedical text.

BiomedBERT and BioLinkBERT also come with domain-specific tokenizers which
allows them to represent words from the biomedical jargon as entire tokens,
whereas Biomed-RoBERTa uses the vanilla RoBERTa tokenizer. We conjecture that
domain-specific pretraining leads to a measurable performance gain over the
mixed-domain base models.

#### Biomed-RoBERTa-Long pretrained on long PubMed abstracts

One limitation of BERT-based models (including RoBERTa-based models) is their
limited context length of 512 tokens. We propose a new domain-specific model,
Biomed-RoBERTa-Long, which can handle sequences with a maximum length of 1024
tokens.

Starting from a pretrained Biomed-RoBERTa snapshot we first adjust the
embedding layer to handle 1024 tokens and reinitialize its weights to random
values. In the same manner as in ([2]), we duplicate the first 512 positional
encodings and concatenate the resulting two identical blocks to extend the
positional encodings to 1024 tokens. The rest of the model stays unmodified
compared to the last Biomed-RoBERTa snapshot.

We then perform continued domain-specific pretraining on a corpus of long
PubMed abstracts121212https://huggingface.co/datasets/pubmed. This corpus
consists of $\sim$800,000 abstracts which are comprised of at least 750 tokens
each. We conjecture that this model should lead to further increased
downstream performance compared to Biomed-RoBERTa. This approach is
conceptually similar to Longformer ([2]), which uses a windowed version of
self-attention in order to scale to even longer input lengths (up to 4096
tokens).

#### GPT-4 pretrained on general data combined with N-shot learning

We evaluate how well OpenAI’s GPT-4 ([29]) can perform on CIViC Evidence in an
N-shot learning setting. To this end, we leverage OpenAI’s
APIs131313https://platform.openai.com/docs/api-reference exposing GPT-4 with a
context length of 128,000 tokens141414gpt-4-0125-preview as listed here:
https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo. This approach
allows for an evaluation of In-Context Learning capabilities of state-of-the-
art LLMs as applied to CIViC Evidence. No domain-specific pretraining is used,
neither is the model fine-tuned for CIViC Evidence.

To limit the scope of this thesis, we only use standard prompting strategies
(i.e. a prompt aiming at obtaining a direct answer to our classification
problem from the model) and leave the evaluation of more sophisticated
prompting strategies such as Chain-of-Though-Prompting or Self-Questioning-
Prompting ([40]) to future work. The exact prompt we use is shown in Fig. 6.

###  4.3 Evaluation setup

We base model evaluation on both, intra evidence level F1 scores, and support-
weighted global F1 scores. For each class
$c\in\\{\text{A},\text{B},\text{C},\text{D},\text{E}\\}$, we have:

$\displaystyle F_{1,c}$
$\displaystyle=2\cdot\frac{\text{precision}_{c}\cdot\text{recall}_{c}}{\text{precision}_{c}+\text{recall}_{c}}$
(3) $\displaystyle\text{precision}_{c}$
$\displaystyle=\frac{\text{TP}_{c}}{\text{TP}_{c}+\text{FP}_{c}}$ (4)
$\displaystyle\text{recall}_{c}$
$\displaystyle=\frac{\text{TP}_{c}}{\text{TP}_{c}+\text{FN}_{c}}$ (5)

Here, TP, FP, FN are the numbers of true positives, false positives, and false
negatives, respectively. We follow the convention to define $F_{1,c}$ to be
zero, should both, recall and sensitivity be zero.

The support-weighted global F1 score is defined to be:

$\displaystyle F_{1}$
$\displaystyle=\sum_{c\in\\{\text{A},\text{B},\text{C},\text{D},\text{E}\\}}{w_{c}F_{1,c}}$
(6) $\displaystyle w_{c}$
$\displaystyle=\frac{\text{support}(c)}{\sum_{k\in\\{\text{A},\text{B},\text{C},\text{D},\text{E}\\}}{\text{support}(k)}}$
(7)

Here, $\text{support}(c)$ simply denotes the number of samples which are
tagged with evidence class $c$.

Classification thresholds (potentially different from $0.5$ ) per evidence
level are calibrated on the validation dataset to optimize intra-class F1
scores. Reported metrics are then calculated, post inference, on the test
dataset. We report averages over five random seeds to account for the
stochastic influence of the neural network training procedure (e.g. weight
initialization).

Figure 6: Prompt for GPT-4 used for few-shot prediction of CIViC evidence
levels

##  5 Results & Comparative study

###  5.1 Continued pretraining of Biomed-RoBERTa-Long

We initialize Biomed-RoBERTa-Long from the last Biomed-RoBERTa snapshot as
described in the previous section. Following the authors of ([2]), we continue
pretraining on long abstracts from PubMed for 3,000 gradient update steps
where each step accounts for processing $2^{18}$ tokens. We report the masked-
language-modeling151515Essentially a variant of cross-entropy which only takes
into account masked out tokens ([5]). loss on 3,000 randomly held-out
abstracts from our pretraining corpus before and after pretraining in Tab. 4:

Table 4: MLM-loss of Biomed-RoBERTa-Long & Biomed-RoBERTa before and after pretraining on long PubMed abstracts Model | $\mathcal{L}_{MLM}$
---|---
BioMed-RoBERTa | 1.032
BioMed-RoBERTa-Long | 1.908
+3K gradient updates | 0.994

The benchmark MLM-loss of 1.032 for Biomed-RoBERTa matches well with the loss
that the authors of Biomed-RoBERTa report in Tab. 1 of ([11]) for a test
dataset from the biomedical domain and their implementation of Biomed-RoBERTa
(they report an MLM-loss of 0.99).

We observe that after 3,000 gradient updates, Biomed-RoBERTa-Long slightly
outperforms Biomed-RoBERTa on our validation in terms of MLM-loss. Our exact
pretraining setup is shown in Tab. 8 of the appendix for reproducibility
purposes. We mostly use the settings suggested by the authors of
([2])161616https://colab.research.google.com/github/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb.

###  5.2 Fine-tuning results for CIViC Evidence

Table 5: Fine-tuning results for CIViC Evidence by model architecture | $F_{1,A}$ | $F_{1,B}$ | $F_{1,C}$ | $F_{1,D}$ | $F_{1,E}$ | $F_{1}$
---|---|---|---|---|---|---
BERT | 51.6 | 83.9 | 80.0 | 88.0 | - | 81.3
+BiomedBERT | 58.3 | 84.2 | 81.9 | 87.2 | - | 82.1
+BioLinkBERT | 61.8 | 84.1 | 81.2 | 87.9 | - | 82.2
RoBERTa | 48.4 | 82.9 | 80.1 | 88.4 | 2.0 | 80.9
+BioMed-RoBERTa | 55.3 | 84.3 | 81.8 | 87.8 | 2.4 | 82.2
+BioMed-RoBERTa-Long | 58.2 | 84.7 | 81.6 | 88.3 | 5.9 | 82.5
Logistic regression (tf-idf features) | 50.0 | 83.4 | 77.3 | 86.5 | - | 79.8

In this section, we evaluate the impact of domain-specific pretraining on
downstream performance in the context of CIViC Evidence. In addition, we will
elicit whether, ceteris paribus, there is a positive impact from increasing
model context width. For this end, we fine-tune all BERT- and RoBERTa-based
pretrained models on CIViC Evidence.

For all models we perform a hyperparameter search across three learning rates
and two batch sizes: For each combination of learning-rate and batch size, we
fine-tune the model starting from three different weight initializations for
20 epochs and early-stop at the best performing model on the validation set.
We then select the hyperparameter combination with lowest average validation
loss across three training runs.

As a next step we train five instances of the models using their optimal
hyperparameters. Finally, for every model class, we evaluate the five
resulting model runs on the test set.

It is worth mentioning that, for every model, we calibrate an optimal
probability threshold per evidence level based on the underlying precision-
recall curve. That is, for every evidence level, we choose the threshold which
maximizes the F1 score on the validation dataset.

We report here the average performance across five fitted models per model
class on the test dataset. The validation dataset evaluation results for the
hyperparameter grid search described above are reported in Tab. 9 of the
appendix. There, we also document the technical training setup for
reproducibility purposes in Tab. 8.

As shown in Tab. 5, our transformer-based models clearly outperform a simple
logistic regression model based on bigram tf-idf score features (see e.g.
([34]) for a brief introduction). Tab. 5 shows improvements in most metrics
for the models that were pretrained on biomedical text (or underwent continued
domain-specific pretraining in the case of Biomed-RoBERTa). Our own Biomed-
RoBERTa-Long exhibits further improved downstream performance compared to
Biomed-RoBERTa.

While all models perform reasonably on CIViC Evidence in general, performance
on the rare classes A and B is significantly worse. This is expected given the
lack of available training data for these two classes. Furthermore, it is
important to mention that model robustness with respect to stochastic
influence varies among architectures, as can be seen in Fig. 10 in the
appendix. In fact, when looking at median performance, BiomedBERT is the best
performing model. In this case, average performance is greatly reduced by one
outlier.

###  5.3 Comparison with GPT-4

Given our constraint resources, evaluation of GPT-4 on our entire test dataset
is unfeasible. Hence, we use a slightly different evaluation setup:

From the CIViC Evidence test dataset we carve out four items per evidence
level as an additional test dataset and have OpenAI’s GPT-4 predict the
matching levels in a few-shot setting. For the prompt (Fig. 6) discussed in
the previous section we obtain results as presented in Tab. 6.

We report averages across three API-calls per few-shot setting. Here, for
every API-call we sample $N$ example abstracts from the CIViC Evidence
training dataset.

The evaluation dataset remains constant and consists of the aforementioned
four items per evidence level. It is important to mention that for this
analysis we also evaluate the fine-tuned models on this reduced test dataset,
hence the results do not coincide with Tab. 5.

Table 6: Few-shot learning results of GPT-4 on CIViC Evidence | $F_{1,A}$ | $F_{1,B}$ | $F_{1,C}$ | $F_{1,D}$ | $F_{1,E}$ | $F_{1}$
---|---|---|---|---|---|---
BERT | 83.7 | 77.3 | 83.6 | 95.0 | - | 70.7
+BiomedBERT | 90.5 | 87.2 | 68.3 | 86.5 | - | 71.3
+BioLinkBERT | 94.9 | 84.9 | 64.0 | 77.1 | - | 68.7
RoBERTa | 81.6 | 80.2 | 71.7 | 91.1 | 13.3 | 70.9
+BioMed-RoBERTa | 87.1 | 81.8 | 79.8 | 83.6 | 5.7 | 70.9
+BioMed-RoBERTa-Long | 90.5 | 83.4 | 68.3 | 81.8 | 18.3 | 71.8
Logistic regression (tf-idf features) | 50.0 | 83.4 | 77.3 | 86.5 | - | 67.7
GPT-4 (zero shot) | 61.8 | 66.6 | 53.3 | 88.4 | 24.4 | 61.5
+1 example per class | 55.6 | 55.6 | 56.0 | 88.6 | - | 53.4
+2 examples per class | 61.9 | 64.8 | 51.5 | 74.2 | 16.7 | 56.7
+3 examples per class | 86.1 | 38.3 | 63.4 | 72.1 | 50.0 | 57.8
+4 examples per class | 90.5 | 59.8 | 69.4 | 59.3 | 61.1 | 66.1
+5 examples per class | 79.4 | 39.4 | 52.5 | 62.7 | 45.5 | 53.0
+10 examples per class | 76.2 | 48.2 | 22.2 | 36.1 | 45.5 | 45.8

It becomes clear that CIViC Evidence presents a significant challenge for
GPT-4, at least without further prompt engineering. While performance gets
close to the logistic regression based on tf-idf scores, it remains clearly
poorer than the performance of the fine-tuned models.

We observe that F1 score performance drops when including one example per
class as compared to zero-shot performance. It then increases super-linearly,
peeking at $N_{\text{shots}}=4$. Then performance starts to degrade again.
Interestingly, GPT-4 can significantly outperform our models on evidence level
E. Given the lack of proper explainability approaches for this closed weights
model, this phenomenon is not further investigated.

##  6 Error analysis & Interpretability

In this section, we aim to perform a more systematic error analysis. If it
underscores salient points, we supplement our exposition by findings from an
explainability analysis based on integrated gradients.

We begin by quantifying the difficulty of test dataset evidence items. To this
end, Fig. 7 ranks evidence items by the number of models171717we take the
median run for each model class which label them correctly. Given the fact
that an evidence item can be tagged with multiple evidence classes, we define
correctly classified items to be those items, which have all evidence classes
predicted correctly. We see that around half of the test dataset samples are
labeled correctly by all models. Similarly, we also observe that a significant
fraction of samples is misclassified for all models we consider.

Figure 7: Number of correctly annotating models by test dataset evidence items

#### Reasons for misclassification: Insufficient information in the abstract

Investigating these hard abstracts, we find that some of them simply do not
contain enough information to classify them correctly. In these cases, the
human annotator most likely read the full-text associated with the abstract
and then performed the classification.

To clarify this phenomenon, it is illustrative to look at an example:

> Larotrectinib can be effective in patients with ETV6-NTRK3–positive B-cell
> lymphoblastic leukemia, inducing prolonged molecular remission. Single-agent
> tyrosine kinase inhibitor treatment could be a valuable treatment option in
> subgroups of kinase fusion–positive ALL patients.
> — ([35])

This abstract is annotated with the two evidence levels C and D. When looking
at the description (another column in the CIViC evidence table) for this
evidence item, we find:

> Conventional cytogenetics, FISH, and an Abbott break-apart probe of a
> peripheral blood sample collected at diagnosis of a 6-year-old boy with
> B-cell ALL and CNS infiltration (National Cancer Institute high risk)
> revealed an unbalance translocation of the ETV6 locus. RNA sequencing of
> detected a cytogenetically cryptic ETV6-NTRK3 fusion. Larotrectinib therapy
> achieved remission of the patient’s two separate relapses. Patient derived
> xenografts injected into mice and treated with Larotrectinib demonstrated
> significant reductions in spleen volume, blasts in spleen, and blast in bone
> marrow (p <.0001) compared to controls 25 days after injection.
> Larotrectinib treatment was effective regardless of the day of initiation
> post xenograft injection (1 day or 8 days after).’
> — ([9])

Here, the mice experiment is the decisive factor for tagging this item with
evidence label D. However, this information is nowhere to be found in the
abstract.

In contrast to this example, we can examine the abstract of another evidence
item, which is correctly labeled with evidence class D by all model classes:

> We investigated the efficacy of the Wee1 inhibitor MK-1775 in combination
> with radiation for the treatment of pediatric high-grade gliomas (HGGs)
> […]
> Finally, combined MK-1775 and radiation conferred greater survival benefit
> to mice bearing engrafted, orthotopic HGG and DIPG tumors, compared with
> treatment with radiation alone (BRAF(V600E) model P = .0061 and DIPG
> brainstem model P = .0163).Our results highlight MK-1775 as a promising new
> therapeutic agent for use in combination with radiation for the treatment of
> pediatric HGGs, including DIPG.
> — ([28])

In this example, the findings from mice experiments are explicitly mentioned.

#### Reasons for misclassification: Presence of adversarial tokens

Tab. 10 and Tab. 11 (see Appendix) show the tokens associated with the highest
attribution impact per model and evidence class for a subset of four evidence
items per evidence level sampled from the test dataset.

While we do observe strong idiosyncratic influence resulting from the 20
selected examples (particularly for class C, tokens with high attribution
mostly stem from the phrase "Von Hippel-Lindau disease (VHL)"), we also see
that classes A and B are closely linked to vocabulary describing clinical
trials on patients (which their respective evidence level definitions
suggest).

Fig. 8 shows an abstract which is correctly tagged with evidence level A. We
observe that the model puts high emphasis on the tokens phase 3 trial, in line
with expectations.

Figure 8: BioLinkBERT token-level attributions for class A on ([26]) \-
correct label: A \- predicted label: A

In contrast, top tokens for class D include the word mice (as in mouse
experiment), and cell as in cell models. These words are in good agreement
with the defining characteristics of evidence level D.

However, we observe that e.g. the word mice also plays some role for the
(mostly incorrectly) classified samples of evidence level E (e.g. looking at
BERT). We conjecture that models may mistake samples of evidence level E for
samples belonging to evidence level D (which has mouse experiments as one
possible defining characteristic). Hence, one possible conclusion is that the
presence of the word mice might actually be seen as a kind of adversarial
example for evidence items which do not, in fact, belong to class D. Phenomena
like this, paired with lack of training data may partially explain poor model
performance for evidence label E.

#### Degrees of similarity among model misclassifications

Apart from examples which are unanimously classified correctly or incorrectly,
models tend to make different errors, whereby a significant overlap exists. We
attempt to quantify this overlap in Fig. 9.

We see that particularly the models pretrained on mixed-domain text exhibit
high degrees of overlap regarding misclassified examples. Biomed-RoBERTa and
Biomed-RoBERTa-Long also share a relatively large amount of errors, which is
expected, given that the models have many similarities.

On the other hand, degrees of overlap reduce when moving from the base model
to the domain-specifically pretrained models.

Figure 9: Left: Percentage of shared erroneously classified test evidence by
model class. Right: Test set evidence items by model class - light blue:
Wrongly classified, dark blue: Correctly classified

#### Domain-specific pretraining and tokenizers – An example

It seems reasonable to assume that descriptions of randomized clinical trials
are strongly linked to class A and may even be a differentiator of evidence
level A and B, because they present a particularly rigorous way of clinical
evidence testing ([13]).

Interestingly, the BiomedBERT tokenizer is able to recognize the word
randomized as a single token, presumably because it occurs in sufficiently
many abstracts from its pretraining corpus PubMed.

The standard BERT-tokenizer on the other hand does not model the word
randomized as one token.

Fig. 12 (see Appendix) shows BiomedBERT to put high emphasis on randomized
when compared to attributions made by BERT (Fig. 11, see Appendix). While BERT
wrongly classifies this example, BiomedBERT labels this item correctly as
belonging to evidence level A. Examples like this, underline the potential
benefits of domain-specific pretraining and domain-specific tokenizers for
downstream classification tasks.

##  7 Discussion

We close our exposition by discussing alternatives to our approach as well as
weaknesses and limitations. Where relevant, we include pointers to possible
future work.

#### The choice of classification problem: Multi-class vs. Multi-label

We choose to formulate CIViC Evidence as a multi-label classification problem.
That is, evidence items can be tagged with potentially all of the five
evidence classes.

An alternative would be to include relevant metadata into the abstract to
achieve a unique mapping of abstract, molecular profile, and treatment
approaches to evidence level. However, as illustrated in section 4, this kind
of metadata contains a lot of information which models can use to infer the
evidence level:

In fact, if we train a simple logistic regression model on only this metadata
(excluding the abstract itself) we will achieve weighted F1 scores of around
85%.

Including this kind of metadata becomes even more problematic when we think of
real world model applications. In a realistic setting, we would use our
machine learning models to label evidence data found in the wild. This data is
not likely to be accessible in structured formats, which already exhibit
annotations for molecular profile and treatment approaches. More likely,
applications will try to label abstracts, which will in fact belong to
multiple evidence levels at the same time, directly. This is why we deem the
multi-label approach more suitable.

#### Adding contextual information from article full-text

While inclusion of structured metadata would likely distort models from their
intended applications, we do think that inclusion of parts of the underlying
full-texts is thinkable and likely to improve model performance.

As discussed in the previous section, some abstracts simply do not contain the
information necessary to classify them correctly. This is why we think that
full-texts will eventually be necessary to push model performance beyond
certain boundaries.

However, this is not without its challenges, ranging from more involved data
retrieval processes to demand for drastically larger context sizes of language
models. This is where classical fine-tuning may reach limits (at least for
users and institutions with sharply constrained computing power) and few-shot
learning of models with extremely large context widths like GPT-4-128K or
Google’s Gemini 1.5181818https://blog.google/technology/ai/google-gemini-next-
generation-model-february-2024/ (context width of up to 1,000,000 tokens) may
be the the only feasible approach.

#### Sophisticated prompting approaches & fine-tuning of larger language
models

As shown in the result section, the few-shot performance of GPT-4 on CIViC
Evidence falls short of the performance of fine-tuned models. To mitigate
this, future work may consider Self-Questioning-Prompting as in ([40]).

A first step towards more elaborate prompts might be to ask the model to
explicitly provide reasoning for its classification decision rooted in the
CIViC evidence level guidelines outlined in Tab. 2.

In addition, future work may also leverage OpenAI’s fine-tuning
API191919https://platform.openai.com/docs/guides/fine-tuning to provide a
fairer comparison with BERT- and RoBERTa-based models.

A further natural candidate model fine-tuning could be Stanford’s BiomedLM
([3]) which is based on GPT-2. We omit this model in this exploration due to
our constrained computational resources.

#### The performance of Biomed-RoBERTa-Long on long abstracts

We have seen that Biomed-RoBERTa-Long is the best performing model on our test
dataset (Tab. 5). The motivation behind introducing this model as a variant of
Biomed-RoBERTa with doubled context length, is the fact that a significant
portion of the abstracts in CIViC Evidence exceeds the maximum context length
of Biomed-RoBERTa (512 tokens).

We would expect Biomed-RoBERTa-Long to also outperform Biomed-RoBERTa on the
subset of test items which are longer than 512 tokens. Evaluation results for
this subset of the test dataset are shown in Tab. 7.

Table 7: Fine-tuning results on CIViC Evidence for Biomed-RoBERTa and Biomed-RoBERTa-Long - long test abstracts only | $F_{1,A}$ | $F_{1,B}$ | $F_{1,C}$ | $F_{1,D}$ | $F_{1,E}$ | $F_{1}$
---|---|---|---|---|---|---
BioMed-RoBERTa | 57.2 | 82.4 | 31.9 | 66.7 | - | 67.6
+BioMed-RoBERTa-Long | 57.6 | 82.3 | 30.1 | 66.7 | - | 67.3

Contrary to our expectation, we do not observe a performance increase for
Biomed-RoBERTa-Long over Biomed-RoBERTa on long CIViC Evidence
abstracts202020Abstracts longer than 512 tokens according to the BiomedBERT
tokenizer. At this point this observation is not understood and provides an
opportunity for further analysis. Moreover, it would be interesting to
determine whether prolonged pretraining on long abstracts (e.g. training for
65,000 gradient updates as in ([2])) will mitigate this phenomenon.

In addition, it is important to note that this observation does not contradict
our conjecture that increased context-width should lead to improved downstream
performance: Given that the test dataset of long abstracts is relatively small
(20 items), results shown in Tab. 7 may lack statistical robustness. In turn,
this means that most long abstracts remain in the train dataset. This
effectively implies an increase in available training data for Biomed-RoBERTa-
Long, because it is possible that long training abstracts contain information
after the cutoff of 512 tokens which only Biomed-RoBERTa-Long can leverage to
learn better classification decisions. This may explain the performance
increase we observe for Biomed-RoBERTa-Long on the entire test dataset. A more
systematic investigation of this thematic complex, may be of value.

##  8 Conclusion

In this thesis we investigate the impact of (continued) domain-specific
pretraining on downstream performance for a sequence classification task from
the realm of biomedical NLP, CIViC Evidence. Our investigation is based on two
popular encoder-only transformer models: BERT and RoBERTa.

We show that for CIViC Evidence the benefit from domain-specific pretraining
can amount to up to 1% improvement on weighted F1 score. In addition, we show
that without further prompt engineering, even very large LLMs such as GPT-4
cannot reach equal levels of performance on CIViC Evidence in a few-shot
setting.

In order to process sequences longer than 512 tokens, we introduce Biomed-
RoBERTa-Long as a variant of Biomed-RoBERTa which can handle 1024 input
tokens. We show that this adjusted variant performs slightly better in terms
of F1 score on our test dataset.

One potential opportunity for future work is to continue pretraining for more
gradient updates (e.g. 65,000 as in ([2])) and see if pretraining and
downstream performance can be improved further.

Our analysis shows that model performance is significantly worse on evidence
levels A and E, probably owing to lack of training data and certain keywords
which are indicative of other classes but tend to occur in classes A and E, as
well. Future work could explore the use of up-sampling and data augmentation
techniques to improve model performance for these classes.

Given our constraint computational resources we have not explored the option
of fine-tuning even larger domain-specific models.

Lastly, more sophisticated prompting techniques such as Chain-of-Though-
Prompting, Self-Questioning-Prompting ([40]) or even fine-tuning. may be of
interest for further exploring the capabilities of GPT-4 on CIViC Evidence.

The downstream applications of these models include auto-labeling additional
medical papers on molecular tumor research, contributing to the construction
of extensive knowledge bases that medical practitioners can leverage for
informed decision-making. Further research can explore the option of
incorporating CIViC Evidence classifiers as one component of elaborate
information systems, which aim at benefitting target patient care in clinical
practice.

## Acknowledgements

We would like to thank Ulf Leser and Alan Akbik for their useful feedback and
input.

## References

  * [1] David Ashley et al. “Accepting risk in the acceleration of drug development for rare cancers” In _Lancet Oncol._ 16.4 Elsevier BV, 2015
  * [2] Iz Beltagy, Matthew E. Peters and Arman Cohan “Longformer: The Long-Document Transformer”, 2020 arXiv:2004.05150 [cs.CL]
  * [3] Elliot Bolton et al. “Stanford CRFM Introduces PubMedGPT 2.7B” Available at https://hai.stanford.edu/news/stanford-crfm-introduces-pubmedgpt-27b. Last checked: 2023-06-23, 2022 URL: https://hai.stanford.edu/news/stanford-crfm-introduces-pubmedgpt-27b
  * [4] Tom B. Brown et al. “Language Models are Few-Shot Learners”, 2020 arXiv:2005.14165 [cs.CL]
  * [5] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, 2019 arXiv:1810.04805 [cs.CL]
  * [6] Jay DeYoung et al. “Evidence Inference 2.0: More Data, Better Models”, 2020 arXiv:2005.04177 [cs.CL]
  * [7] Alexey Dosovitskiy et al. “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”, 2021 arXiv:2010.11929 [cs.CV]
  * [8] Carol Friedman, Thomas C. Rindflesch and Milton Corn “Natural language processing: State of the art and prospects for significant progress, a workshop sponsored by the National Library of Medicine” In _Journal of Biomedical Informatics_ 46.5, 2013
  * [9] Malachi Griffith et al. “CIViC is a community knowledgebase for expert crowdsourcing the clinical interpretation of variants in cancer” In _Nature Genetics_ 49.2, 2017
  * [10] Yu Gu et al. “Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing” In _ACM Transactions on Computing for Healthcare_ 3.1 Association for Computing Machinery (ACM), 2021
  * [11] Suchin Gururangan et al. “Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks”, 2020 arXiv:2004.10964 [cs.CL]
  * [12] Ryuji Hamamoto et al. “Introducing AI to the molecular tumor board: one direction toward the establishment of precision medicine using large-scale cancer clinical and biological information.” In _Exp Hematol Oncol_ 11.1, 2022
  * [13] Eduardo Hariton and Joseph J Locascio “Randomised controlled trials - the gold standard for effectiveness research: Study design: randomised controlled trials.” In _BJOG_ 125.13, 2018
  * [14] Florian Horm “Biochemie des Menschen. Das Lehrbuch für das Medizinstudium” Thieme, 2020
  * [15] Nida Iqbal and Naveed Iqbal “Imatinib: a breakthrough of targeted therapy in cancer.” In _Chemother Res Pract_ 2014, 2014
  * [16] Halil Kilicoglu, Graciela Rosemblat, Marcelo Fiszman and Dongwook Shin “Broad-coverage biomedical relation extraction with SemRep” In _BMC Bioinformatics_ 21.1, 2020
  * [17] Su Nam Kim, David Martinez, Lawrence Cavedon and Lars Yencken In _BMC Bioinformatics_ 12 Suppl 2, 2011
  * [18] Eric Lehman, Jay DeYoung, Regina Barzilay and Byron C. Wallace “Inferring Which Medical Treatments Work from Reports of Clinical Trials” In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ Minneapolis, Minnesota: Association for Computational Linguistics, 2019
  * [19] Jonas Leichsenring et al. “Variant classification in precision oncology” In _International Journal of Cancer_ 145.11, 2019
  * [20] Jake Lever et al. “Text-mining clinically relevant cancer biomarkers for curation into the CIViC database” In _Genome Medicine_ 11.1, 2019, pp. 78
  * [21] Yikuan Li et al. “Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences”, 2022 arXiv:2201.11838 [cs.CL]
  * [22] Tianyang Lin, Yuxin Wang, Xiangyang Liu and Xipeng Qiu “A Survey of Transformers”, 2021 arXiv:2106.04554 [cs.LG]
  * [23] Tianyang Lin, Yuxin Wang, Xiangyang Liu and Xipeng Qiu “A survey of transformers” In _AI Open_ 3, 2022
  * [24] Yinhan Liu et al. “RoBERTa: A Robustly Optimized BERT Pretraining Approach”, 2019 arXiv:1907.11692 [cs.CL]
  * [25] Claudio Luchini, Rita T. Lawlor, Michele Milella and Aldo Scarpa “Molecular Tumor Boards in Clinical Practice” In _Trends in Cancer_ 6.9, 2020
  * [26] Pau Montesinos et al. “Ivosidenib and Azacitidine in IDH1-Mutated Acute Myeloid Leukemia.” In _N Engl J Med_ 386.16, 2022
  * [27] Mohammad Amin Morid et al. “Classification of clinically useful sentences in clinical evidence resources” In _Journal of Biomedical Informatics_ 60, 2016
  * [28] Sabine Mueller et al. “Targeting Wee1 for the treatment of pediatric high-grade gliomas.” In _Neuro Oncol_ 16.3, 2014
  * [29] OpenAI “GPT-4 Technical Report”, 2023 arXiv:2303.08774 [cs.CL]
  * [30] Solange Peters et al. “Alectinib versus Crizotinib in Untreated ALK-Positive Non-Small-Cell Lung Cancer.” In _N Engl J Med_ 377.9, 2017
  * [31] Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever “Improving language understanding by generative pre-training” In _OpenAI Blog_ , 2018
  * [32] Alec Radford et al. “Language Models are Unsupervised Multitask Learners” In _OpenAI Blog_ , 2019
  * [33] Yizhar Regev et al. “Rule-based extraction of experimental evidence in the biomedical domain: the KDD Cup 2002 (task 1)” In _Sigkdd Explorations_ 4, 2002
  * [34] Neil Christian R. Riego and Danny Bell Villarba “Utilization of Multinomial Naive Bayes Algorithm and Term Frequency Inverse Document Frequency (TF-IDF Vectorizer) in Checking the Credibility of News Tweet in the Philippines”, 2023 arXiv:2306.00018 [cs.CL]
  * [35] Denis M Schewe et al. “Larotrectinib in TRK fusion-positive pediatric B-cell acute lymphoblastic leukemia.” In _Blood Adv_ 3.22, 2019
  * [36] Seung Ho Shin, Ann M Bode and Zigang Dong “Addressing the challenges of applying precision oncology.” In _NPJ Precis Oncol_ 1.1, 2017
  * [37] Johannes Starlinger et al. “Variant information systems for precision oncology” In _BMC Medical Informatics and Decision Making_ 18.1, 2018
  * [38] Mukund Sundararajan, Ankur Taly and Qiqi Yan “Axiomatic Attribution for Deep Networks” In _arXiv e-prints_ , 2017, pp. arXiv:1703.01365 arXiv:1703.01365 [cs.LG]
  * [39] Ashish Vaswani et al. “Attention Is All You Need”, 2017 arXiv:1706.03762 [cs.CL]
  * [40] Yuqing Wang, Yun Zhao and Linda Petzold “Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical Language Understanding”, 2023 arXiv:2304.05368 [cs.CL]
  * [41] Jenn-Yu Wu and Jin-Yuan Shih “Effectiveness of tyrosine kinase inhibitors on uncommon E709X epidermal growth factor receptor mutations in non-small-cell lung cancer.” In _Onco Targets Ther_ 9, 2016
  * [42] Michihiro Yasunaga, Jure Leskovec and Percy Liang “LinkBERT: Pretraining Language Models with Document Links”, 2022 arXiv:2203.15827 [cs.CL]

## Appendix

Table 8: Pretraining and fine-tuning training setup and hyperparameters Hyperparameter |  Continued pretraining | Civic Evidence fine-tuning
---|---|---
Number of steps |  3000 | 20 epochs
Batch size per GPU |  8 | 8/16
Gradient accumulation steps |  16 | 0
Number of GPUs |  2 | 2
GPU type |  2x Nvidia A100 80GB | 2x Nvidia A100 80GB
Effective batch size |  256 | 16/32
Learning rate |  3e-4 | Unweighted Loss
|  | 1e-6/3e-6/6e-6
Learning rate optimizer |  Adam | Adam
Adam epsilon |  1e-6 | 1e-6
Adam beta weights |  0.9, 0.999 | 0.9, 0.999
Learning rate scheduler |  Linear Warmup with linear decay to zero | constant
Warmup steps |  500 | 0
Weight decay |  None | None
Max. gradient norm for gradient clipping |  5.0 | None
Table 9: Hyperparameter search results - best performing combinations are highlighted in bold. Batchsize | 16 | 32
---|---|---
Learning Rate | $1e^{-6}$ | $3e^{-6}$ | $6e^{-6}$ | $1e^{-6}$ | $3e^{-6}$ | $6e^{-6}$
BERT | 0.243 | 0.224 | 0.221 | 0.273 | 0.228 | 0.231
BiomedBERT | 0.208 | 0.204 | 0.206 | 0.223 | 0.205 | 0.210
BioLinkBERT | 0.222 | 0.216 | 0.214 | 0.236 | 0.217 | 0.218
RoBERTa | 0.228 | 0.226 | 0.228 | 0.238 | 0.227 | 0.223
BioMed-RoBERTa | 0.216 | 0.213 | 0.211 | 0.225 | 0.213 | 0.215
BioMed-RoBERTa-Long | 0.215 | 0.213 | 0.213 | 0.227 | 0.216 | 0.214
Figure 10: F1 score distribution across five fitted instances per model class
Figure 11: BERT token-level attributions for class A on ([30]) \- correct
label: A, B \- predicted label: B Figure 12: BiomedBERT token-level
attributions for class A on ([30]) \- correct label: AB \- predicted label: AB

Table 10: Top ten tokens with highest explainability attribution by evidence class for the BERT-based models | BERT | BiomedBERT | BioLinkBERT
---|---|---|---
A |  | 1\. patients
---
2\. progression
3\. dose
4\. confidence
5\. survival
| 6\. median
---
7\. daily
8\. clinical
9\. months
10\. dine
| 1\. primary
---
2\. patients
3\. tidline
4\. aza
5\. ci
| 6\. b
---
7\. 45
8\. randomly
9\. assessed
10\. consistent
| 1\. patients
---
2\. trial
3\. with
4\. or
5\. phase
| 6\. assigned
---
7\. dose
8\. treatment
9\. response
10\. random
B |  | 1\. the
---
2\. survival
3\. was
4\. progression
5\. confidence
| 6\. hort
---
7\. dose
8\. is
9\. median
10\. associated
| 1\. patients
---
2\. survival
3\. the
4\. (
5\. %
| 6\. )
---
7\. associated
8\. braf
9\. trial
10\. months
| 1\. patients
---
2\. %
3\. survival
4\. p
5\. tumor
| 6\. median
---
7\. (
8\. dlbcl
9\. 0
10\. progression
C |  | 1\. patients
---
2\. hl
3\. disease
4\. with
5\. mutation
| 6\. patient
---
7\. a
8\. ma
9\. who
10\. fusion
| 1\. vhl
---
2\. two
3\. ecs
4\. report
5\. cancers
| 6\. for
---
7\. association
8\. fusions
9\. boy
10\. reported
| 1\. vhl
---
2\. patients
3\. patient
4\. mutations
5\. mutation
| 6\. disease
---
7\. cases
8\. 2
9\. gene
10\. alk
D |  | 1\. cells
---
2\. cell
3\. mice
4\. we
5\. confidence
| 6\. that
---
7\. therapeutic
8\. of
9\. lines
10\. mutations
| 1\. mice
---
2\. r3
3\. models
4\. fgf
5\. signaling
| 6\. mutant
---
7\. cell
8\. lines
9\. raf
10\. kras
| 1\. .
---
2\. of
3\. we
4\. )
5\. that
| 6\. cells
---
7\. in
8\. is
9\. the
10\. to
E |  | 1\. mice
---
2\. confidence
3\. d
4\. progression
5\. lines
| 6\. dine
---
7\. daily
8\. systemic
9\. kit
10\. mg
| 1\. models
---
2\. consistent
3\. kit
4\. underpinning
5\. acute
| 6\. cri
---
7\. lines
8\. ltd
9\. evaluated
10\. dab
| 1\. three
---
2\. associated
3\. never
4\. smoker
5\. ecs
| 6\. vhl
---
7\. p
8\. 1
9\. median
10\. iv
Table 11: Top ten tokens with highest explainability attribution by evidence class for the RoBERTa-based models | RoBERTa | BioMed-RoBERTa | BioMed-RoBERTa-Long
---|---|---|---
A |  | 1\. the
---
2\. survival
3\. (
4\. is
5\. )
| 6\. ib
---
7\. V
8\. 1
9\. randomly
10\. patients
| 1\. ib
---
2\. trial
3\. patients
4\. phase
5\. mg
| 6\. event
---
7\. treatment
8\. azine
9\. CL
10\. group
| 1\. trial
---
2\. of
3\. the
4\. The
5\. TA
| 6\. to
---
7\. patients
8\. events
9\. CL
10\. CNS
B |  | 1\. survival
---
2\. (
3\. patients
4\. outcome
5\. median
| 6\. HL
---
7\. cohort
8\. %
9\. tumor
10\. response
| 1\. survival
---
2\. L
3\. %
4\. a
5\. associated
| 6\. in
---
7\. lect
8\. P
9\. free
10\. endpoint
| 1\. patients
---
2\. K
3\. the
4\. survival
5\. ase
| 6\. Ther
---
7\. endpoint
8\. D
9\. oma
10\. OR
C |  | 1\. ,
---
2\. the
3\. )
4\. and
5\. –
| 6\. with
---
7\. a
8\. or
9\. an
10\. gene
| 1\. HL
---
2\. cases
3\. patients
4\. with
5\. V
| 6\. mutation
---
7\. oma
8\. BRA
9\. and
10\. usions
| 1\. HL
---
2\. Disease
3\. Hipp
4\. Lind
5\. V
| 6\. au
---
7\. Von
8\. mutations
9\. mutation
10\. unknown
D |  | 1\. is
---
2\. ib
3\. cells
4\. mice
5\. As
| 6\. that
---
7\. never
8\. free
9\. pl
10\. V
| 1\. cells
---
2\. mutations
3\. mice
4\. mutant
5\. CNS
| 6\. inhibitor
---
7\. signaling
8\. that
9\. ogenic
10\. inhibition
| 1\. mice
---
2\. in
3\. cell
4\. models
5\. a
| 6\. cular
---
7\. the
8\. and
9\. Ther
10\. mutations
E |  | 1\. the
---
2\. )
3\. V
4\. ib
5\. is
| 6\. 1
---
7\. (
8\. free
9\. CC
10\. ;
| 1\. V
---
2\. CNS
3\. d
4\. smoker
5\. event
| 6\. untreated
---
7\. children
8\. transcript
9\. hazard
10\. ations
| 1\. V
---
2\. the
3\. in
4\. TOR
5\. ase
| 6\. of
---
7\. that
8\. mutants
9\. imus
10\. survival