#  CSMeD: Bridging the Dataset Gap in Automated Citation Screening for
Systematic Literature Reviews

Wojciech Kusa1 , Oscar E. Mendoza2, Matthias Samwald3, Petr Knoth4, Allan
Hanbury1
1TU Wien 2University Milano-Bicocca 3Medical University of Vienna 4The Open
University
Corresponding author<EMAIL_ADDRESS>
###### Abstract

Systematic literature reviews (SLRs) play an essential role in summarising,
synthesising and validating scientific evidence. In recent years, there has
been a growing interest in using machine learning techniques to automate the
identification of relevant studies for SLRs. However, the lack of standardised
evaluation datasets makes comparing the performance of such automated
literature screening systems difficult. In this paper, we analyse the citation
screening evaluation datasets, revealing that many of the available datasets
are either too small, suffer from data leakage or have limited applicability
to systems treating automated literature screening as a classification task,
as opposed to, for example, a retrieval or question-answering task. To address
these challenges, we introduce CSMeD, a meta-dataset consolidating nine
publicly released collections, providing unified access to 325 SLRs from the
fields of medicine and computer science. CSMeD serves as a comprehensive
resource for training and evaluating the performance of automated citation
screening models. Additionally, we introduce CSMeD-ft, a new dataset designed
explicitly for evaluating the full text publication screening task. To
demonstrate the utility of CSMeD, we conduct experiments and establish
baselines on new datasets.

##  1 Introduction

_Systematic literature reviews_ (_SLRs_ , or meta-reviews) are a critical tool
in scientific research, used for synthesising and summarising evidence from
multiple studies. The SLR process involves several stages, including _citation
screening_ (_CS_ , or selection of primary studies) which is, in itself, a
time-consuming step [57, 10]. CS involves identifying studies relevant to the
SLR based on a set of, often complex, inclusion and exclusion criteria (e.g.,
the study must be examining the efficacy of Drug X on Condition Y).

In recent years, there has been an increasing interest in automating the SLR
process [78, 59, 55, 22, 2], with works often focusing on improving the CS
step by (a) using machine learning (ML) [43, 44], (b) natural language
processing (NLP) [27, 36, 77], and (c) information retrieval (IR) [68, 88]
techniques. Automated CS systems have the potential to significantly reduce
the time and resources required for this critical step, thereby speeding up
the SLRs production [73].

The development of standards provides invaluable resources for evaluating and
comparing different models. Benchmarks, such as BEIR [72], GLUE [83] or BLURB
[20] have shown improvements in reproducibility and progress tracking of
machine learning models in various domains. Unfortunately, in the context of
SLR automation, the absence of standard benchmarks and evaluation
methodologies still hampers progress and inhibits the development of reliable
and effective solutions.

With the fast-evolving landscape of machine learning, identifying state-of-
the-art performance has become especially challenging and inefficient in the
context of CS. The notorious proliferation of small custom CS datasets and
single-usage evaluation approaches further exacerbates this issue. We show
that current CS datasets exhibit several shortcomings that hinder their
applicability for comprehensive and standardised evaluations. These datasets
are poorly documented, with most lacking datasheets, clear licenses and terms
of use. In addition, the limited applicability of older datasets arises from
their small size and lack of crucial metadata, restricting their use to
classification tasks. Finally, data leakage and dataset overlap is another
issue, with some SLRs present in multiple collections.

To address these limitations, we present CSMeD (Citation Screening Meta-
Dataset), a comprehensive collection of CS datasets that can be used to
benchmark and evaluate automated screening systems. Our collection builds upon
nine existing datasets, and a new dataset for evaluating the full text
classification task, counting 325 SLRs from the fields of medicine and
computer science. Thanks to the data harmonisation, our new collection can
mitigate the issues of lack of canonical splits, limited applicability, and
dataset overlap. Our contributions are as follows:

  1. 1.

We create CSMeD, a meta-collection of nine datasets comprising 325 SLRs. CSMeD
is built upon BigBio [16] and can be used to evaluate and benchmark automated
CS systems. We also provide a comprehensive summary of existing citation
screening datasets.

  2. 2.

We extend CSMeD with additional metadata after analysing issues on the
existing collections and previous evaluation frameworks. Our extended dataset
can be used to evaluate CS as question answering or textual pairs
classification tasks.

  3. 3.

Using new metadata, we introduce CSMeD-ft, a new dataset for the task of full
text screening. To the best of our knowledge, this is the first dataset
designed explicitly on screening long documents in SLR. This dataset can be
used for the evaluation of the inference capabilities based on a very long
context (4,000+ words).

The remainder of the paper is structured as follows. In Section 2, we define
the task of citation screening for systematic literature reviews. Section 3
provides an overview of related work in SLR automation and available
benchmarks. Section 4 describes the CSMeD meta-dataset in detail, including
its creation, analysis and extension. In Section 5, we introduce the full text
screening dataset together with baseline results on this dataset, and in
Section 6, we discuss the implications of our work and potential extensions.

##  2 Task formulation

We start by introducing the task of citation screening for SLRs and presenting
the notation used for its formulation. An SLR is characterised by various
attributes, including the title, abstract, research question $\mathcal{RQ}$,
and eligibility criteria $\mathcal{C}$. We refer to all these attributes as
the SLR protocol. Eligibility criteria comprise a set of rules and conditions
that a document must meet for inclusion in the SLR. Given a large pool of
documents denoted as $\mathcal{D}$, the main goal of automated citation
screening is to assist researchers in identifying relevant publications for
inclusion in an SLR. Each document $d\in\mathcal{D}$ has attributes such as
its title, abstract, main content, authors, and publication year. The task of
CS for SLRs can be formally defined as follows:

######  Definition 2.1 (CS).

Given a set of documents $\mathcal{D}$ and a set of eligibility criteria
$\mathcal{C}$, the task of CS for SLR is to determine for each document
$d\in\mathcal{D}$ whether it satisfies the criteria $\mathcal{C}$. This
decision can be represented as a binary label $y_{d}\in\\{0,1\\}$, where
$y_{d}=1$ if document $d$ satisfies the criteria $\mathcal{C}$, and $y_{d}=0$
otherwise.

It is important to note that the manual CS is conducted in two steps, as shown
in Figure 1: title and abstract screening and full text screening. In the
first step, the relevance of each document is evaluated based on its title and
abstract, while in the second step, a more thorough assessment is performed by
examining the full text of the document.

#### Document retrieval

The initial step involves document retrieval, which aims to generate a set of
potentially relevant documents $\mathcal{D^{\prime}}\subseteq\mathcal{D}$
given $\mathcal{RQ}$. This step commonly involves querying bibliographic
databases with specific keywords and Boolean expressions. We can formulate
this step as a retrieval function $r$, such that
$r(\mathcal{RQ},\mathcal{C})=\mathcal{D^{\prime}}$. However, the retrieved set
$\mathcal{D^{\prime}}$ may contain a large number of false positives
(irrelevant documents).

#### Binary classification for relevance prediction

Following document retrieval, the primary task is to assess the relevance of
each document in the set $\mathcal{D^{\prime}}$ concerning the eligibility
criteria $\mathcal{C}$. This is conducted in two stages, differing in which
attributes of documents are considered (titles and abstracts vs. full texts).
We treat this as a binary classification problem, where each document
$d\in\mathcal{D^{\prime}}$ is assigned a binary label $y_{d}\in\\{0,1\\}$ to
indicate its relevance ($y_{d}=1$) or irrelevance ($y_{d}=0$) to the SLR per
the criteria $\mathcal{C}$.

#### Question answering for relevance

An alternative formulation of the citation screening task is to frame it as a
question-answering problem. In this approach, we transform the eligibility
criteria $\mathcal{C}$ into a set of questions
$\mathcal{Q}=\\{q_{1},\cdots,q_{|C|}\\}$, where each question $q_{k}$
corresponds to a specific criterion in $\mathcal{C}$. For each document
$d\in\mathcal{D^{\prime}}$, we obtain a set of predicted answers
$\hat{A}^{d}=\\{\hat{a}_{k}^{d}|\text{meets}(q_{k},\hat{a}_{k}^{d})\\}$, where
$\text{meets}(q_{k},\hat{a}_{k}^{d})$ denotes that the document $d$ should
meet the criterion expressed by the question $q_{k}$. The final relevance
label $\hat{y}_{d}$ of a document $d$ can be determined by aggregating the
predicted answers $\hat{A}^{d}$ using a logical combination function, such as
the logical AND operation.

This question-answering formulation offers a more fine-grained assessment of a
document’s relevance concerning various aspects of the eligibility criteria
$\mathcal{C}$. Other similar formulations of the CS task include document
ranking or natural language inference (NLI).

Figure 1: Illustration of the citation screening process, separated into two
tasks (1) title and abstract screening and (2) full text screening. Tasks are
represented as a specific example of question-answering when a single question
asks for a fullfilment of all eligibility criteria $\mathcal{C}$ at once.

##  3 Related work

We first motivate the work by providing context on the importance of SLRs and
then focus on reviewing citation screening automation methods. Finally, we
outline limitations of existing CS datasets.

###  3.1 Systematic literature reviews

SLRs are particularly important in the medical domain [29]. The Cochrane
Collaboration,111https://www.cochrane.org the largest organisation responsible
for creating SLRs in medicine, has created the foundations of Evidence-Based
Medicine [24]. There are more than 220,000 records published between 2000 and
2022 tagged as SLRs in PubMed222https://pubmed.ncbi.nlm.nih.gov/ meaning that,
on average, there were 10,000 SLRs published per year.

As SLRs focus on reproducibility and finding all relevant evidence about a
given topic, the traditional framework involves tasks mainly done manually. It
includes steps like defining the search strategy (designing complex Boolean
queries) or the screening of every document by at least two reviewers,
resulting in an average production time of more than one year [75].

Previous research focused on evaluating automation capabilities for several
steps of the traditional framework, such as citation screening (CS) [11, 27,
73], search query (re-)formulation [68, 70], data extraction [58], SLR
summarisation [84] or generation of reviews based on the title [92].

###  3.2 Citation screening automation

As described in Section 2, CS can be seen as a binary classification problem.
However, due to a large number of retrieved studies, the significant class
imbalance, and the need to identify _nearly all_ relevant documents, this task
is inherently complex. Screening automation is a general term for various
approaches aimed at reducing workload during the CS stage [59]. These
approaches can be classified as either screening reduction, which involves
using classification or ranking algorithms to automatically exclude non-
relevant publications or screening prioritisation, which focuses on ranking
relevant records earlier in the screening process [55]. Automated screening
systems leverage techniques from NLP, ML, IR, and statistics, all with the
common objective of reducing manual screening time. The disparity in
strategies from different fields hinders direct comparison and benchmarking.
Next, we discuss some points of disagreement.

NLP approaches typically focus on the level of individual SLRs, treating each
review as an independent dataset; whereas IR approaches would consider a set
of reviews as a collection, the topics of the reviews analogous to queries,
and report aggregated evaluation. Moreover, different publications across
various venues adopt diverse evaluation measures, making even more complex the
assessment of similar, if not identical, tasks. Evaluation of automatic
approaches traditionally relies on binary relevance ratings, very often
obtained from the title and abstract screening [59, 32]. When the screening
problem is treated as a ranking task, such as screening prioritisation or
stopping prediction; the performance is measured in terms of rank-based
metrics and metrics at a fixed cut-off, such as $nDCG@n$, $Precision@n$, and
_last relevant found_ [69, 28]. On the other hand, when the screening problem
is treated as a classification task, the performance in this case is measured
based on the confusion matrix and the notions of Precision and Recall are
commonly used [41, 59]. One challenge arising from these two distinct
approaches is the difficulty in going beyond simple effectiveness measures and
comparing the real-world savings for users. Further details on datasets and
evaluation approaches can be found in a comprehensive review in the Appendix
A.

###  3.3 Limitations of existing datasets

Through our review (see Appendix A), we identified twelve CS datasets reported
in former research papers, of which ten have been publicly released. During
this analysis, we identified several shortcomings; some are also prevalent in
other machine learning problems. Below, we summarise our findings,
highlighting the key issues.

#### Poor documentation

One major concern with previous datasets is the lack of sufficient
documentation. None of the datasets we examined implement a datasheet [18],
which is an essential tool for ensuring transparency and reproducibility.
Additionally, seven datasets do not provide clear licenses or terms of use. An
inconsistency was also found for one of the datasets [69] in terms of the
number of the available content: the paper states 93 SLRs, but we found a list
of 176 reviews on the corresponding GitHub repository.

#### Limited applicability

Previous datasets are often small and lack crucial metadata like SLR research
question or eligibility criteria, limiting their use to only evaluation of
classification tasks. Older datasets typically provide only the title of the
review, which limits their applicability for the comprehensive evaluation of
neural language understanding models. The most widely used dataset to date
[11] was released in 2006. As ML and NLP techniques continue to advance
rapidly, it is crucial to have up-to-date datasets that reflect the
complexities and nuances of the current research landscape.

#### Lack of canonical splits

Another significant challenge of previous datasets is the absence of canonical
train-test splits. Depending on the field of research, practices may vary. As
discussed before, in the ML and NLP domains, the prevailing practice is to use
inter-review splits, where each review is treated as an individual dataset,
and a set of citations is selected for training and testing. Conversely, IR
publications often report intra-review splits, treating each review as a
“topic” or query, and averaging the results across multiple queries.

In this sense, only the three TAR333TAR stands for Technology-Assisted Reviews
and was a shared task organised at CLEF between 2017 and 2019 by Kanoulas et
al. [30, 31, 32]. datasets contain pre-defined canonical splits, yet only at
the intra-review level. For three other datasets [11, 80, 27], previous works
have demonstrated significant variability in model evaluation based on the
selection of cross-validation splits, particularly for the smallest datasets
that contain a limited number of relevant documents [77, 38]. The lack of
standardised splits, especially in collections with fewer SLRs, makes it
challenging to compare different approaches and hinders the fair evaluation of
models’ performance.

#### Dataset overlap

We also evaluated overlapping throughout the previous datasets and discovered
that at least 11 SLRs were present in multiple collections [69, 30, 31, 32].
Additionaly, the TAR 2019 dataset contains three SLRs that are present both in
its training and test splits, accounting for approximately 6% of the test
partition [32]. While this overlap is not a significant concern when
evaluating unsupervised methods like BM25 [67], it poses a potential threat to
conducting fair comparisons with large language models (LLMs). Machine
learning models, and especially LLMs, have the capability to memorise their
training data, making it critical to address dataset overlap to ensure
unbiased evaluations [25] (see Appendix E for a detailed analysis of the
overlapping in previous datasets).

#### Lack of common evaluation

Another notable deficiency among the previous datasets is the absence of a
common set of evaluation measures. Only the three TAR datasets provide scripts
for evaluating submissions. For example, the most widely used dataset by Cohen
et al. [11] was evaluated using several disparate evaluation measures such as
$WSS$ [11], $AUC$ or $Precision@r\%$. However, recent research has exposed
limitations and problems with both $WSS$ and $AUC$ as metrics for this task
[40].

#### Availability in biomedical benchmarks

Recent efforts have focused on creating larger collections of more diverse
datasets to evaluate the performance of biomedical NLP models. These efforts
include benchmarks like BLUE [63], HunFlair [90], BLURB [20], and BigBio [16],
which provide datasets and tasks for evaluating biomedical language
understanding and reasoning. Additionally, there are biomedical datasets
geared towards prompt-based learning and evaluation of few and zero-shot
classification, such as Super-NaturalInstructions [89] and BoX [61]. Out of
all benchmarks mentioned above, only BoX contains one CS dataset covering five
SLRs, however, this dataset is private. Coverage for other SLR tasks is also
limited.

To summarise, previous datasets exhibit certain drawbacks that limit their
suitability for comprehensive and standardised evaluation. While the TAR
2017-19 collections stand out as the only ones containing canonical splits and
a set of evaluation measures, some of their topics overlap with another
previous dataset [69], and we also identified data leakage in the newest TAR
2019 dataset. Consequently, we believe that developing a new collection is
necessary to address these issues and establish a robust foundation for
evaluation of CS and SLR automation.

##  4 The CSMeD meta-dataset

The recent advancements and paradigm shifts in NLP and ML; with the extensive
use of pre-trained models and transfer learning [45, 15], and the more recent
prompt-based learning [48, 9]; have significantly transformed the field and
enhanced the predictive capabilities of models across various tasks. Inspired
by the success of benchmark collections in the field of biomedical NLP, we
conducted a thorough review of available datasets and benchmarks to identify
the most representative datasets for the task of citation screening, finding
that this task is heavily underrepresented. The available datasets still
primarily cater to training supervised algorithms, lacking the scale and
granularity necessary to evaluate state-of-the-art models. To address these
limitations and provide a more comprehensive resource for training and
evaluating data-centric methods in SLR automation, we create CSMeD,
consolidating nine previously released collections of SLRs. We further extend
a subset of SLRs in CSMeD with additional metadata coming from the review
protocol.

Our data analysis methodology involved creating visualisations and summary
tables based on curated datasets. We analyse dataset statistics like available
data splits, licensing information, dataset and reviews size as well as
dataset overlap. This allows us to provide both a detailed view of individual
reviews and an overview of datasets containing multiple reviews (see Appendix
B for further details on visualisations).

###  4.1 Dataset creation

Currently, nine out of ten public CS datasets we identified have been included
in CSMeD, with the remaining one to be included. We provide a summary of the
datasets in Table 1, and further details can be found in Appendix A. In total,
CSMeD consists of 325 SLRs, making it the largest publicly available
collection in this domain and the only one providing access to the datasets
via a harmonised API.

Table 1: A list of source citation screening datasets included in the CSMeD.
The first four datasets contain non-Cochrane SLRs, whereas the other five are
based on Cochrane reviews. ‘Avg. ratio of included’ column presents ratio of
included publication from the title and abstract screening stage, ‘Avg. size’
refers to averaged across SLRs document count in the dataset. The ‘Additional
data’ column describes if the review contains metadata other than coming from
the citation list: (A): Search queries, (B): Review protocol containing review
title, abstract and search strategy, (C): Review updates consisting of changes
to included papers. ‘DTA’ stands for diagnostic test accuracy reviews. ‡ –
indicates a discrepancy in the number of reviews in the paper versus the
GitHub repository. † – indicates the total count of reviews from all nine
datasets before duplicates were removed.

| Source
---
# reviews | Domain |  | Avg.
---
size
| Avg. ratio
---
of included
| Additional
---
data
| Cochrane
---
reviews
[11] | 15 | Drug | 1,249 | 7.7% | — | —
[80] | 3 | Clinical | 3,456 | 7.9% | — | —
[27] | 5 | Mixed | 19,271 | 4.6% | — | —
[22] | 7 | Comp. Science | 340 | 11.7% | B | —
[69] | 93/176‡ | Clinical | 1,159 | 1.2% | A | $\checkmark$
[30] | 50 | DTA | 5,339 | 4.4% | B | $\checkmark$
[31] | 30 | DTA | 7,283 | 4.7% | B | $\checkmark$
[32] | 49 | Mixed | 2,659 | 8.9% | B | $\checkmark$
[2] | 25 | Clinical | 4,402 | 0.4% | C | $\checkmark$
Total | 360† |  | 3,471 | 4.4% |  |

To ensure interoperability and facilitate the ease of use, we designed data
loaders for the datasets in accordance with the BigBio text classification
schema [16]. This choice offers several advantages. BigBio has the largest
coverage of biomedical datasets and supports access to the datasets via API.
Moreover, it is compatible with popular libraries such as Hugging Face’s
datasets [46] and the EleutherAI Language Model Evaluation Harness [17],
thereby reducing the experimental costs.

Taking advantage of the lists of publications that most of the sources of
datasets share as PubMed IDs, we extend the BigBio data loaders to enable the
download of PubMed articles. Our harmonised data loaders selectively load the
PubMed articles that are a part of each dataset. The single exception is the
dataset by Hannousse and Yahiouche [22], which is the only publicly available
collection of non-medical SLRs. For this dataset, we extract the content using
the SemanticScholar API.444https://www.semanticscholar.org/product/api As a
result, CSMeD serves also as the first resource that gathers SLRs from diverse
domains.

###  4.2 Extending metadata

Our goal at this stage is not to create yet another gold standard dataset for
SLRs, but rather improve the quality of current data and provide insights into
promising avenues for future research. We begin by presenting the
possibilities of extending the subset of Cochrane SLRs to experiment with
screening beyond supervised classification.

We then categorise CSMeD datasets into two groups: (1) datasets containing
Cochrane medical SLRs and (2) datasets comprising other SLRs. This distinction
is made because from following the Cochrane protocol, more extensive
information on the review is provided. We use the additional data available on
reviews websites to extend CSMeD. Among the new information, we find the
eligibility criteria most valuable—the inclusion of eligibility criteria no
longer limits the data to the evaluation of supervised binary classification
but opens its application to question-answering or language inference tasks.

We carefully examine the subset of SLRs produced by Cochrane, aiming to
identify potential enhancements and extensions that would help mitigating the
existing limitations of previous datasets. Every Cochrane SLR first registers
and publishes the protocol containing the review title, abstract, search
strategy and the eligibility criteria. This information is all that human
experts need to produce the final review, i.e., they first find the relevant
studies and then conduct the meta-analysis of their results. As described in
Section 2, the screening process can be also modelled as a question-answering,
where every publication is compared against the eligibility criteria in order
to make the decision about the inclusion,555In the current approach, we
consider only binary relevance (included versus excluded). However, in the
real life, more categories can be defined reviewers (e.g. a study can be
assigned as a background publication or meta-analysis). similar to the
clinical decision support task of matching clinical trials to patients [65,
66].

To expand CSMeD, we searched the Cochrane
Library666https://www.cochranelibrary.com/cdsr/reviews for all SLRs from the
meta-dataset based on the Cochrane review ID and take their latest open-access
version. We extract available information about the review: review title and
abstract, eligibility criteria, search strategy and references. Cochrane
reports a list of included and excluded publications at the full text
screening stage (this can be treated as approximately all included
publications during the title and abstract screening stage). Moreover, all
excluded publications have a reason for exclusion selected by a reviewer. As
the original relevance judgements were limited to publications from the PubMed
database, we assign PubMed IDs to these publications. We also define
appropriate BigBio data loaders so the task can be seen as question-answering
or textual pairs classification task.

Table 2: Details of the CSMeD expanded meta-dataset. Column ‘#docs’ refers to
the total number of documents included in all SLRS within the dataset,
‘#included’ mentions number of included documents on the title and abstract
screening stage and ‘Avg. %included’ the percentage of included publications
averaged from all reviews.

Dataset name | #reviews | #docs | #included |  | Avg.
---
#docs
| Avg. %
---
included
| Avg. #words
---
in document
CSMeD-train-basic | 30 | 128,438 | 7,958 | 4,281 | 9.6% | 229
CSMeD-train-cochrane | 195 | 372,422 | 7,589 | 1,910 | 21.9% | 180
CSMeD-dev-cochrane | 100 | 229,376 | 4,365 | 2,294 | 20.8% | 201
CSMeD-all | 325 | 730,236 | 19,912 | 2,247 | 20.5% | 195

Details of the new expanded CSMeD are provided in Table 2. We were not able to
find suitable data for all SLRs, hence the expanded CSMeD is smaller than the
original meta-dataset. In total, the new expanded dataset consists of 295
unique Cochrane SLRs and 30 non-Cochrane SLRs. The entire set of basic SLRs is
designated for training. From the Cochrane reviews, we randomly selected 195
to the training split and the remaining 100 to the development split. We
abstain from designating a test split because CSMeD aggregates existing
datasets. Given the constraints of these datasets, creating a new, unbiased
test collection is recommended.

###  4.3 Baseline experiments

We evaluate two models in a zero-shot setting: the traditional BM25 ranking
function and the MiniLM-L6-v2777https://huggingface.co/sentence-
transformers/all-MiniLM-L6-v2 Transformer-based model implemented in the
retriv Python package [6]. Predictions are run on the CSMeD-dev-cochrane
split, and we test three different input representations using the following
sections from the SLR protocol: (1) title, (2) abstract, and (3) eligibility
criteria section. We evaluate models using $nDCG@10$, $MAP$, Recall at rank
$k$ with $k$ in {10, 50, 100} ($R@k$). Additionally, we compute three measures
specifically designed for the task of CS: True Negative Rate at 95% Recall
($TNR@95\%$) [40, 41], normalised Precision at 95% Recall ($nP@95\%$) [41],
and average position at which the last relevant item is found [30, 31, 32],
calculated as a percentage of the dataset size, where a lower value indicates
better performance ($Last~{}Rel$).

Table 3:  Baseline results on CSMeD-dev-cochrane dataset. Bold values indicate
best score.

Model | Representation | TNR@95% | nP@95% | Last Rel | NDCG@10 | MAP | R@10 | R@50 | R@100
---|---|---|---|---|---|---|---|---|---
BM25 | Title | 0.469 | 0.142 | 72.2 | 0.438 | 0.388 | 0.349 | 0.623 | 0.704
Abstract | 0.474 | 0.170 | 63.6 | 0.503 | 0.453 | 0.379 | 0.657 | 0.757
Criteria | 0.404 | 0.122 | 84.9 | 0.286 | 0.258 | 0.224 | 0.418 | 0.510
MiniLM | Title | 0.472 | 0.217 | 68.1 | 0.470 | 0.414 | 0.379 | 0.673 | 0.763
Abstract | 0.492 | 0.240 | 65.5 | 0.517 | 0.451 | 0.398 | 0.682 | 0.766
Criteria | 0.347 | 0.144 | 75.4 | 0.299 | 0.299 | 0.233 | 0.492 | 0.588

Table 3 presents the baseline results. Both for the BM25 and MiniLM models,
using the systematic review abstract text as a query representation achieved
the highest performance in all metrics compared to using the SLR title and
criteria sections. The MiniLM models consistently outperform their BM25
variant for all query representations on all evaluation measures except Last
Relevant found. The best-performing model, MiniLM using SLR abstracts,
achieves $TNR@95\%$ equal to almost 0.5, meaning that this model can remove,
on average, almost half of the true negatives when achieving a recall of 95%.
Interestingly, despite the significance attributed to the eligibility criteria
section by reviewers during paper screening, these models seemed challenged in
leveraging the criteria information. It should be noted, however, that this
section is typically more relevant to full-text screening rather than title
and abstract. Exploring more advanced language models might reveal the
potential for using this underutilised information.

##  5 CSMeD-ft: Full text classification dataset

In this section, we introduce CSMeD-ft full text screening dataset and present
baseline experiments.

###  5.1 Dataset creation

LLM advancements have enabled processing long text snippets [7, 93, 54, 21].
Commercial tools now support inputs of up to 32k [60] or even up to 100k
tokens [4]. We propose CSMeD-ft, the full text screening dataset to enable
research associated with the comprehensive understanding of very long
documents, and evaluate such capabilities. We first gather full text versions
of publications from CSMeD SLRs, and then create the appropriate setting with
canonical splits.

We use SemanticScholar and CORE [35, 34] APIs to find URLs to open-access full
text documents. This process successfully finds URLs to, on average, 27% of
all included and excluded publications from SLRs. After downloading full text
PDFs, we use GROBID [1] to parse the content of these documents into an xml
format.

We establish canonical splits considering the timestamps, such that the newest
reviews belong to the test set. Specifically, we select 31 Cochrane reviews
published in the last year (between 01/06/2022 and 31/05/2023) to create a
test set, another 60 reviews (mentioned in Nussbaumer-Streit et al. [56]) for
the development set, and 176 reviews (listed by Scells et al. [69]) as the
training set. Filtering out reviews with no associated available full text
publications results in 148/36/29 reviews in train/dev/test splits.

Details of CSMeD-ft are presented in Table 4. We also release a subset of 50
randomly selected documents from the test set as CSMeD-ft-test-small. At the
moment of writing this publication, creating a prompt for LLMs with an input
of few thousands tokens is feasible albeit costly,888According to the OpenAI
model pricing on 22/09/2023, screening 500 full text documents with the
GPT-4-32k model would cost more than 400 USD. See Appendix D for further
details on the creation of CSMeD-ft.

Table 4: Details of the CSMeD-ft dataset. Column ‘#docs’ refers to the total
number of documents included in the dataset and ‘#included’ mentions number of
included documents on the full text step. CSMeD-ft-test-small is a subset of
CSMeD-ft-test.

Dataset name | #reviews | #docs. | #included |  | % included
---
| Avg. #words
---
in document
| Avg. #words
---
in review
CSMeD-ft-train | 148 | 2,053 | 904 | 44.0% | 4,535 | 1,493
CSMeD-ft-dev | 36 | 644 | 202 | 31.4% | 4,419 | 1,402
CSMeD-ft-test | 29 | 636 | 278 | 43.7% | 4,957 | 2,318
CSMeD-ft-test-small | 16 | 50 | 22 | 44.0% | 5,042 | 2,354

CSMeD-ft could be a proxy for a very long document natural language inference
(NLI) task. Popular NLI datasets (SciTail [33], McTest [64] or DocNLI [91])
contain both hypotheses and premises of an average length considerably shorter
than 1,000 words; whereas in CSMeD-ft, the premise (review protocol) has an
average length of more than 1,000 words, and the hypothesis (publication)
contains more than 4,000 words.

###  5.2 Experiment

We present how CSMeD-ft can be used to evaluate LLMs capabilities in making
eligibility decisions on very long documents. We run experiments both on fine-
tuning of Transformer models and zero-shot prompting of GPT models.

#### Model selection

As the combined input size of systematic review and publication can be big
(9,246 mean number of tokens on a training split measured with a GPT-4
tokeniser), we only select models that allow inputs of at least 4k tokens
context. We fine-tune the open-domain Longformer and BigBird, and domain-
specific models pre-trained on clinical data: Clinical-BigBird and
ClinicalLongformer. For zero-shot evaluation, we select GPT-3.5-turbo-0301,
GPT-4-8k and GPT-3.5-turbo-16k accessed via OpenAI API. GPT-4-8k and
GPT-3.5-turbo-16k are the only models capable of handling more than 4k-input
tokens, with context window size of 8k and 16k tokens respectively.

#### Preprocessing and evaluation

For all models, we concatenate the screening protocol with each publication;
we truncate the review description text to half of the available context
window (2,000 tokens for 4k models, 4,000 tokens for 8k model and 8,000 tokens
for 16k model) and complete the input with a publication.

For GPT models, if a whole publication text does not fit the context window,
we run multiple predictions with a sliding window and aggregate the results.
In the case of GPT-3.5-turbo-16k model, only for 4 out of 50 documents the
model was unable to process the full text of combined review and publication
inside one prompt.

We fine-tune the Transformer models on CSMeD-ft-train for four epochs and run
evaluation on CSMeD-ft-dev. Due to the budget limitation, for the GPT-4-8k
model, we run the evaluation only on CSMeD-ft-test-small (see Appendix F for
further details on experimental settings). Finally, we evaluate the models
using macro-averaged Precision, Recall and F1-score measures.

#### Results

Results of the full text experiment are summarised in Table 5. On CSMeD-ft-
test-small, GPT-4-8k strongly outperforms other models. However, this
difference is not statistically significant. The GPT-3.5-turbo-16k achieves
the highest Precision; this improvement can be attributed to the model’s
expanded context window and the limitations other GPT-based models have with
our simple aggregation rules. However, this might also be caused by
overfitting towards the positive class, as this model includes almost twice as
many publications as other models. On CSMeD-ft-test set, Clinical-BigBird,
significantly outperforms zero-shot GPT-3.5 model and pre-trained models based
on the LongFormer architecture.

Interestingly, both BigBird-based models outperform their counterparts using
the Longformer architecture. The typical overall tendency to domain-pre-
trained models achieving higher scores over their open-domain counterparts is
also preserved. We believe that fine-tuning the Transformer models first on
larger NLI/QA corpora could help improve the results.

Table 5: Results of the full text screening experiment averaged over
documents. The statistical significance was assessed with a McNemar’s t-test
(p < 0.05) with Bonferroni correction for multiple testing. _Clinical-BigBird_
on the CSMeD-ft-test split showed statistically significant improvements
compared to the _stratified random_ baseline, _Longformer_ , _Clinical-
Longformer_ , and _GPT-3.5-turbo-16k_ , indicated by †. Stratified baseline is
averaged from 100 different random seeds. ‘% incl.’ describes the percentage
of documents predicted as relevant by models.

| CSMeD-ft-test-small | CSMeD-ft-test
---|---|---
| % incl. | Precision | Recall | F1-score | % incl. | Precision | Recall | F1-score
oracle | 44% | — | — | — | 43.7% | — | — | —
stratified random | 50% | 0.497 | 0.498 | 0.495 | — | 0.499 | 0.499 | 0.498
‘include all’ | 100% | 0.220 | 0.500 | 0.306 | 100% | 0.219 | 0.500 | 0.304
Longformer [7] | 40% | 0.467 | 0.468 | 0.466 | 40.4% | 0.398 | 0.400 | 0.398
BigBird-roberta-base [93] | 42% | 0.572 | 0.571 | 0.572 | 45.1% | 0.575 | 0.575 | 0.575
Clinical-Longformer [47] | 36% | 0.547 | 0.544 | 0.542 | 35.1% | 0.436 | 0.441 | 0.435
Clinical-BigBird [47] | 36% | 0.590 | 0.584 | 0.583 | 32.8% | 0.623† | 0.611† | 0.609†
GPT-3.5-turbo-0301 | 54% | 0.585 | 0.586 | 0.580 | — | — | — | —
GPT-4-8k-0314 | 58% | 0.674 | 0.672 | 0.660 | — | — | — | —
GPT-3.5-turbo-16k | 80% | 0.712 | 0.638 | 0.576 | 75.9% | 0.538 | 0.528 | 0.475

##  6 Discussion

In this paper, we have addressed the challenge of standardised evaluation in
CS automation. By revisiting existing screening datasets, we evaluated their
suitability as benchmarks in the context of modern ML methods. Our analysis
revealed limitations such as small size and data leakage issues.

To overcome these challenges, we introduced CSMeD, a meta-dataset
consolidating nine publicly released collections, providing programmatic
access to 325 SLRs. CSMeD serves as a comprehensive resource for training and
evaluating automated citation screening models and can be used for tasks that
involve textual pairs classification, question answering and NLI.
Additionally, we included a new dataset within CSMeD for evaluating full text
publication classification and conducted initial experiments showing that
there is a room for improvement in understanding long contexts.

The focus of CSMeD on providing unified access over a number of diverse
citation screening datasets has many benefits. First, the evaluation code can
be re-used, making sure that the evaluation is handled properly. Secondly,
integration with the BigBio framework enables quick prototyping of prompts. We
also improve the documentation for existing datasets and provide a
comprehensive data card for CSMeD-ft. Our extended version of CSMeD is also
deduplicated. Finally, it is a step towards providing a multi-domain SLR
dataset and bridging the gap between IR and NLP research in the domain of
screening automation, enabling direct comparisons of the methods.

#### Limitations and future work

While we attempted to extract the data protocols as accurately as possible,
extraction of data was not possible for all previous reviews. This was
primarily due to the changing standards in Cochrane reviews throughout the
years. In future work, ideally, direct access to Cochrane metadata would be
needed to make sure that all information is covered. Even though the PubMed
publications most probably will not change, what can change is the API and
scripts necessary to download the data. There exists also the possibility that
one of the sources will introduce a restriction on using their data for
training and evaluation of machine learning models. We tried to further
mitigate this potential issue by selecting open-access SLRs produced by
Cochrane. Finally, we acknowledge that using machine assistance for citation
selection can raise concerns about research quality, emphasising the vital
role of human oversight throughout the process.

Future work will focus on further improving data quality, connecting the
output reviews from screening tools like _CRUISE-Screening_ [39], adding
datasets covering other domains and different SLR tasks and designing a
dataset for a prospective evaluation of review automation which could ensure
no data leakage [12]. For the prospective dataset, predictions could be made
as soon as the protocol is published, and the gold standard data becomes
available when the review is eventually published, albeit with the drawback of
a potentially long waiting time for review publication.

##  7 Conclusion

Our paper introduces CSMeD, a meta-dataset that addresses the lack of
standardisation in SLR automation. By consolidating datasets and providing a
unified access point, CSMeD facilitates the development and evaluation of
automated citation screening and full text classification models. We believe
it has the potential to advance the field and lead to more robust automated
SLR systems. We envision CSMeD as a living, evolving collection, and we invite
researchers to contribute to expanding it with SLR datasets from other
domains.

## Acknowledgments and Disclosure of Funding

This work was supported by the EU Horizon 2020 ITN/ETN on Domain Specific
Systems for Information Extraction and Retrieval – DoSSIER (H2020-EU.1.3.1.,
ID: 860721).

## References

  * GRO [2008–2022] Grobid.  https://github.com/kermitt2/grobid, 2008–2022.
  * Alharbi and Stevenson [2019] Amal Alharbi and Mark Stevenson.  A dataset of systematic review updates.  _SIGIR 2019 - Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1257–1260, 7 2019.  doi: 10.1145/3331184.3331358.
  * Alharbi and Stevenson [2020] Amal Alharbi and Mark Stevenson.  Refining Boolean queries to identify relevant studies for systematic review updates.  _Journal of the American Medical Informatics Association_ , 27(11):1658–1666, 10 2020.  ISSN 1527-974X.  doi: 10.1093/jamia/ocaa148.  URL https://doi.org/10.1093/jamia/ocaa148.
  * Anthropic [2023] Anthropic.  claude-v1.3-100k, Blog post ’Introducing 100K Context Windows’.  https://www.anthropic.com/index/100k-context-windows, 2023.
  * Badami et al. [2023] Maisie Badami, Boualem Benatallah, and Marcos Baez.  Adaptive search query generation and refinement in systematic literature review.  _Information Systems_ , page 102231, 2023.  ISSN 0306-4379.  doi: https://doi.org/10.1016/j.is.2023.102231.  URL https://www.sciencedirect.com/science/article/pii/S0306437923000674.
  * Bassani [2023] Elias Bassani.  retriv: A Python Search Engine for the Common Man, May 2023.  URL https://github.com/AmenRa/retriv.
  * Beltagy et al. [2020] Iz Beltagy, Matthew E Peters, and Arman Cohan.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_ , 2020.
  * Blagec et al. [2022] Kathrin Blagec, Jakob Kraiger, Wolfgang Frühwirt, and Matthias Samwald.  Benchmark datasets driving artificial intelligence development fail to capture the needs of medical professionals.  1 2022.  URL https://arxiv.org/abs/2201.07040v1.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Clark et al. [2020] Justin Clark, Paul Glasziou, Chris Del Mar, Alexandra Bannach-Brown, Paulina Stehlik, and Anna Mae Scott.  A full systematic review was completed in 2 weeks using automation tools: a case study.  _Journal of clinical epidemiology_ , 121:81–90, 2020.
  * Cohen et al. [2006] A. M. Cohen, W. R. Hersh, K. Peterson, and Po Yin Yen.  Reducing workload in systematic review preparation using automated citation classification.  _Journal of the American Medical Informatics Association_ , 13(2):206–219, 3 2006.  ISSN 10675027.  doi: 10.1197/jamia.M1929.  URL /pmc/articles/PMC1447545//pmc/articles/PMC1447545/?report=abstracthttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC1447545/.
  * Cohen et al. [2010] Aaron M Cohen, Kyle Ambert, and Marian McDonagh.  A prospective evaluation of an automated classification system to support evidence-based medicine and systematic review.  In _AMIA annual symposium proceedings_ , volume 2010, page 121. American Medical Informatics Association, 2010.
  * Dalal et al. [2013] Siddhartha R Dalal, Paul G Shekelle, Susanne Hempel, Sydne J Newberry, Aneesa Motala, and Kanaka D Shetty.  A pilot study using machine learning and domain knowledge to facilitate comparative effectiveness review updating.  _Medical Decision Making_ , 33(3):343–355, 2013.
  * DeYoung et al. [2021] Jay DeYoung, Iz Beltagy, Madeleine van Zuylen, Bailey Kuehl, and Lucy Lu Wang.  Msˆ2: Multi-document summarization of medical studies.  In _EMNLP_ , 2021.
  * Dong et al. [2019] Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon.  Unified language model pre-training for natural language understanding and generation.  _Advances in neural information processing systems_ , 32, 2019.
  * Fries et al. [2022] Jason Fries, Leon Weber, Natasha Seelam, Gabriel Altay, Debajyoti Datta, Samuele Garda, Sunny Kang, Rosaline Su, Wojciech Kusa, Samuel Cahyawijaya, Fabio Barth, Simon Ott, Matthias Samwald, Stephen Bach, Stella Biderman, Mario Sänger, Bo Wang, Alison Callahan, Daniel León Periñán, Théo Gigant, Patrick Haller, Jenny Chim, Jose Posada, John Giorgi, Karthik Rangasai Sivaraman, Marc Pàmies, Marianna Nezhurina, Robert Martin, Michael Cullan, Moritz Freidank, Nathan Dahlberg, Shubhanshu Mishra, Shamik Bose, Nicholas Broad, Yanis Labrak, Shlok Deshmukh, Sid Kiblawi, Ayush Singh, Minh Chien Vu, Trishala Neeraj, Jonas Golde, Albert Villanova del Moral, and Benjamin Beilharz.  Bigbio: A framework for data-centric biomedical natural language processing.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, _Advances in Neural Information Processing Systems_ , volume 35, pages 25792–25806. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/a583d2197eafc4afdd41f5b8765555c5-Paper-Datasets_and_Benchmarks.pdf.
  * Gao et al. [2021] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.  A framework for few-shot language model evaluation, September 2021.  URL https://doi.org/10.5281/zenodo.5371628.
  * Gebru et al. [2021] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daumé III, and Kate Crawford.  Datasheets for datasets.  _Commun. ACM_ , 64(12):86–92, 2021.  doi: 10.1145/3458723.  URL https://doi.org/10.1145/3458723.
  * Grossman et al. [2016] Maura R Grossman, Gordon V Cormack, and Adam Roegiest.  TREC 2016 Total Recall Track Overview.  In _TREC_ , 2016.
  * Gu et al. [2022] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon.  Domain-specific language model pretraining for biomedical natural language processing.  _ACM Trans. Comput. Heal._ , 3(1):2:1–2:23, 2022.  doi: 10.1145/3458754.  URL https://doi.org/10.1145/3458754.
  * Guo et al. [2021] Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang.  Longt5: Efficient text-to-text transformer for long sequences.  _arXiv preprint arXiv:2112.07916_ , 2021.
  * Hannousse and Yahiouche [2022] Abdelhakim Hannousse and Salima Yahiouche.  A semi-automatic document screening system for computer science systematic reviews.  In _Mediterranean Conference on Pattern Recognition and Artificial Intelligence_ , pages 201–215. Springer, 2022.
  * Hashimoto et al. [2016] Kazuma Hashimoto, Georgios Kontonatsios, Makoto Miwa, and Sophia Ananiadou.  Topic detection using paragraph vectors to support active learning in systematic reviews.  _Journal of Biomedical Informatics_ , 62:59–65, 8 2016.
  * Higgins et al. [2019] Julian PT Higgins, James Thomas, Jacqueline Chandler, Miranda Cumpston, Tianjing Li, Matthew J Page, and Vivian A Welch.  _Cochrane handbook for systematic reviews of interventions_.  John Wiley & Sons, 2019.
  * hitz-zentroa [2023] hitz-zentroa.  LM Contamination Index.  GitHub repository, 2023.  URL {https://github.com/hitz-zentroa/lm-contamination}.
  * Hou et al. [2022] Jingwen Hou, Xiaochen Wang, Jean-Jacques Dubois, R. Byron Rice, Amanda Haddock, and Yue Wang.  Extreme systematic reviews: A large literature screening dataset to support environmental policymaking.  In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_, CIKM ’22, page 4029–4033, New York, NY, USA, 2022. Association for Computing Machinery.  ISBN 9781450392365.  doi: 10.1145/3511808.3557600.  URL https://doi.org/10.1145/3511808.3557600.
  * Howard et al. [2016] Brian E. Howard, Jason Phillips, Kyle Miller, Arpit Tandon, Deepak Mav, Mihir R. Shah, Stephanie Holmgren, Katherine E. Pelch, Vickie Walker, Andrew A. Rooney, Malcolm Macleod, Ruchir R. Shah, and Kristina Thayer.  SWIFT-Review: A text-mining workbench for systematic review.  _Systematic Reviews_ , 5(1):1–16, 5 2016.  ISSN 20464053.  doi: 10.1186/s13643-016-0263-z.  URL https://link.springer.com/articles/10.1186/s13643-016-0263-zhttps://link.springer.com/article/10.1186/s13643-016-0263-z.
  * Howard et al. [2020] Brian E. Howard, Jason Phillips, Arpit Tandon, Adyasha Maharana, Rebecca Elmore, Deepak Mav, Alex Sedykh, Kristina Thayer, B. Alex Merrick, Vickie Walker, Andrew Rooney, and Ruchir R. Shah.  SWIFT-Active Screener: Accelerated document screening through active learning and integrated recall estimation.  _Environment International_ , 138:105623, 5 2020.  ISSN 0160-4120.  doi: 10.1016/J.ENVINT.2020.105623.
  * Jo et al. [2009] Akers Jo, Aguiar-Ibáñez Raquel, Burch Jane, Chambers Duncan, Eastwood Alison, Fayter Debra, Hempel Susanne, Light Kate, Rice Stephen, Rithalia Amber, Stewart Lesley, Stock Christian, Wilson Paul, and Woolacott Nerys.  _Systematic Reviews: CRD’s guidance for undertaking reviews in health care_.  CRD, University of York, York, 1 2009.  ISBN 978-1-900991-19-3.  URL www.york.ac.uk/inst/crd.
  * Kanoulas et al. [2017] Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker.  CLEF 2017 technologically assisted reviews in empirical medicine overview.  _CEUR Workshop Proceedings_ , 1866:1–29, 9 2017.  ISSN 1613-0073.  URL https://pureportal.strath.ac.uk/en/publications/clef-2017-technologically-assisted-reviews-in-empirical-medicine-.
  * Kanoulas et al. [2018] Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker.  CLEF 2018 technologically assisted reviews in empirical medicine overview.  _CEUR Workshop Proceedings_ , 2125, 7 2018.  ISSN 1613-0073.  URL https://pureportal.strath.ac.uk/en/publications/clef-2018-technologically-assisted-reviews-in-empirical-medicine-.
  * Kanoulas et al. [2019] Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker.  CLEF 2019 Technology Assisted Reviews in Empirical Medicine Overview.  In _CLEF_ , 2019.
  * Khot et al. [2018] Tushar Khot, Ashish Sabharwal, and Peter Clark.  Scitail: A textual entailment dataset from science question answering.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Knoth and Zdrahal [2012] Petr Knoth and Zdenek Zdrahal.  Core: three access levels to underpin open access.  _D-Lib Magazine_ , 18(11/12), 2012.  URL http://oro.open.ac.uk/35755/.
  * Knoth et al. [2023] Petr Knoth, Drahomira Herrmannova, Matteo Cancellieri, Lucas Anastasiou, Nancy Pontika, Samuel Pearce, Bikash Gyawali, and David Pride.  Core: A global aggregation service for open access papers.  _Nature Scientific Data_ , 10(1):366, June 2023.
  * Kontonatsios et al. [2017] Georgios Kontonatsios, Austin J. Brockmeier, Piotr Przybyła, John McNaught, Tingting Mu, John Y. Goulermas, and Sophia Ananiadou.  A semi-supervised approach using label propagation to support citation screening.  _Journal of Biomedical Informatics_ , 72:67–76, 8 2017.
  * Kontonatsios et al. [2020] Georgios Kontonatsios, Sally Spencer, Peter Matthew, and Ioannis Korkontzelos.  Using a neural network-based feature extraction method to facilitate citation screening for systematic reviews.  _Expert Systems with Applications: X_ , 6:100030, 7 2020.  ISSN 25901885.  doi: 10.1016/j.eswax.2020.100030.
  * Kusa et al. [2022] Wojciech Kusa, Allan Hanbury, and Petr Knoth.  Automation of citation screening for systematic literature reviews using neural networks: A replicability study.  In Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil Nørvåg, and Vinay Setty, editors, _Advances in Information Retrieval_ , pages 584–598, Cham, 2022. Springer International Publishing.  ISBN 978-3-030-99736-6.  URL https://arxiv.org/abs/2201.07534v1.
  * Kusa et al. [2023a] Wojciech Kusa, Petr Knoth, and Allan Hanbury.  CRUISE-Screening: Living Literature Reviews Toolbox.  In _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_ , CIKM ’23, page 5071–5075, New York, NY, USA, 2023a. Association for Computing Machinery.  ISBN 9798400701245.  doi: 10.1145/3583780.3614736.  URL https://doi.org/10.1145/3583780.3614736.
  * Kusa et al. [2023b] Wojciech Kusa, Aldo Lipani, Petr Knoth, and Allan Hanbury.  An Analysis of Work Saved over Sampling in the Evaluation of Automated Citation Screening in Systematic Literature Reviews.  _Intelligent Systems with Applications_ , 18:200193, 2023b.  ISSN 2667-3053.  doi: https://doi.org/10.1016/j.iswa.2023.200193.  URL https://www.sciencedirect.com/science/article/pii/S2667305323000182.
  * Kusa et al. [2023c] Wojciech Kusa, Aldo Lipani, Petr Knoth, and Allan Hanbury.  VoMBaT: A Tool for Visualising Evaluation Measure Behaviour in High-Recall Search Tasks.  In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’23)_ , page 5, Taipei, Taiwan, July 23–27 2023c. ACM.  URL https://doi.org/10.1145/3539618.3591802.
  * Kusa et al. [2023d] Wojciech Kusa, Guido Zuccon, Petr Knoth, and Allan Hanbury.  Outcome-based evaluation of systematic review automation.  In _Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval_ , ICTIR ’23, page 125–133, New York, NY, USA, 2023d. Association for Computing Machinery.  ISBN 9798400700736.  doi: 10.1145/3578337.3605135.  URL https://doi.org/10.1145/3578337.3605135.
  * Lee and Ho [2023a] Eric W Lee and Joyce C Ho.  Pgb: A pubmed graph benchmark for heterogeneous network representation learning.  _arXiv preprint arXiv:2305.02691_ , 2023a.
  * Lee and Ho [2023b] Eric W Lee and Joyce C Ho.  Sr-comber: Heterogeneous network embedding using community multi-view enhanced graph convolutional network for automating systematic reviews.  In _European Conference on Information Retrieval_ , pages 553–568. Springer, 2023b.
  * Lewis et al. [2020] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.703.  URL https://aclanthology.org/2020.acl-main.703.
  * Lhoest et al. [2021] Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, et al.  Datasets: A community library for natural language processing.  _arXiv preprint arXiv:2109.02846_ , 2021.
  * Li et al. [2022] Yikuan Li, Ramsey M Wehbe, Faraz S Ahmad, Hanyin Wang, and Yuan Luo.  Clinical-longformer and clinical-bigbird: Transformers for long clinical sequences.  _arXiv preprint arXiv:2201.11838_ , 2022.
  * Liu et al. [2021] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig.  Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing.  7 2021.  URL https://arxiv.org/abs/2107.13586v1.
  * Loshchilov and Hutter [2018] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_ , 2018.
  * Marshall et al. [2016] Iain J Marshall, Joël Kuiper, and Byron C Wallace.  Robotreviewer: evaluation of a system for automatically assessing bias in clinical trials.  _Journal of the American Medical Informatics Association_ , 23(1):193–201, 2016.
  * Matwin et al. [2010] Stan Matwin, Alexandre Kouznetsov, Diana Inkpen, Oana Frunza, and Peter O’Blenis.  A new algorithm for reducing the workload of experts in performing systematic reviews.  _Journal of the American Medical Informatics Association_ , 17(4):446–453, 7 2010.  doi: 10.1136/JAMIA.2010.004325.
  * McInnes et al. [2018] Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger.  Umap: Uniform manifold approximation and projection.  _The Journal of Open Source Software_ , 3(29):861, 2018.
  * Miwa et al. [2014] Makoto Miwa, James Thomas, Alison O’Mara-Eves, and Sophia Ananiadou.  Reducing systematic review workload through certainty-based screening.  _Journal of Biomedical Informatics_ , 51:242–253, 10 2014.  ISSN 1532-0464.  doi: 10.1016/J.JBI.2014.06.005.
  * Mohtashami and Jaggi [2023] Amirkeivan Mohtashami and Martin Jaggi.  Landmark attention: Random-access infinite context length for transformers.  _arXiv preprint arXiv:2305.16300_ , 2023.
  * Norman [2020] Christopher Norman.  _Systematic review automation methods_.  PhD thesis, Université Paris-Saclay ; Universiteit van Amsterdam, 2 2020.  URL https://tel.archives-ouvertes.fr/tel-03060620.
  * Nussbaumer-Streit et al. [2018] Barbara Nussbaumer-Streit, Irma Klerings, Gernot Wagner, Thomas L. Heise, Andreea I. Dobrescu, Susan Armijo-Olivo, Jan M. Stratil, Emma Persad, Stefan K. Lhachimi, Megan G. Van Noord, Tarquin Mittermayr, Hajo Zeeb, Lars Hemkens, and Gerald Gartlehner.  Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study.  _Journal of Clinical Epidemiology_ , 102:1–11, 2018.  ISSN 0895-4356.  doi: https://doi.org/10.1016/j.jclinepi.2018.05.022.  URL https://www.sciencedirect.com/science/article/pii/S0895435618300179.
  * Nussbaumer-Streit et al. [2021] Barbara Nussbaumer-Streit, Moriah Ellen, Irma Klerings, Raluca Sfetcu, Nicoletta Riva, Mersiha Mahmić-Kaknjo, Georgios Poulentzas, P Martinez, Eduard Baladia, Liliya Eugenevna Ziganshina, et al.  Resource use during systematic review production varies widely: a scoping review.  _Journal of Clinical Epidemiology_ , 139:287–296, 2021.
  * Nye et al. [2018] Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain J Marshall, Ani Nenkova, and Byron C Wallace.  A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature.  In _Proceedings of the conference. Association for Computational Linguistics. Meeting_ , volume 2018, page 197. NIH Public Access, 2018.
  * O’Mara-Eves et al. [2015] Alison O’Mara-Eves, James Thomas, John McNaught, Makoto Miwa, and Sophia Ananiadou.  Using text mining for study identification in systematic reviews: A systematic review of current approaches.  _Systematic Reviews_ , 4(1):5, 1 2015.  ISSN 20464053.  doi: 10.1186/2046-4053-4-5.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report, 2023.
  * Parmar et al. [2022] Mihir Parmar, Swaroop Mishra, Mirali Purohit, Man Luo, Murad Mohammad, and Chitta Baral.  In-BoXBART: Get instructions into biomedical multi-task learning.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 112–128, Seattle, United States, July 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.findings-naacl.10.  URL https://aclanthology.org/2022.findings-naacl.10.
  * Parmar [2021] Mihir Prafullsinh Parmar.  Automation of title and abstract screening for clinical systematic reviews.  Master’s thesis, Arizona State University, 2021.  URL https://keep.lib.asu.edu/_flysystem/fedora/c7/Parmar_asu_0010N_21179.pdf.
  * Peng et al. [2019] Yifan Peng, Shankai Yan, and Zhiyong Lu.  Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets.  In _Proceedings of the 18th BioNLP Workshop and Shared Task_ , pages 58–65, Florence, Italy, August 2019. Association for Computational Linguistics.  doi: 10.18653/v1/W19-5006.  URL https://aclanthology.org/W19-5006.
  * Richardson et al. [2013] Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw.  MCTest: A challenge dataset for the open-domain machine comprehension of text.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 193–203, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.  URL https://aclanthology.org/D13-1020.
  * Roberts et al. [2017] Kirk Roberts, Dina Demner-Fushman, Ellen M Voorhees, William R Hersh, Steven Bedrick, Alexander J Lazar, and Shubham Pant.  Overview of the trec 2017 precision medicine track.  In _TREC_ , 2017.
  * Roberts et al. [2021] Kirk Roberts, Dina Demner-Fushman, Ellen M Voorhees, Steven Bedrick, and Willian R Hersh.  Overview of the trec 2021 clinical trials track.  In _Proceedings of the Thirtieth Text REtrieval Conference (TREC 2021)_ , 2021.
  * Robertson et al. [1995] Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al.  Okapi at trec-3.  _Nist Special Publication Sp_ , 109:109, 1995.
  * Scells and Zuccon [2018] Harrisen Scells and Guido Zuccon.  Generating better queries for systematic reviews.  In _The 41st international ACM SIGIR conference on research & development in information retrieval_, pages 475–484, 2018.
  * Scells et al. [2017] Harrisen Scells, Guido Zuccon, Bevan Koopman, Anthony Deacon, Leif Azzopardi, and Shlomo Geva.  A test collection for evaluating retrieval of studies for inclusion in systematic reviews.  _SIGIR 2017 - Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1237–1240, 8 2017.  doi: 10.1145/3077136.3080707.
  * Scells et al. [2021] Harrisen Scells, Guido Zuccon, and Bevan Koopman.  A comparison of automatic boolean query formulation for systematic reviews.  _Information Retrieval Journal_ , 24(1):3–28, 2021.
  * Singh et al. [2018] Gaurav Singh, James Thomas, and John Shawe-Taylor.  Improving active learning in systematic reviews.  _arXiv preprint arXiv:1801.09496_ , 2018.
  * Thakur et al. [2021] Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych.  BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models.  4 2021.  URL https://arxiv.org/abs/2104.08663v4.
  * Tsafnat et al. [2013] Guy Tsafnat, Adam Dunn, Paul Glasziou, and Enrico Coiera.  The automation of systematic reviews, 2013.
  * Tsafnat et al. [2014] Guy Tsafnat, Paul Glasziou, Miew K. Choong, Adam Dunn, Filippo Galgani, and Enrico Coiera.  Systematic review automation technologies, 4 2014.  ISSN 20464053.
  * Tsafnat et al. [2018] Guy Tsafnat, Paul Glasziou, George Karystianis, and Enrico Coiera.  Automated screening of research studies for systematic reviews using study characteristics.  _Systematic Reviews 2018 7:1_ , 7(1):1–9, 4 2018.  ISSN 2046-4053.  doi: 10.1186/S13643-018-0724-7.  URL https://link.springer.com/articles/10.1186/s13643-018-0724-7https://link.springer.com/article/10.1186/s13643-018-0724-7.
  * van Dinter et al. [2021a] Raymon van Dinter, Cagatay Catal, and Bedir Tekinerdogan.  A decision support system for automating document retrieval and citation screening.  _Expert Systems with Applications_ , 182, 11 2021a.
  * van Dinter et al. [2021b] Raymon van Dinter, Cagatay Catal, and Bedir Tekinerdogan.  A Multi-Channel Convolutional Neural Network approach to automate the citation screening process.  _Applied Soft Computing_ , 112:107765, 11 2021b.  ISSN 1568-4946.  doi: 10.1016/J.ASOC.2021.107765.
  * van Dinter et al. [2021c] Raymon van Dinter, Bedir Tekinerdogan, and Cagatay Catal.  Automation of systematic literature reviews: A systematic literature review.  _Information and Software Technology_ , 136:106589, 8 2021c.  ISSN 09505849.  doi: 10.1016/j.infsof.2021.106589.  URL https://linkinghub.elsevier.com/retrieve/pii/S0950584921000690.
  * Wallace et al. [2010a] Byron C. Wallace, Kevin Small, Carla E. Brodley, and Thomas A. Trikalinos.  Active learning for biomedical citation screening.  _Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 173–181, 2010a.  doi: 10.1145/1835804.1835829.
  * Wallace et al. [2010b] Byron C Wallace, Thomas A Trikalinos, Joseph Lau, Carla Brodley, and Christopher H Schmid.  Semi-automated screening of biomedical citations for systematic reviews.  _BMC bioinformatics_ , 11(1):1–11, 2010b.
  * Wallace et al. [2011] Byron C Wallace, Kevin Small, Carla E Brodley, and Thomas A Trikalinos.  Who should label what? instance allocation in multiple expert active learning.  In _Proceedings of the 2011 SIAM international conference on data mining_ , pages 176–187. SIAM, 2011.
  * Wallace et al. [2020] Byron C. Wallace, Sayantani Saha, Frank Soboczenski, and Iain James Marshall.  Generating (factual?) narrative summaries of rcts: Experiments with neural multi-document summarization.  _AMIA Annual Symposium_ , abs/2008.11293, 2020.
  * Wang et al. [2018] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_ , 2018.
  * Wang et al. [2022a] Lucy Lu Wang, Jay DeYoung, and Byron Wallace.  Overview of MSLR2022: A shared task on multi-document summarization for literature reviews.  In _Proceedings of the Third Workshop on Scholarly Document Processing_ , pages 175–180, Gyeongju, Republic of Korea, October 2022a. Association for Computational Linguistics.  URL https://aclanthology.org/2022.sdp-1.20.
  * Wang et al. [2021] Shuai Wang, Harrisen Scells, Ahmed Mourad, and Guido Zuccon.  Seed-driven Document Ranking for Systematic Reviews: A Reproducibility Study.  12 2021.  URL https://arxiv.org/abs/2112.04090v1.
  * Wang et al. [2022b] Shuai Wang, Harrisen Scells, Justin Clark, Bevan Koopman, and Guido Zuccon.  From little things big things grow: A collection with seed studies for medical systematic review literature search.  _arXiv preprint arXiv:2204.03096_ , 2022b.
  * Wang et al. [2023a] Shuai Wang, Hang Li, and Guido Zuccon.  Mesh suggester: A library and system for mesh term suggestion for systematic review boolean query construction.  In _Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining_ , WSDM ’23, page 1176–1179, New York, NY, USA, 2023a. Association for Computing Machinery.  ISBN 9781450394079.  doi: 10.1145/3539597.3573025.  URL https://doi.org/10.1145/3539597.3573025.
  * Wang et al. [2023b] Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon.  Can ChatGPT write a good boolean query for systematic review literature search?  _arXiv preprint arXiv:2302.03495_ , 2023b.
  * Wang et al. [2022c] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen.  Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5085–5109, Abu Dhabi, United Arab Emirates, December 2022c. Association for Computational Linguistics.  URL https://aclanthology.org/2022.emnlp-main.340.
  * Weber et al. [2021] Leon Weber, Mario Sänger, Jannes Münchmeyer, Maryam Habibi, Ulf Leser, and Alan Akbik.  Hunflair: an easy-to-use tool for state-of-the-art biomedical named entity recognition.  _Bioinformatics_ , 37(17):2792–2794, 2021.
  * Yin et al. [2021] Wenpeng Yin, Dragomir Radev, and Caiming Xiong.  DocNLI: A large-scale dataset for document-level natural language inference.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4913–4922, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.findings-acl.435.  URL https://aclanthology.org/2021.findings-acl.435.
  * Yun et al. [2023] Hye Sun Yun, Iain J Marshall, Thomas Trikalinos, and Byron C Wallace.  Appraising the potential uses and harms of LLMs for medical systematic reviews.  _arXiv preprint arXiv:2305.11828_ , 2023.
  * Zaheer et al. [2020] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al.  Big bird: Transformers for longer sequences.  _Advances in Neural Information Processing Systems_ , 33:17283–17297, 2020.
  * Zhang et al. [2022] Ningyu Zhang, Mosha Chen, Zhen Bi, Xiaozhuan Liang, Lei Li, Xin Shang, Kangping Yin, Chuanqi Tan, Jian Xu, Fei Huang, Luo Si, Yuan Ni, Guotong Xie, Zhifang Sui, Baobao Chang, Hui Zong, Zheng Yuan, Linfeng Li, Jun Yan, Hongying Zan, Kunli Zhang, Buzhou Tang, and Qingcai Chen.  CBLUE: A Chinese biomedical language understanding evaluation benchmark.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7888–7915, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.544.  URL https://aclanthology.org/2022.acl-long.544.

## Appendix overview

This section provides an overview of the supplementary materials required by
NeurIPS for our submission.

Appendix A offers an extended literature review encompassing citation
screening datasets, evaluation measures used, and dataset coverage for other
systematic literature review (SLR) steps. Appendix B presents detailed
descriptions of the visualisations we have created. Appendix C provides
documentation for the CSMeD meta-dataset, including the datasheet. For the
CSMeD-ft dataset, please refer to Appendix D for its detailed documentation.
In Appendix E, we delve into the specifics of dataset overlap, while Appendix
F contains the experimental details.

All data loaders and data preprocessing scripts for CSMeD are available under
the following URL: https://github.com/WojciechKusa/systematic-review-datasets.
CSMeD-ft can also be accessed under the following URL:
https://github.com/WojciechKusa/systematic-review-
datasets/raw/main/data/CSMeD/CSMeD-FT.zip

##  Appendix A Detailed literature review of datasets

We base our literature review on three recent surveys, which we extend to
cover the results until May 2023:

  * •

Systematic review conducted by O’Mara-Eves et al. [59] in 2015.

  * •

Update to the review above, completed by Norman [55] in 2020.

  * •

Systematic review conducted by van Dinter et al. [76] in 2021.

###  A.1 Citation screening datasets

We searched Google Scholar and Semantic Scholar for publications introducing
new datasets for the citation screening task. We then searched for the forward
citations of the original publication to find usages of the datasets. From our
list, we excluded private datasets used in only one publication. We found 12
datasets fulfilling the criteria.999Between the submission of the main paper
and the supplementary materials, one more new citation screening dataset with
10 SLRs was released on 5 June 2023 [5]. Table 6 presents a summary of these
datasets.

Table 6: Systematic literature review datasets with their characteristics,
sorted by the publication year. We included all publicly available datasets
and private datasets which were used in more than one publication.

|  | Publication
---
# reviews | Domain |  | Data
---
URL
| Publicly
---
available
| In
---
CSMeD
(1) |  Cohen et al. [11], 2006 | 15 | Drug | URL | $\checkmark$ | $\checkmark$
(2) |  Wallace et al. [80], 2010 | 3 | Clinical | URL | $\checkmark$ | $\checkmark$
(3) |  Miwa et al. [53], 2014 | 4 | Social science | — | — | —
(4) |  Howard et al. [27], 2016 | 5 | Mixed | URL | $\checkmark$ | $\checkmark$
(5) |  Scells et al. [69], 2017 | 93 | Clinical | URL | $\checkmark$ | $\checkmark$
(6) |  Kanoulas et al. [30], 2017 | 50 | DTA | URL | $\checkmark$ | $\checkmark$
(7) |  Kanoulas et al. [31], 2018 | 30 | DTA | URL | $\checkmark$ | $\checkmark$
(8) |  Kanoulas et al. [32], 2019 | 49 | Mixed | URL | $\checkmark$ | $\checkmark$
(9) |  Alharbi and Stevenson [2], 2019 | 25 | Clinical | URL | $\checkmark$ | $\checkmark$
(10) |  Parmar [62], 2021 | 6 | Biomedical | — | — | —
(11) |  Wang et al. [86], 2022 | 40 | Clinical | URL | $\checkmark$ | —
(12) |  Hannousse and Yahiouche [22], 2022 | 7 | Comp. Science | URL | $\checkmark$ | $\checkmark$

A dataset created by Cohen et al. [11] containing 15 SLRs is the first and, up
until today, one of the most commonly used to evaluate the effectiveness of
machine learning models. Since then, more datasets have been introduced, and
starting in 2016, a new dataset was released almost every year. All these
datasets differ in the total number of reviews, subdomain, average review
size, and percentage of included studies. However, the overall tendency shows
a very high-class imbalance towards the negative class (i.e., irrelevant
publications). Datasets introduced by Parmar [62] and Miwa et al. [53] are not
publicly available, yet they were used in two and three research papers,
respectively, so we included them in our comparison.

Until 2017 all of the datasets contained only the citation list with
eligibility decisions [55]. More recently, datasets started to include titles
of SLRs and search queries used for finding publications. Additional metadata
is limited to search queries [69], review protocols (three datasets released
as a part of the CLEF TAR shared-task by Kanoulas et al. [30, 31, 32]), review
updates [2] and seed studies [86]. However, none of the datasets includes the
eligibility criteria, the most critical section of SLR text used by manual
annotators when assessing the relevance of publications. They also do not
contain the information about why a particular paper was excluded from the
review. Without this data, the automated citation screening problem cannot be
tackled in any other way than a binary decision. This is not the case in real
life, as a typical SLR contains at least several exclusion and inclusion
criteria, and the decision about every paper can be presented as a multi-
dimensional relevance problem.

So far, there has been little attention to review automation outside of the
medical domain. The only available datasets are four social science reviews by
Miwa et al. [53], and seven computer science reviews by Hannousse and
Yahiouche [22]. Compared to the general interest and rate of production of
SLRs in other domains, this overall underrepresentation of benchmark datasets
could be improved. We also found one dataset containing one large SLR of
environmental policies [26], which has a different scope and format than other
datasets, so we decided not to include it in CSMeD yet.

Papers from the ML and NLP domains, very often evaluate their approaches on
datasets introduced by Cohen et al. [11], which is, at the moment of writing
this review, 17 years old. On the other hand, IR focused papers present their
evaluation on CLEF TAR task datasets.

In terms of evaluation of classification approaches, aside from Precision and
Recall, metrics include variations of the harmonised mean between the two,
i.e. Fβ–score, $utility$, $U19$ [80, 79, 81], sensitivity-maximising
thresholds [13], and $AUC$ [12]. Work Saved over Sampling ($WSS$) was proposed
as a custom metric for evaluating this task as it measures the amount of work
saved when using machine learning models to screen irrelevant publications
[11, 51, 37, 38]. The True Negative Rate ($TNR$) was proposed as an
alternative as it addresses some of the limitations of WSS regarding averaging
scores from multiple datasets [40]. The measures of normalised Precision at r%
recall (nPrecision@r%) and normalised rectified TNR at r% recall (nReTNR@r%)
have also been introduced to focus on other important aspects of screening
task: screening full texts and estimating users’ time savings when compared to
the random ranking, respectively [41].

Cost-based and economic-based metrics were also used, especially in the
context of the query formulation task in the CLEF TAR shared task [32, 30,
31], e.g., total cost (TC) or total cost with a weighted penalty (TCW). The
TREC Total Recall track [19] also used a cut-off based metric, $recall@aR+b$,
which is defined as the recall achieved when $aR+b$ documents have been
identified, where $R$ is the number of relevant documents in the collection
and $a$ and $b$ are parameters. When $a=1$ and $b=0$, $recall@aR+b$ is
equivalent to R-precision. Finally, there has been a proposal to shift away
from measuring Recall and instead evaluate how accurately automated methods
can replicate the original systematic review outcomes [42].

The practical relevance of evaluating CS with metrics like the area under the
ROC curve (AUC) [43] has been called into question, as it may not align with
the goals of the citation screening task. Given that the CS task is primarily
focused on achieving high recall, using AUC as an evaluation metric can be
misleading, as it may highlight model improvements at lower recall values
[40]. Having a unified benchmarking approach would also help to resolve these
problems.

Finally, we were interested in checking how recently each dataset was used,
where that usage was published, and what kind of evaluation measures were
applied to that data. Table 7 presents the summary of our findings. We can see
that to this date, most datasets were used in the past two years and
simultaneously used by different publications. There is also a disparity in
used evaluation measures, yet the basic Precision, Recall and F1-score
prevail.

Table 7: Usage statistics of the SLR datasets, including the latest
publication year, venue and evaluation measure. We report two usages in case
there was a more recent pre-print published.

| Release - last time used |  | Evaluation schema (latest)
---
| Venue (latest)
---
(1) | 2006 - 2023 [44, 40] | TNR [40], AUC [44] | ECIR
(2) | 2010 - 2022 [38] | WSS, Precision@95% [38] | ECIR
(3) | 2014 - 2016 [23] | Yield, Burden, WSS [23] | JBI
(4) | 2016 - 2022 [38], 2023 [43] | WSS, Precision@95% [38], AUC [43] | ECIR
(5) | 2017 - 2018 [68] | Precision, Recall, WSS [68] | SIGIR
(6) | 2017 - 2023 [87] | Precision, F1, Recall [87] | WSDM
(7) | 2018 - 2023 [87] | Precision, F1, Recall [87] | WSDM
(8) | 2019 - 2022 [85], 2023 [43] | MAP, Precision, nDCG [85], AUC [43] | ECIR
(9) | 2019 - 2020 [3] | Recall, Precision [3] | JAMIA
(10) | 2021 - 2022 [61] | F1-Score [61] | NAACL
(11) | 2022 - 2023 [88] | Precision, F1, F3, Recall [88] | SIGIR
(12) | 2022 - 2022 [22] | Recall, Precision, Macro F1, Accuracy [22] | MedPRAI

###  A.2 SLR datasets in biomedical benchmarks

Systematic literature reviews consist of multiple steps, and depending on the
granularity, previous studies enumerated between four and up to 15 tasks that
might be included in the SLR process [74]. High-level tasks include steps of
preparation, followed by the search and appraisal of primary studies and then
synthesis and write-up of the evidence. According to van Dinter et al. [76],
citation screening (selection of primary studies) was the step for which most
of the automation-related research was happening. Among other steps, the tasks
of query formulation, information extraction, risk of bias assessment, and,
more recently, text summarisation were also introduced.

Marshall et al. [50] introduced a large dataset with Cochrane reviews for the
task of assessing the risk of bias – a procedure aiming at establishing the
quality of input studies. Nye et al. [58] proposed a PICO (Population,
Intervention, Comparison and Outcome) extraction dataset containing 5,000
annotated abstracts of biomedical publications. In the query formulation,
often the models evaluate their performance on the CLEF TAR 2017-2018 datasets
[30, 31]. For the task of systematic review summarisation, a shared task was
introduced [84] consisting of two datasets: [82, 14].

In a comprehensive catalogue of medical artificial intelligence datasets and
benchmarks by Blagec et al. [8], only three citation screening datasets are
mentioned: Cohen et al. [11], Wallace et al. [79], and Miwa et al. [53]. Of
these three datasets, only two are publicly available, and both are already
implemented in CSMeD. Additionally, another five private SLRs used in only one
publication [71] are mentioned.

There is poor coverage of SLR datasets among biomedical benchmarks, especially
for the task of citation screening. None of the existing benchmarks contains
any publicly available citation screening dataset. Only the BoX [61] benchmark
uses five SLRs, but these datasets are private and cannot be obtained even
through a DUA (Data Use Agreement).

From other SLR automation tasks, BigBio [16] and BLURB [20] benchmarks contain
only one information extraction dataset by Nye et al. [58]. BLUE [63] and
CBLUE [94] benchmarks do not contain any SLR-related task. Therefore, there is
a clear need to develop and include publicly available SLR datasets in
biomedical benchmarks, particularly for citation screening tasks, to
facilitate further research and progress in this field.

The latest advances in Large language models (LLMs) offer significant
potential for aiding in SLR automation but simultaneously raise several
concerns. A user study by Yun et al. [92] mentions that SLR practitioners
acknowledged the potential utility of LLMs in various tasks, such as
generating the first draft of a review, writing plain language summaries, and
extracting information from longer texts. On the other hand, domain experts
have highlighted several crucial issues, including concerns about
hallucinations, the untraceable origins of generated content, and the
proliferation of bad-quality reviews.

##  Appendix B Visualisations

We leverage Streamlit101010https://streamlit.io to create interactive
visualisations for our meta-dataset. We present essential details for every
dataset, such as the number of training samples, character and word counts,
and labels and token lengths distribution across dataset splits (example on
Figure 2). We build upon the existing BigBio schemas and visualisations,
extending them to incorporate citation screening-specific details. We also
build a dedicated page to explore CSMeD-ft dataset containing full text
documents.

Figure 2: Example visualidation with statistics for a “Proton Pump Inhibitors”
SLR dataset.

We further focus on measuring the overlap between datasets. We check for the
overlap on the level of systematic reviews based on the review’s Cochrane ID.
This can help researchers understand potential biases, redundancy, or
complementary aspects across various datasets.

We use a TF-IDF-based document vectoriser with UMAP [52] to plot two-
dimensional representations of the datasets. This approach allows us to
effectively capture and display the structural patterns and similarities
within a single systematic literature review, aiding researchers in
identifying clusters, outliers, and potential data correlations. An example of
UMAP clustering of publications is presented in Figure 3.

Figure 3: Example visualisations with TF-IDF and UMAP representation of
documents for a “CS-Goulao-2016” SLR. Based on the plot, one can see that the
retrieved documents are grouped in two clusters with all relevant publications
belonging to one of them (bottom-right part of the plot). This can be an
indicator that any model will likely remove the other “non-relevant” cluster
of documents and hence achieve good score in detecting true negatives.

A live demo of the visualisation interface is available under the following
URL: https://systematic-review-datasets.streamlit.app/. Some features require
data preprocessing; they are unavailable in the demo but can be run locally
using the code from the GitHub repository.

##  Appendix C CSMeD data card

Dataset Description: CSMeD is a meta-dataset consisting of nine different
citation screening datasets containing 325 systematic literature reviews
(SLRs). Each systematic review consists of a list of publications that need to
be classified as either _relevant_ or _irrelevant_. All datasets have data
loader scripts providing programmatic access aligned with the BigBio framework
and HuggingFace datasets library. We preserve the original splits of the
datasets. We also generate data cards for every dataset which is part of the
CSMeD. CSMeD allows for accessing independent datasets and single systematic
reviews, which are part of each dataset.

train-cochrane split contains expanded metadata about systematic reviews such
as systematic review title, abstract, eligibility criteria and search
strategy. train-basic is a set of SLRs for which such meta-data was
unavailable and it is characterised by the systematic literature review title.
train-cochrane split is suitable for the tasks of question answering, natural
language inference, and text pair classification. train-basic is suitable only
for the text classification task.

Homepage: https://github.com/WojciechKusa/systematic-review-datasets

URL: https://github.com/WojciechKusa/systematic-review-datasets

Licensing: CC BY 4.0

Languages: English

Tasks: text classification (TXTCLASS), question answering (QA), natural
language inference (NLI), text pairs classification (PAIRS).

Schemas: Text (TEXT), Text pairs classification (PAIRS). Question Answering
(QA), source (source).

Splits: train-basic, train-cochrane, _all_

##  Appendix D CSMeD-ft

CSMeD-ft is an extension of the CSMeD meta-dataset that specifically focuses
on the full text screening step in SLRs. CSMeD-ft is to the best of our
knownledge, it is the first dataset explicitly targeted at the screening of
the full text of publication. While previously researchers already used full
text screening labels from other datasets to evaluate their models, the input
to these models constituted only the titles and abstracts of publications
[28].

###  D.1 Dataset construction details

To construct CSMeD-ft, we collected various elements of SLRs from the Cochrane
Library website, including the title, abstract and eligibility criteria
sections of the SLR and SLRs’ appendix and references. The appendix contains a
search strategy, while the references list papers categorised as: “studies
included in the review”, “studies excluded from the review”, and “additional
references”. We decided to focus solely on the “included” and “excluded”
categories as there is no definitive way to determine the intended meaning
when researchers added papers as additional references. However, in future
work, we plan to explore the possibility of extending the dataset to encompass
publications from the “additional references” category.

To obtain the full texts of references, we used the DOI (Digital Object
Identifier) of each publication. While some references directly provided the
DOI, for others, we initially attempted to match them to PubMed IDs and then
extracted the DOIs from PubMed and Semantic Scholar. To assign PubMed IDs to
the publications parsed from the Cochrane website, we followed a four-step
process:

  * •

We check if the PubMed ID information is provided on the Cochrane references
webpage.

  * •

We conduct search in PubMed using
Entrez111111https://www.ncbi.nlm.nih.gov/search/ by searching for the same
title and authors.

  * •

We search for the PubMed ID in SemanticScholar using publication DOI from
Cochrane references webpage.

  * •

We search again in PubMed, this time with a relaxed requirement by searching
for an exact match in the title only.

We then use the PubMed ID to resolve the DOI of the publication. We could
match the DOI for more than 61% of references.

We adopted a time-wise construction approach for CSMeD-ft canonical splits to
ensure the integrity and avoid data contamination. Therefore, we selected 29
SLRs not part of any previously released datasets to form our test set. We
used data from previous publications to construct a testing and development
set: dataset used by Nussbaumer-Streit et al. [56] for the development set and
dataset introduced by Scells et al. [69] for training split. It should be
noted that newer SLRs tend to have more comprehensive metadata and more open-
access full text publications available. This resulted in token length and
label frequency differences across the dataset splits (Figure 4). Despite
these variations, we decided to retain these splits as they present a more
realistic and challenging scenario, closely reflecting real-life
circumstances.

We have made the entire dataset construction procedure available in our
repository, enabling transparency and reproducibility.

###  D.2 CSMeD-ft Data Card

Figure 4: Token frequency distribution by split (top) and frequency of
different kind of instances (bottom).

Dataset Description The dataset focused on the task of full text screening for
systematic literature review creation. It contains 3,333 systematic literature
review and publication pairs with decisions if the publication was included in
the systematic literature review. Every excluded publication also contains a
textual explanation of why it was excluded. Systematic literature reviews are
formatted in a json format, whereas publications are stored as csv files.
CSMeD-ft-sample is a subset of CSMeD-ft-test dataset. We intend to store the
dataset on the TU Wien Research Data
repository,121212https://researchdata.tuwien.ac.at currently the dataset is
available on the project GitHub repository.

Homepage: https://github.com/WojciechKusa/systematic-review-datasets

URL: https://github.com/WojciechKusa/systematic-review-
datasets/raw/main/data/CSMeD/CSMeD-FT.zip

Licensing: CC BY 4.0

Languages: English

Tasks: text pairs classification, natural language entailment

Schemas: TEXT, PAIRS, source.

Splits: train, dev, test, sample

Dataset size (document pairs): train: 2,053, dev: 644, test: 636, sample: 50

Size of downloaded dataset files: 33.5 MB

Size of the generated dataset files: 112.2 MB

##  Appendix E Examining dataset overlap

We evaluate the overlap between datasets at the level of entire systematic
reviews. This analysis aims to understand the potential duplication of
information and data leakage across different datasets.

Table 8 presents the extent of overlap observed between the train and test
splits of the datasets. The TAR 2019 collection is most severely affected,
with 3 SLRs duplicated in its train and test splits. SLRs released as part of
the SIGIR 2017 collection [69] are also present among the test splits in CLEF
TAR 2017 and 2019 collections.

Table 8: List of overlapping Cochrane systematic literature reviews between datasets. Cochrane review ID | First collection | Other collections
---|---|---
CD011145 | sigir2017 (train) | tar2017 (test)
CD010633 | sigir2017 (train) | tar2017 (test), tar2018 (train), tar2019 (train)
CD010653 | sigir2017 (train) | tar2017 (test), tar2018 (train), tar2019 (train)
CD010542 | sigir2017 (train) | tar2017 (test), tar2018 (train), tar2019 (train)
CD009185 | sigir2017 (train) | tar2017 (test), tar2018 (train), tar2019 (train)
CD008081 | sigir2017 (train) | tar2017 (test), tar2018 (train), tar2019 (train)
CD002143 | sigir2017 (train) | sigir2017 (train)
CD001261 | sigir2017 (train) | tar2019 (test)
CD011571 | tar2019 (train) | tar2019 (test)
CD012164 | tar2019 (train) | tar2019 (test)
CD011686 | tar2019 (train) | tar2019 (test)

It is worth noting that we did not explicitly report the overlap between
different CLEF TAR datasets [30, 31, 32]. The owners of the dataset have
already acknowledged that each new edition of the dataset includes SLRs from
the previous editions as part of the training data. As the older datasets did
not share metadata about the considered reviews (except for the very high-
level title of the review (e.g. ADHD or COPD), we did not have access to the
mapping to the published reviews.

##  Appendix F Experimental setup

###  F.1 Transformer model fine-tuning

We select the following model checkpoints from HuggingFace Transformers
library:

  * •

Longformer-base – https://huggingface.co/allenai/longformer-base-4096

  * •

BigBird-roberta-base – https://huggingface.co/google/bigbird-roberta-base

  * •

Clinical-Longformer – https://huggingface.co/yikuan8/Clinical-Longformer

  * •

Clinical-BigBird – https://huggingface.co/yikuan8/Clinical-BigBird

We want to decide whether a publication fulfils all inclusion criteria and
none of the exclusion criteria to include it in the SLR. Specifically, this
means matching the eligibility criteria of SLR with the full text of the
candidate publication. As input, the model receives the text of the review and
publication and is asked to predict a binary category. We concatenate the
review title with the eligibility criteria section to create the review text.
For publications, we concatenate the title, abstract and the main text.

As available input text (review text + publication text) almost always exceeds
the available context window of considered models (4,096 tokens), we use the
following approach to allocate available space. We use the TokenTextSplitter
method from the langchain library131313https://github.com/hwchase17/langchain
with the gpt-3.5-turbo-0301 model to select the review text that would fit the
context window. We select at most half of the available context window, so in
the context of all Transformer models, review text equals at most 2,048
tokens. This action truncates some part of the eligibility criteria section,
i.e. for 13% of items in the trainset and 42% in the test set (Table 9). We
fill the remaining input sequence with the publication text.

Table 9: Statistics of a review text with respect to the fit within 2,048
tokens context window.

| CSMeD-ft-train | CSMeD-ft-dev | CSMeD-ft-test | CSMeD-ft-sample
---|---|---|---|---
Avg # splits | 1.13 | 1.24 | 1.83 | 1.74
Median # splits | 1 | 1 | 1 | 1
Max # splits | 2 | 2 | 4 | 4
Min # splits | 1 | 1 | 1 | 1
More than 1 splits | 13% | 24% | 42% | 42%

We run our experiments on a single server with 4 Nvidia RTX 3090 GPUs with
24GB of RAM each. We use a per-device batch size of 1 with eight gradient
accumulation steps. We test several learning rates with the best results for
1e-5, and we set weight decay to 0.01. We use AdamW [49] with default values
of $\beta_{1}=0.9$ and $\beta_{2}=0.999$. We evaluate models after each epoch
on the validation set and select the model with the highest macro f1-score.

One training epoch took around 30 minutes both for BigBird and Longformer-
based models. For inference, Longformer architecture processed, on average,
2.9 samples per second, whereas BigBird models 2.65 samples per second. Making
predictions on the entire test split of 636 documents took less than 4 minutes
for all models.

###  F.2 Zero-shot language model evaluation

Similarly, as for the fine-tuned classification models, we reserve at most
half of the context window size for the systematic literature review
description and fill the remaining tokens with the publication text. We
measure the text length using the OpenAI library
tiktoken141414https://github.com/openai/tiktoken, which provides tokenisers
for GPT-3.5 and GPT-4 models. We use the openai python library version 0.27.7,
and use the default chat completion function parameters of temperature = 1 and
top_p = 1.

We set our total budget to 50 USD and conduct the experiments only on the
CSMeD-ft-test-small subset for GPT-4 model. For the GPT-3.5-turbo-16k model,
making predictions on all 636 examples of the CSMeD-ft-test split took 44
minutes. However, this value was heavily influenced by the default OpenAI’s
rate limits of 180,000 tokens per minute for our organisation. We use the
following prompt template:

#### Input Template:

⬇

Does the following scientific paper fulfill all eligibility criteria and \

should it be included in the systematic review? \

Answer ‘Included’ or ‘Excluded’. \

Systematic review: "{{r.title}}" \n "{{r.criteria}}" \n\n \

Publication: "{{p.title}}" \n "{{p.abstract}}" \n "{{p.main_text}} \n\n \

Answer:

#### Output Template:

⬇

{{label}}

#### Answer Choices:

⬇

Included ||| Excluded