# The Promise and Challenges of Using LLMs to Accelerate the Screening Process
of Systematic Reviews

Aleksi Huotala<EMAIL_ADDRESS>University of HelsinkiDepartment of
Computer ScienceHelsinkiFinland ,  Miikka Kuutila<EMAIL_ADDRESS>Dalhousie UniversityFaculty of Computer ScienceHalifaxNova Scotia, Canada ,
Paul Ralph<EMAIL_ADDRESS>Dalhousie UniversityFaculty of Computer
ScienceHalifaxNova Scotia, Canada and  Mika Mäntylä
<EMAIL_ADDRESS>University of Helsinki / University of
OuluDepartment of Computer Science / M3SHelsinki / OuluFinland

(2024)

###### Abstract.

Context: Systematic review (SR) is a popular research method in software
engineering (SE). However, conducting an SR takes an average of 67 weeks.
Thus, automating any step of the SR process could reduce the effort associated
with SRs. Objective: Our objective is to investigate the extent to which Large
Language Models (LLMs) can accelerate title-abstract screening by (1)
simplifying abstracts for human screeners, and (2) automating title-abstract
screening entirely. Method: We performed an experiment where human screeners
performed title-abstract screening for 20 papers with both original and
simplified abstracts from a prior SR. The experiment with human screeners was
reproduced by instructing GPT-3.5 and GPT-4 LLMs to perform the same screening
tasks. We also studied whether different prompting techniques (Zero-shot (ZS),
One-shot (OS), Few-shot (FS), and Few-shot with Chain-of-Thought (FS-CoT)
prompting) improve the screening performance of LLMs. Lastly, we studied if
redesigning the prompt used in the LLM reproduction of title-abstract
screening leads to improved screening performance. Results: Text
simplification did not increase the screeners’ screening performance, but
reduced the time used in screening. Screeners’ scientific literacy skills and
researcher status predict screening performance. Some LLM and prompt
combinations perform as well as human screeners in the screening tasks. Our
results indicate that a more recent LLM (GPT-4) is better than its predecessor
LLM (GPT-3.5). Additionally, Few-shot and One-shot prompting outperforms Zero-
shot prompting. Conclusion: Using LLMs for text simplification in the
screening process does not significantly improve human performance. Using LLMs
to automate title-abstract screening seems promising, but current LLMs are not
significantly more accurate than human screeners. To recommend the use of LLMs
in the screening process of SRs, more research is needed. We recommend future
SR studies to publish replication packages with screening data to enable more
conclusive experimenting with LLM screening.

Screening Process of Systematic Reviews, LLMs, Text Simplification, ChatGPT,
GPT-3.5, GPT-4

††copyright: none††conference: 28th International Conference on Evaluation and
Assessment in Software Engineering; June 18–21, 2024; Salerno,
Italy††booktitle: 28th International Conference on Evaluation and Assessment
in Software Engineering (EASE 2024), June 18–21, 2024, Salerno,
Italy††copyright: none

##  1\. Introduction

Systematic review (SR) is a popular research method in software engineering;
since 1994, over a hundred SRs about software testing (Garousi and Mäntylä,
2016) have been published. By a recent estimate, conducting a SR takes about
67 weeks (Marshall and Wallace, 2019). A significant amount of time is spent
on manual labor, which prior studies have tried to automate (van Dinter et
al., 2021; Marshall and Wallace, 2019; Guo et al., 2024; Khraisha et al.,
2024; Bolanos et al., 2024; Wilkins, 2023; Robinson et al., 2023; Wang et al.,
2024). Our motivation for this study is to investigate whether the effort
required for SRs can be reduced by using Large Language Models (LLMs) in text
simplification or automating the whole screening process.

The SR process defines a screening process by which primary studies are
included or excluded (Kitchenham and Charters, 2007). This process usually has
two phases, the first using just the title and abstract; and the second using
the full text. Title-abstract screening is usually faster and done before
full-text screening (Kitchenham and Charters, 2007). The ability of LLMs to do
complex language tasks makes them good candidates for speeding up the
screening based on the titles and abstracts (Zhao et al., 2023). Similarly,
the ability of LLMs to do text simplification, which has been shown to
increase reading comprehension (Crossley et al., 2014), suggests their
potential for improving screening performance and reducing manual labor in the
screening process. We, therefore, pose the following research question:

Research Question: Can Large Language Models (LLMs) facilitate title-abstract
screening in systematic reviews (SRs)?

To address this question, we first perform a controlled experiment comparing
the performance of human screeners with original abstracts vs. LLM-simplified
abstracts. We hypothesize that text simplification will improve human
screening performance by making text more readily understandable. We then
evaluate LLM-automated screening by reproducing the screening process of a
prior SR using LLMs. We hypothesize that LLMs can do title-abstract screening
more accurately than humans and that applying prompt optimization improves
screening performance.

This paper is structured as follows. First, we introduce background
information in Section 2. Section 3 presents our hypotheses and experimental
methodology comparing human screeners with LLMs. Section 4 presents the
results in relation to the formulated hypotheses. The results are then
discussed together with threats to validity in Section 5 and limitations in
Section 6. Finally, Section 7 presents the conclusions of our study.

##  2\. Background

###  2.1. Systematic Review Procedures

SRs are secondary research. Primary studies are those in which the authors
generate or collect data. Secondary research aggregates the findings of
primary studies (Ralph and Baltes, 2022). Many types of SRs exist, and
protocols vary by type; however, SR protocols typically define:

  1. (1)

A search process that identifies primary studies.

  2. (2)

Selection criteria that define which primary studies should be included.

  3. (3)

A screening process by which selection criteria are applied to classify
primary studies as included or excluded (Kitchenham and Charters, 2007; Ralph
and Baltes, 2022)

Done well, screening involves two or more human screeners applying selection
criteria to (possibly thousands of) primary studies, meeting frequently to
resolve disagreements and improve their decision rules. To reduce effort,
screening is frequently divided into two stages:

  1. (1)

Title and abstract screening: judges read each paper’s title and abstract,
excluding those that unambiguously do not meet the screening criteria

  2. (2)

Full-text screening: judges review each paper’s complete text, excluding the
remaining papers that do not meet the screening criteria.

Overall, conducting SRs can be time- and labor-intensive (van Dinter et al.,
2021; Marshall and Wallace, 2019). Any technique that can exclude papers with
high accuracy may therefore significantly accelerate the screening process.

###  2.2. Text Simplification

Text simplification is used to improve the readability and understandability
of text while retaining its original meaning (Al-Thanyyan and Azmi, 2021).
Text simplification is often used with second language learners, children,
people with disabilities, and people with low literacy (Štajner, 2021). It
seems plausible that text simplification could improve abstract comprehension
by human screeners, especially for non-native English speakers.

###  2.3. Scientific Literacy

The ability to correctly screen primary studies depends on the screener’s
scientific literacy. Scientific literacy is the ability to “use evidence and
data to evaluate the quality of science information and arguments put forth by
scientists and in the media” (Gormally et al., 2012). Put another way, it is
“the capacity to use scientific knowledge to identify questions and to draw
evidence-based conclusions in order to understand and help make decisions
about the natural world and the changes made to it through human activity”
(Gormally et al., 2012). Scientific literacy skills thus concern the use of
data, evaluating the quality of scientific literature, making decisions, and
reaching conclusions. Scientific literacy can be evaluated using a
standardized test such as the Test Of Scientific Literacy Skills (TOSLS)
(Gormally et al., 2012).

###  2.4. Using Software to Help Systematic Reviews

Software should be able to automate or accelerate several aspects of SRs
including constructing search strings, retrieving PDFs, extracting data, and
automating screening. However, most existing studies focus on automating
primary study screening and acknowledge that the main goal of automation is to
reduce manual labor (van Dinter et al., 2021). Prior work on using software to
facilitate SRs focuses on reducing manual effort but stops short of automating
the whole review process. For example, a review of text mining techniques used
on SRs found that text mining techniques could reduce effort by 30–70%, but
led to a $\approx 5\%$ loss of relevant studies (O’Mara-Eves et al., 2015).

Marshall and Wallace (Marshall and Wallace, 2019) provide a practical guide
for using machine learning (ML) tools for literature search, screening, data
extraction, or bias assessment. Several abstract-screening tools are
available, of which Abstrackr and Research Screener tools reduce screening
workload by 9–89%, with precision ranging from 16% to 45% (Chai et al., 2021;
Gates et al., 2018; Rathbone et al., 2015). All of these tools require a human
in the loop, instructing the ML tool to identify relevant research papers.

Recent studies on using LLMs to automate the screening process in SRs have
shown that LLMs can match human performance in paper screening, with some
limitations (Guo et al., 2024; Khraisha et al., 2024; Bolanos et al., 2024;
Wilkins, 2023; Robinson et al., 2023; Wang et al., 2024). However, most of
these approaches have been studied in the domain of medical sciences, the
findings suggest that research on automating title-abstract screening could
also be applied to other domains, such as SE.

###  2.5. Prompt Optimization

Incorporating task-specific examples and specially crafted instructions into
LLM prompts has been shown to improve their performance in specific tasks (Wei
et al., 2023; Kojima et al., 2022). Zero-shot (ZS) prompting refers to
situations where no task-specific examples are provided. One-shot (OS) and
Few-shot (FS) prompting are typical prompting techniques in natural language
processing (NLP), providing task-specific examples (Kojima et al., 2022). OS
prompting incorporates one task example, while FS prompting includes multiple
examples, with the number of examples depending on the context size of the LLM
(Brown et al., 2020). Chain-of-Thought (CoT) prompting incorporates a “Let’s
think step by step” phrase at the end of the prompt, instructing the LLM to
decompose the task into smaller sub-tasks for more accurate task solving (Wei
et al., 2023). CoT prompting can be combined with FS prompting, which is known
as Few-shot with Chain-of-Thought (FS-CoT) prompting, to further improve task
accuracy (Wei et al., 2023).

##  3\. Methodology

Figure 1 provides a visual overview of our study. Our study consists of two
experiments related to title-abstract screening for SRs. In the first
experiment, we investigate the effect of abstract simplification by LLMs on
the performance of humans in title-abstract screening. In the second
experiment, we examine the efficacy of completely automated title-abstract
screening using different LLMs and prompting techniques. We refer to this
second experiment as the LLM reproduction experiment because it aims to
reproduce the first experiment but with screening decisions made by LLMs. We
provide a comprehensive replication package including data, source code, and
materials (see Data Availability).

[Diagram detailing the conducted experiments, datasets and actions.]Diagram
detailing the conducted experiments, datasets and actions.

Figure 1. Diagram detailing the conducted experiments, datasets and actions.

###  3.1. Hypotheses

For the purposes of this study, we formulated the following hypotheses:

  1. (H1)

Using LLMs to simplify abstracts improves human screening performance

  2. (H2)

LLMs screen articles better than human screeners

  3. (H3)

LLM selection and prompt optimization improve the screening performance

  4. (H4)

Redesigning the LLM reproduction prompt leads to improved screening
performance

Hypothesis (H1) concerns the experiment with human screeners. It was
formulated on the basis that a simplified abstract is easier to comprehend,
enhancing the screening performance of screeners.

Hypothesis (H2) suggests that the tested LLMs outperform humans in the
screening task. We believe that LLMs can potentially identify key concepts
better and also detect irrelevant information better than humans, leading to
improved screening performance.

Hypothesis (H3) suggests that LLM selection and prompt optimization improve
screening performance, compared to the LLM reproduction in (H2). As noted in
Section 2.5, prompt optimization has been shown to improve task accuracy in
other contexts. Thus as prior work supports prompt optimization to increase
task accuracy, we assert that prompt optimization can increase the LLM’s
performance in title-abstract screening. We also posit that LLM selection can
improve the performance in screening because newer LLMs are trained with more
data and contain improved architectures.

Finally, hypothesis (H4) suggests that redesigning the prompt, used in
hypothesis (H2) (the LLM reproduction), leads to improved screening
performance. The rationale for hypothesis (H4) is the usage of a prompt, that
1) describes the inclusion and exclusion criteria in more detail; and 2) is
formulated in a way that the LLM can make more accurate decisions when
including and excluding articles.

###  3.2. Abstract Simplification

We conducted an experiment to study the effects of text simplification,
scientific literacy, and academic experience on the screening performance of
human screeners.

From a pool of 5,414 papers originally screened for a prior SR (Kuutila et
al., 2020), we selected 10 papers that were eventually included and 10 that
were eventually excluded (top-left of Figure 1). We used our expertise to
purposefully select difficult-to-screen papers. All selected papers have time-
related terminology in their abstracts and mention empirical results related
to time pressure. In the excluded papers, time pressure is used as a
motivating factor for building something (e.g., testing framework), or time-
related terminology is used in relation to other concepts such as temporal
dispersion.

The inclusion criteria in the prior SR were: (1) “The main focus is time
pressure in software engineering“; and (2) “The paper presents empirical
evidence of time pressure in software engineering“. (Kuutila et al., 2020)

We selected papers from this particular SR because two of the authors of this
paper were co-authors, the subject matter is relatively easy for non-experts
to understand, and the search process included many synonyms.

Abstract simplification (top middle in Figure 1) was done using ChatGPT
version 3.5. The prompt used to simplify abstracts was:

> Simplify the following text so that a high school student can understand it,
> mention X on the simplified text and make the simplified text at least Y
> words long.

where X was the identified time-related term of each original abstract and Y
was a number slightly less than the number of words in the original abstract.
The prompt was used as many times as necessary to get an abstract that
included the time-related terms identified in the original abstract.

Table 1 presents the lengths of the original and simplified abstracts and two
readability metrics. The readability metrics were calculated using version
3.3.1 of the Quanteda library for the R programming language (Benoit et al.,
2018). The Flesch-Kincaid Grade level (FK) (Kincaid et al., 1975) and Simple
Measure of Gobbledygook (SMOG) (Mc Laughlin, 1969) are readability metrics
that estimate the US educational grade level (FK) or years of education (SMOG)
required to completely understand a text. That is, lower scores indicate
easier-to-read text. All selected papers and their original and simplified
abstracts can be found in the replication package (see Data Availability).

Table 1. Original vs. simplified abstracts in the experiment with human screeners. Means | Sample | Original | Simplified
---|---|---|---
Abstract length (words) | include | 228.9 | 196.6
Abstract length (words) | exclude | 181.4 | 176.7
Flesch-Kincaid level | include | 16.99 | 11.33
Flesch-Kincaid level | exclude | 15.43 | 10.79
SMOG | include | 17.48 | 12.78
SMOG | exclude | 16.18 | 12.50
Accuracy |  | 69% | 69%
Time per task (seconds) |  | 85.95 | 76.26
Confidence |  | 7.77 | 7.79

###  3.3. Title-Abstract Screening Experiment With Human Screeners

Table 2. Demographic information, test of scientific literacy (TOSLS) scores and screening performance of humans by group. | MSc Students | Researchers
---|---|---
Respondents | 13 | 16
Male | 11 | 8
Female | 2 | 8
Nationalities* | Finnish (8) | Finnish (8)
| Chinese (2) | Iran (2)
| Portuguese (2) | Others (8)
| Other (1) |
Min Age | 21 | 23
Max Age | 33 | 64
Mean Age | 26.2 | 39.8
Mean Academic Exp | 0 | 4.3
Mean Industry Exp | 1 | 6.5
Mean TOSLS | 19 | 18.9
% correct tasks | 0.64 | 0.73
Avg time for a task | 82.77s | 79.94s
Avg confidence for a task | 7.08 | 8.34

We recruited both master’s students and researchers at a Finnish university.
The students were enrolled in either a combined bachelor’s / master’s degree
program, or an international master’s degree program, both of which are
course-based — students are not expected or paid to produce publishable
research. In contrast, all of the researchers were paid to do research and
publish their results. The student screeners participated in the experiment
voluntarily as part of a course, where participation netted extra points for
completion. Researcher participants were recruited through social contacts and
mailing lists and were compensated with items of their choosing from a
university gift shop.

The experiment began with a presentation on what the screeners would be doing.
Participants were told they could ask questions about the experiment at any
time. Participants then signed a consent form and completed a demographic
questionnaire including age, nationality, gender, and highest academic
qualification (see Table 2). Next, participants had 35 minutes to complete the
TOSLS (see Section 2.3).

Each paper screening task consisted of an A4 paper, where the start time of
screening would be marked by hour, minute, and second. The reverse side
presented first the title and then the abstract of the paper. This was
followed by the inclusion and exclusion rules, where the participant’s choice
was crossed on a checkbox. After this, the confidence in the judgment was
asked with the single item “On a scale of 1 to 10, how confident are you that
your previous answer is correct?”. Lastly, the time the paper was completed
was marked at the bottom of the page. The invigilator demonstrated the filling
of the task sheet, first from the time started and then turning the paper, and
so on. The invigilator emphasized that there was no time limit for this task.
For time recording purposes, a digital clock with seconds was visible to
participants at all times during this part.

Each participant was given a set of four sets of five studies to screen:

  1. (1)

original abstracted / excluded from original study

  2. (2)

original abstracted / included in original study

  3. (3)

simplified abstracted / excluded from original study

  4. (4)

simplified abstracted / included in original study

The creation of the task sets is the last action before the H1 experiment in
Figure 1. Before creating the task sets, we divided the 40 possible tasks into
two creation sets with 20 tasks, where the conditions for each paper were
mirrored, that is the same paper had the original abstract in the first set
and the simplified one in the second. Both of these creation sets had 5 tasks
with the aforementioned 4 conditions. By doing this we ensured that each
condition for each paper got a similar amount of ratings. The task sheets were
then printed out and it was made so that the first four tasks were one of each
of the four conditions. Additionally, these were made so that roughly quarter
of the participants received each condition as their first task and so on, for
each of the first four tasks. After the fourth task, the remaining tasks were
shuffled.

At the end, there was a debriefing about the purposes of the study, where the
aims and methodology of the study were briefly explained, and the participants
could ask further questions if they so wished.

###  3.4. LLM Reproduction of Title-Abstract Screening

We reproduced the experiment from Section 3.3 using LLMs, as shown in Figure
1. We used the gpt-3.5-turbo-0613 and gpt-4-0613 chat model implementations,
as GPT-3.5 and GPT-4 are commonly used LLMs in SE-related research (Hou et
al., 2023). The corresponding LLM configurations are shown in Table 3. As the
prompts in this experiment fit in the context of both models, the 16K context
version (gpt-3.5-turbo-16k-0613) is unnecessary. We used a temperature value
of zero to minimize the non-deterministic behavior of the tested LLMs,
although some variation remains visible. The model cut-off date for all LLMs
was set to June 2023 and a seed parameter of 128 was used in this and
consequent experiments to make comparing the results more coherent and
reproducible.

The reproduction uses the same screening task set of 20 papers from the prior
SR (Kuutila et al., 2020), as in the experiment done with humans. We
constructed a prompt template following the screening procedure of the prior
SR as closely as possible. Using this template, we generated prompts using the
ZS prompting technique (see Section 2.5) to use in the reproduction. These
prompts were then processed with a Python script, which called the OpenAI API.
The prompt template, the generated prompts, the Python script, and the API
responses are provided in the replication package (see Data Availability).

Table 3. LLM configuration parameters.

Model | t | Seed | Context | Cut-off date
---|---|---|---|---
gpt-3.5-turbo-0613 | 0.0 | 128 | 4K | June 13th 2023
gpt-3.5-turbo-16k-0613 | 0.0 | 128 | 16K | June 13th 2023
gpt-4-turbo-0613 | 0.0 | 128 | 8K | June 13th 2023

###  3.5. LLM Reproduction With Prompt Optimization

The experiment for applying prompt optimization to the reproduction prompts
roughly follows the design of the simple reproduction experiment, as detailed
in Section 3.4. There are, however, some changes in the prompt generation
process, to accommodate the requirements of prompt optimization.

We selected 20 additional papers from the prior SR (Kuutila et al., 2020) as
“seed papers”, of which 10 should be included and 10 should be excluded. The
seed papers were used as examples with OS, FS, and FS-CoT prompting (see
Section 2.5).

For OS prompting, we generated 20 prompts for each of the 20 seed papers,
producing a table with 20 rows and 20 columns for each seed paper and their
corresponding screening result. For FS prompting, we used a subset of the seed
papers from the OS prompting. We specifically selected four seed papers that
performed the best in the OS prompting experiment; two each with the highest
$F_{1}$-scores from the included and excluded categories. We used seed papers
from the GPT-3.5 OS prompting experiment to maintain model specificity, and
likewise for GPT-4.

For FS-CoT prompting, we reused the FS prompts. The phrase “Let’s think step
by step” was appended to the end of each prompt. This phrase is used in Chain-
of-Thought (CoT) prompting and improves task accuracy (Wei et al., 2023;
Kojima et al., 2022).

###  3.6. Redesigning the Prompt for LLM Reproduction of Title-Abstract
Screening

The prompt redesign experiment closely follows the LLM reproduction and the
LLM reproduction with prompt optimization experiments from Sections 3.4 and
3.5, respectively.

To redesign the prompt, we first chose two title-abstract pairs to run our
evaluations against. We chose pairs that were challenging for both humans and
LLMs to evaluate. Second, we instructed ChatGPT (GPT-3.5) to optimize the
prompt using the prompt template from the LLM reproduction experiment. If the
optimization led to an incorrect screening decision, we ran the task again.
Once the evaluation was successful, we manually removed any information from
the prompt that was irrelevant for making the screening decision based on the
provided title-abstract pairs. However, after the removal of non-redundant
information, we observed that the screening decision was again incorrect. To
solve this, we appended “Even shallow empirical evidence should be
considered.” to the end of the inclusion criteria.

The redesigned prompt is included in the replication package (see Data
Availability), which shows that: 1) the list of synonyms was streamlined into
the first inclusion criterion to improve readability and broaden the scope of
aspects related to time pressure in software engineering; and 2) the second
inclusion criterion was made more inclusive, to allow even shallow empirical
evidence, as long as it is related to the impact of time-related factors on
software engineering. Together, these two modifications make the prompt
shorter, essentially easier to understand by humans and LLMs.

For the experiments incorporating ZS and OS prompting, the remaining test
setup is identical to the prompt optimization setup described in Section 3.5,
using the optimized prompt template (see Data Availability). Furthermore, we
did not test FS and FS-CoT prompting and decided to solely focus on
experimenting with ZS and OS prompting.

###  3.7. Answering TOSLS With LLMs

To answer the TOSLS from Section 3.3 with LLMs, a prompt using ZS prompting
was constructed (included in the replication package; see Data Availability).
The prompt includes the task description and the 28 questions of the TOSLS.
Nine of the questions contain background information that incorporates images.
Out of the nine images, six were manually converted to a textual
representation. For the remaining three images (questions 10, 21, and 28),
which are challenging for a human to describe textually, we used ChatGPT’s
(GPT-4) image interpretation functionality. The images were also described
textually so we could answer the whole TOSLS with LLMs that don’t support
embedding images. For question 10, we used the instruction “Describe the
following website for me.” to convert the website image to a textual
representation. For questions 21 and 28, the instruction used was “Describe
the graph below for me.”.

The experiment was conducted using the gpt-3.5-turbo-16k-0613 and gpt-4-0613
chat model implementations, as described in Table 3. The TOSLS in prompt form
has an approximate token count of 5700, so we have to use the 16K context
version of the gpt-3.5-turbo-0613 chat model. For the two LLMs, the TOSLS was
run 10 times in succession. We report the average result of the 10 test runs.

###  3.8. Reproducing a Larger Screening Procedure

We also conducted exploratory analysis on 1,361 Scopus Papers that were
initially gathered for the prior SR (Kuutila et al., 2020) to see how LLMs
perform with a more realistic sample size (see Figure 1). The Scopus Papers do
not encompass all the screened articles for the prior SR, as the screened
articles also included those from Google Scholar and articles resulting from
snowballing, but it is the data that had complete abstracts readily available
for analysis. Articles without abstracts or to which we did not have access
were removed, leaving us with 1,306 papers.

From the Scopus Papers, we extracted the papers’ corresponding titles and
abstracts. For the title-abstract pairs, we constructed prompts using the ZS
prompting technique (see Section 2.5) and then followed the same procedure
used for the LLM reproduction (Section 3.4).

###  3.9. Statistical Analysis

To analyze the data from all of the steps above, we used a variety of
statistical procedures. We checked for normality with the Shapiro-Wilk test.
The Shapiro-Wilk test was significant at p = 0.045 for variables of total
number of correct screening tasks by rater for original abstracts (value
between 0 and 10). Thus, we used the non-parametric paired Wilcoxon signed
rank test for hypothesis H1. The Shapiro-Wilk test was not significant for the
total number of correct screening tasks by researchers and students, with
p-values of 0.33 and 0.59 respectively. Thus we used the Independent samples
t-test when comparing the total number of correct screening tasks (value
between 0 and 20) between students and researchers for exploratory analysis.

For the Ordinary Least Squares (OLS) regression model presented in Table 6,
the predicted variable is the number of correct screening tasks from the
twenty given to the participant, and the predictor variables are the total
TOSLS score and a categorical variable on whether the screener was a
researcher or a student.

To compare human screener performance with LLMs for Hypothesis H2, we
generated a dataset where each LLM and prompt combination was presented with
29 sets of tasks similar to human screeners, thus producing 580 ratings. Thus
we had the same amount of data for human screeners and each LLM and prompt
combination. The sets also mirrored those given to human screeners in other
characteristics, that is each set contained 10 inclusions, 10 exclusions, 10
original abstracts, and 10 simplified abstracts.

This dataset was then used to fit the logistic regression model presented in
Table 10, where the predicted variable is the correctness of a single
judgment, and it is predicted by a categorical variable capturing who (human
or LLM & prompt combination) made the judgment. We also added control
variables to the model on whether the abstract was simplified or the original,
and whether the paper in the task was included or excluded in the original
paper. We chose human performance as the reference category because of the way
our hypothesis was formulated.

For testing hypotheses H3 and H4, we recategorized the categorical variables
from the LLM and prompt combination into two: the LLM used and the prompt
used. We also took out the judgments by human screeners as they are not
relevant to the hypotheses, and they would have a perfect correlation if used
for both LLM and prompt variables. The model is presented in Table 11. We
chose GPT-3.5 and ZS prompt as reference categories, as they are the most
basic versions which the subsequent versions try to improve on. We give Odd’s
ratio as an effect size measure for both logistic regression models and
$R^{2}$ for the OLS linear regression.

##  4\. Results

###  4.1. Text Simplification

####  4.1.1. H1: Using LLMs to simplify abstracts improves human screening
performance

The statistical analysis did not support hypothesis H1. Table 4 presents
Wilcoxon Signed-Rank tests comparing original abstracts with simplified
abstracts. Based on the results, it is evident that whether the abstract was
original or simplified did not improve the performance of the human screeners
in screening tasks or the confidence in their screening. However, screeners
spent less time screening simplified abstracts, as can be seen in Table 4. As
can be seen from Table 1, the difference between the means is 9.39 seconds
(76.26s vs 85.95s).

Table 4. Wilcoxon Signed-Rank Test (Paired) between correctness, time and confidence between original and simplified paper abstracts Comparison | p-value
---|---
Original v Simplified correctness | 0.61
Original v Simplified time | 0.0007***
Original v Simplified confidence | 0.75

####  4.1.2. Exploratory Analysis

Researchers exhibit an average correctness of 73% in paper screening, while
students screen the paper correctly 64% of the time. Based on the Independent
samples t-test, the difference between these groups is statistically
significant, as shown in Table 5. There was no meaningful difference in TOSLS
correctness between students and researchers, as shown in Table 2.

Linear regression model predicting the total number of correct judgments for
human screeners is shown in Table 6, utilizing student/researcher status and
TOSLS score as predictors. Our findings indicate that both a higher TOSLS
score and researcher status are predictive of an increased number of accurate
screening decisions. This model accounts for 33% of the variance in screening
decisions.

Figure 2 displays a Spearman correlation of 0.36 and a scatter plot between
the TOSLS score and the total number of correct screening tasks. It is
noteworthy that the regression line intersects the y-axis at nearly 50%
correctness, aligning with the expected outcome from random guessing in a
binary classification task. Furthermore, the regression model presented in
Table 6 indicates that the TOSLS score is a statistically significant positive
predictor of correct evaluations. Thus, scientific literacy, the TOSLS score
in particular, has a predictive ability on the correctness of the screening
tasks.

Table 5. Independent samples t-test between the total number of correct screening tasks by students and researchers. Comparison | t-value | p-value
---|---|---
Students v Researchers | 2.4227 | 0.02277
Table 6. OLS linear regression model predicting total number correct screening tasks by human screeners. Variable | coef | std.err | t | p-value | 95% CI
---|---|---|---|---|---
TOSLS | 0.17 | 0.06 | 2.48 | 0.02* | 0.03 0.29
Student | -1.81 | 0.71 | -2.56 | 0.02* | -3.27 -0.35
$R^{2}$:0.33 | F-Stat ¿0.001***

[Percentage of correct screening tasks and TOSLS score scatter plot with
Spearman correlation.]Percentage of correct screening tasks and TOSLS score
scatter plot with Spearman correlation.

Figure 2. Percentage of correct screening tasks and TOSLS score scatter plot
with Spearman correlation. Table 7. Human screener majority vote per paper.
Paper references are given in the replication package (Huotala et al., 2024).

Paper | A1 | A2 | A3 | A4 | A5 | A6 | A7 | A8 | A9 | A10 | A11 | A12 | A13 | A14 | A15 | A16 | A17 | A18 | A19 | A20
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
| Include count
---
Original
2 | 3 | 4 | 2 | 4 | 5 | 0 | 8 | 1 | 3 | 3 | 4 | 9 | 10 | 14 | 13 | 6 | 9 | 13 | 6
| Exclude count
---
Original
13 | 11 | 10 | 12 | 10 | 10 | 14 | 8 | 14 | 12 | 12 | 10 | 5 | 3 | 1 | 2 | 9 | 5 | 1 | 9
| Include count
---
Simplified
10 | 1 | 2 | 8 | 4 | 5 | 0 | 5 | 5 | 0 | 4 | 5 | 10 | 14 | 14 | 12 | 9 | 11 | 11 | 6
| Exclude count
---
Simplified
4 | 14 | 13 | 7 | 11 | 9 | 15 | 8 | 9 | 14 | 10 | 10 | 5 | 2 | 0 | 2 | 5 | 4 | 4 | 8
% Correct Orig | 86.6 | 78.6 | 71.4 | 85.7 | 71.4 | 66.6 | 100 | 50 | 93.3 | 80 | 20 | 28.6 | 64.3 | 76.9 | 93.3 | 86.6 | 40 | 64.3 | 92.8 | 40
% Correct Simp | 28.6 | 93.3 | 86.6 | 46.6 | 73.3 | 64.3 | 100 | 61.5 | 64.3 | 100 | 28.6 | 33.3 | 66.6 | 87.5 | 100 | 85.7 | 64.3 | 73.3 | 73.3 | 42.9
Total % Correct | 59 | 86 | 79 | 66 | 72 | 66 | 100 | 55 | 69 | 90 | 24 | 31 | 66 | 83 | 97 | 86 | 52 | 69 | 83 | 41
Judgment in Orig (Kuutila et al., 2020) | Exclude | Include

Table 8. Mean TOSLS scores of the two tested LLM models.

Name | Mean TOSLS | $\rho$ | p
---|---|---|---
Human screeners | 19.068 |  |
gpt-3.5-turbo-16k-0613 | 19.000 | 0.54115 | 0.10623
gpt-4-0613 | 28.000 | 0.89352 | 0.00049

###  4.2. LLM Screening

Table 9. Results of the title-abstract screening of the reproduction of larger screening procedure, using Scopus data set. Correct (%) Model | Exclude | Include | Precision | Recall
---|---|---|---|---
gpt-3.5-turbo-0613 | 95.25% | 65.00% | 0.65 | 0.17567
gpt-4-0613 | 98.91% | 50.00% | 0.5 | 0.41667
Table 10. Logistic regression model predicting the correctness of a single
judgment with the combination of LLM and prompting technique. Reference
category is the category the following categories are compared to.

LLM & prompt | coef | std.err | t | p-value |  | Odd’s
---
ratio
Reference: | Human performance
GPT3.5 - FS | -0.56 | 0.12 | -4.56 | ¡0.001 | 0.57
GPT3.5 - FS-CoT | -0.48 | 0.12 | -3.86 | ¡0.001 | 0.62
GPT3.5 - OS | -0.42 | 0.12 | -3.38 | ¡0.001 | 0.66
GPT3.5 - OSv2 | -0.43 | 0.12 | -3.44 | ¡0.001 | 0.65
GPT3.5 - ZS | -0.91 | 0.12 | -7.36 | ¡0.001 | 0.40
GPT3.5 - ZSv2 | -0.26 | 0.13 | -2.12 | 0.03 | 0.77
GPT4 - FS | 0.02 | 0.13 | 0.19 | 0.85 | 1.02
GPT4 - FS-CoT | 0.13 | 0.13 | 1.03 | 0.30 | 1.14
GPT4 - OS | 0.02 | 0.13 | 0.13 | 0.89 | 1.02
GPT4 - OSv2 | -0.63 | 0.12 | -5.08 | ¡0.001 | 0.53
GPT4 - ZS | -0.20 | 0.13 | -1.63 | 0.10 | 0.82
GPT4 - ZSv2 | -0.31 | 0.12 | -2.48 | 0.01 | 0.73
Reference: | Exclude
Include | 0.40 | 0.05 | 8.39 | ¡0.001 | 1.50
Reference: | Original abs
Simplified abstract | -0.08 | 0.06 | -1.61 | 0.11 | 0.93

FS = Few-shot, FS-CoT = Few-shot with Chain-of-Thought, OS = One-shot, OSv2 =
redesigned One-shot, ZS = Zero-shot, ZSv2 = redesigned Zero-shot. Refer to
Section 2.5 for more in-depth explanations about the prompting techniques.

Table 11. Logistic regression model predicting the correctness of a single
judgment with the LLM and prompting technique as their own variables.
Reference category is the category the following categories are compared to.

Variable | coef | std.err | t | p-value |  | Odd’s
---
ratio
Reference: LLM - GPT-3.5 |  |  |  |  |
LLM - GPT-4 | 0.34 | 0.05 | 6.90 | ¡0.001 | 1.41
Reference: Prompt - ZS |  |  |  |  |
Prompt - FS | 0.29 | 0.09 | 3.33 | ¡0.001 | 1.33
Prompt - FS-CoT | 0.38 | 0.09 | 4.41 | ¡0.001 | 1.46
Prompt - OS | 0.36 | 0.09 | 4.15 | ¡0.001 | 1.43
Prompt - OSv2 | 0.04 | 0.08 | 0.42 | 0.67 | 1.03
Prompt - ZSv2 | 0.28 | 0.09 | 3.24 | 0.0012 | 1.32
Reference: Exclude |  |  |  |  |
Include | 0.47 | 0.05 | 9.49 | ¡0.001 | 1.61
Reference: Original abs |  |  |  |  |
Simplified abstract | -0.08 | 0.05 | -1.64 | 0.10 | 0.92

FS = Few-shot, FS-CoT = Few-shot with Chain-of-Thought, OS = One-shot, OSv2 =
redesigned One-shot, ZS = Zero-shot, ZSv2 = redesigned Zero-shot. Refer to
Section 2.5 for more in-depth explanations about the prompting techniques.

####  4.2.1. H2: LLMs screen articles better than human screeners

As can be seen in Table 10, none of the LLM and prompt combinations have both
a positive coefficient and a statistically significant p-value. Thus
hypothesis H2 is rejected. However, there are several LLM and prompt
combinations that perform roughly as well as human screeners, specifically
these are One-shot (OS), Few-shot (FS) and Few-shot with Chain-of-Thought (FS-
CoT) prompts with the GPT-4 LLM.

The worst performance is by the Zero-shot (ZS) prompt for GPT-3.5 LLM. This
not too surprising, as the zero shot prompt is the most rudimentary of the
prompts. The next worst combinations in order are OSv2 for GPT-4, and FS, FS-
CoT, OSv2, OS for GPT-3.5. Lastly the ZSv2 for both GPT-3.5 and GPT-4 both
perform also worse than humans, but not as significantly.

####  4.2.2. H3: LLM selection and prompt optimization improve the screening
performance

The logistic regression model in Table 11 shows that using prompt optimization
improves performance over ZS prompts. Table also shows that GPT-4 is superior
to GPT-3.5 Thus, based on these results, we accept hypothesis H3. All prompt
optimization techniques, that is OS, FS, and FS-CoT prompting, improve
screening performance over ZS prompting, with odds ratios of 1.33 to 1.46.
This means that all of these prompts are between 33% to 46% more likely to
make a correct judgment on a single task. GPT-4 is 41% more likely to make a
correct judgment than GPT-3.5.

####  4.2.3. H4: Redesigning the LLM reproduction prompt leads to improved
screening performance

The regression model in Table 11 shows that redesigning the prompt (marked
with v2 text in the table) improves screening performance for ZS prompting,
having a 32% increased chance of making a correct judgment on a single task.
However, OS prompting with the redesigned prompt shows no statistical
significance. Based on these observations, we reject hypothesis H4.

###  4.3. Exploratory Analysis

We did an exploratory analysis on conducting the TOSLS with LLMs and
reproducing the title-abstract screening on a larger screening procedure.
Table 8 shows the mean TOSLS score for humans and both tested LLMs and Table 9
presents the results of the larger title-abstract screening procedure.

In the TOSLS experiment, the mean score of humans is very close to the score
of gpt-3.5-turbo-16k-0613, but gpt-4-0613 scored full points from the TOSLS.
For the Scopus paper screening experiment, both GPT-3.5 and GPT-4 classify
correctly over 95% of excluded papers. However, this comes with a significant
tradeoff, missing 35 to 50% of included papers for GPT-3.5 and GPT-4,
respectively. Therefore, using Zero-shot prompts for the whole screening
procedure cannot be recommended.

In the TOSLS experiment involving LLMs, we suspect that GPT-4 may have been
trained with the TOSLS research paper, so it might be aware of the test
questions and their answers. Unfortunately, as OpenAI does not disclose
specifics about the training data for their models, we cannot be certain. The
mean TOSLS for students was 19 and for researchers 18.9, which means GPT-3.5
performed as well as students and researchers. The skill scores of GPT-3.5 are
weakly correlated to human scores, meanwhile, GPT-4 indicates a very strong
correlation. If GPT-4 did not know the correct answers to the TOSLS, the full
points from the test would indicate significantly improved reasoning
abilities.

##  5\. Discussion

The experiment with the human screeners revealed no significant benefits on
accuracy when using summarized abstracts with GPT-3.5. However, simplified
abstracts resulted in significantly faster completion of screening judgments.
On average, the screening task for simplified abstracts was made in 9.39
seconds, or around 12% faster. This demonstrates that our text simplification
procedure had a real and measurable impact. However, considering that in a
real-life use case, researchers would have to generate the simplified versions
of the abstracts themselves, the actualized benefits would be limited.
However, even if this benefit was significant enough, the results in Table 7
call for caution, as it seems that the simplified text was much more likely
included erroneously in some cases (see papers A1, A4 and A9).

Paid researchers outperformed students in the screening task. This result is
partly in conflict with the results of the experiment conducted by Salman et
al. (Salman et al., 2015), where it was concluded that student performance was
similar to professional performance in a programming task when the approach
used was novel for both groups. In our case, the students had been taught
about conducting secondary studies during the course they were attending, thus
it was not completely novel for them, we did not record the experience of the
researchers in performing them. Nevertheless, a noticeable result from the
experiment with human screeners is that being a researcher and having higher
scientific literacy skills, measured with the TOSLS, has a predictive ability
on the number of correct judgments according to our regression model.

In the second experiment (H2), neither GPT-3.5 nor GPT-4 screened papers
better than humans. However, GPT-4 performed significantly better than
GPT-3.5, and we expect this performance improvement to continue with newer
versions of LLMs, as the amount of training data increases and architectures
are improved.

Out of the tested prompting techniques, we observed that FS-CoT prompting
performed the best when used with the GPT-4 LLM. Additionally, we observed
that using seed papers included in the prior SR resulted in increased
performance, that is prompts incorporating OS or FS prompting outperformed ZS
prompts. In our experiment setup, we used an additional set of 20 papers as
seed papers for the prompts and picked the best-performing seed papers. Thus,
for increased performance of these prompts are to be actualized in conducting
actual SRs, the authors have to have a set of screened papers so that best
performing seed papers can be evaluated. Thus prompt optimization would be
easier to use when conducting SRs on topics with prior SRs, as their screening
results could be used for One-shot (OS) and Few-shot (FS) prompting. This
underscores the importance of sharing the screening results in published SRs.

The results for redesigning the prompt were mixed at best. The ZSv2 prompt
outperformed ZS with both GPT-3.5 and GPT-4 LLMs, but the OSv2 prompt did not
outperform OS with either LLMs. It is worth noting that the redesigned prompts
(ZSv2, OSv2) were generally outperformed by prompt optimization techniques
(OS, FS, FS-CoT). Thus it seems that prompt redesign techniques have limited
performance increase in the screening task, and concentrating on prompt
optimization is more warranted. Future work could still examine if complex
prompting optimization strategies, such as FS and FS-CoT, could perform better
together with the redesigned prompt.

Further work is needed to analyze prompt optimization methods on larger
datasets. Our ZS prompt, which was fairly simple, already achieved a 98.91%
accuracy in excluding papers correctly in the exploratory analysis. It would
make sense to sacrifice some of this accuracy to increase inclusion accuracy,
as the needed manual labor for screening the included papers is already
significantly reduced if a significant amount of papers can be automatically
excluded. Thus, investigating how to safely integrate the usage of LLMs in the
screening process while keeping integrity of the existing guidelines for SRs
is called for.

The two existing ML tools, Abstrackr and Research Screener (covered in Section
2.4) had a screening accuracy between 16% and 45% (Chai et al., 2021; Gates et
al., 2018; Rathbone et al., 2015). Compared to our exploratory testing on a
larger screening procedure, in Table 9, our screening performance seems
significantly higher than with Abstrackr and Research Screener. However, the
results are not directly comparable, as the datasets used were different, and
Abstrackr gives recommendations instead of judgments. Also of note is that we
picked 20 screening tasks for our experiments, and the performance for these
particularly difficult cases is between 50% and 75% with a recall between 0.3
to 0.9. Precision and recall for each LLM and prompt combination is given in
the replication package (Huotala et al., 2024). The recent studies on LLMs for
automating the screening process of SRs (Guo et al., 2024; Khraisha et al.,
2024; Bolanos et al., 2024; Wilkins, 2023; Robinson et al., 2023; Wang et al.,
2024) show that the accuracy of title-abstract screening with LLMs have varied
between 0.60 and 0.835, which are similar to our results. Similar to the
studies done with Abstrackr and Research Screener, our results are not
directly comparable with the recent studies on LLMs, because of the datasets
that focus on different aspects. Additionally, not all studies mention the
used LLM versions, seed or temperature parameters making comparisons harder.

##  6\. Limitations

One concern of validity is the accuracy of the original study’s judgments
regarding screening. It is worth noting that the majority vote by the human
screeners in determining screening judgment aligns with the original study 17
out of 20 times. Yet, this was exceeded by the three best individuals that
made correct decisions 19, 18 and 18 times respectively. The three instances
where there is a discrepancy between the original study and the majority vote,
the human screeners of this study exclude studies, even though they were
included in the original review paper. These abstracts were particularly
challenging in judgment, but we maintain that each should be included,
following the screening criteria of the original study.

Another concern is the generalizability of our results. We selected
particularly difficult papers to screen for our experiment, where as the whole
screening procedure for a systematic review can involve very straightforward
decisions depending on the criteria. We reckon that both human screeners and
LLMs would have had better performance with a complete set of papers from an
existing study.

Using a digital clock could have added pressure for the participants to
complete the experiment faster. However, the researcher overseeing the
experiment emphasized to the participants that the clock was there for
recording the times of the individual screening tasks, and there was no need
to hurry to complete the tasks.

In our experiments with LLMs, we used date-fixed versions of GPT-3.5 and GPT-4
LLMs. It is evident that these models will get outdated in the future, but we
wanted to make sure the experiments can be exactly replicated, if necessary.
Another limitation regarding LLMs is the exclusive use of GPT-3.5 and GPT-4 as
the LLMs in our experiments. The choice was intentional to focus on the most
commonly used LLMs in SE-related research. However, the emergence of new LLMs,
like Google Gemini 1.5 and Anthropic Claude 3 pave possibilities for future
research on additional LLMs.

As conducting SRs includes more than just paper screening, another avenue for
future work is considering the use of LLMs in more phases. This includes the
identification of studies for the screening (Wohlin et al., 2022), data
extraction and analysis.

##  7\. Conclusions

In this paper, we studied the effects of text simplification, scientific
literacy, and academic experience in the screening accuracy of human screeners
for an SR and the accuracy of LLMs in reproducing the screening task of a
prior SR. We also studied if applying prompt optimization in the LLM
reproduction of title-abstract screening improves screening performance,
compared to the base case.

Our findings indicate that text simplification does not affect the number of
correct screening decisions, but it does reduce the time spent on paper
screening. Additionally, our linear regression model reveals that higher
scientific literacy skills and being a researcher (as opposed to a student)
both contribute to increased accuracy in screening results.

None of the LLM and prompt combinations outperformed human screeners in the
reproduction experiment. Nevertheless, Few-shot, Few-shot Chain-of-Thought
(CoT), and One-shot prompts together with GPT-4 LLM performed roughly as well
as the human screeners. This shows that there is promise in automating title-
abstract screening with LLMs. However, we note that the comparison here is to
a sample consisting of master’s level students and researchers who were not
experts on the specific subject. The reproduction experiment also shows that
prompt optimization techniques improve the screening performance, whereas the
results with prompt redesign were mixed. Thus based on our results the use of
prompt optimization is recommended. Still, further investigation on
integrating the LLM automated title-abstract screening to the SR process is
needed before it can be recommended for practical application.

## Data Availability

The datasets and source code for the experiments are available at Zenodo
(Huotala et al., 2024). However, the data involving human screeners is not
shared due to the absence of explicit consent for distribution.

###### Acknowledgements.

The first author has been funded by the Strategic Research Council of Research
Council of Finland (Grant ID 358471) and the second author has been funded by
the Killam Postdoctoral Fellowship.

## References

  * (1)
  * Al-Thanyyan and Azmi (2021) Suha S Al-Thanyyan and Aqil M Azmi. 2021.  Automated text simplification: a survey.  _ACM Computing Surveys (CSUR)_ 54, 2 (2021), 1–36.
  * Benoit et al. (2018) Kenneth Benoit, Kohei Watanabe, Haiyan Wang, Paul Nulty, Adam Obeng, Stefan Müller, and Akitaka Matsuo. 2018.  quanteda: An R package for the quantitative analysis of textual data.  _Journal of Open Source Software_ 3, 30 (2018), 774.  https://doi.org/10.21105/joss.00774
  * Bolanos et al. (2024) Francisco Bolanos, Angelo Salatino, Francesco Osborne, and Enrico Motta. 2024.  Artificial Intelligence for Literature Reviews: Opportunities and Challenges.  https://doi.org/10.48550/arXiv.2402.08565 arXiv:2402.08565 [cs]
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Chai et al. (2021) Kevin E. K. Chai, Robin L. J. Lines, Daniel F. Gucciardi, and Leo Ng. 2021.  Research Screener: A Machine Learning Tool to Semi-Automate Abstract Screening for Systematic Reviews.  _Systematic Reviews_ 10, 1 (April 2021), 93.  https://doi.org/10.1186/s13643-021-01635-3
  * Crossley et al. (2014) Scott A Crossley, Hae Sung Yang, and Danielle S McNamara. 2014.  What’s so Simple about Simplified Texts? A Computational and Psycholinguistic Investigation of Text Comprehension and Text Processing.  _Reading in a Foreign Language_ 26, 1 (2014), 92–113.
  * Garousi and Mäntylä (2016) Vahid Garousi and Mika V. Mäntylä. 2016.  A systematic literature review of literature reviews in software testing.  _Information and Software Technology_ 80 (2016), 195–216.  https://doi.org/10.1016/j.infsof.2016.09.002
  * Gates et al. (2018) Allison Gates, Cydney Johnson, and Lisa Hartling. 2018.  Technology-Assisted Title and Abstract Screening for Systematic Reviews: A Retrospective Evaluation of the Abstrackr Machine Learning Tool.  _Systematic Reviews_ 7, 1 (March 2018), 45.  https://doi.org/10.1186/s13643-018-0707-8
  * Gormally et al. (2012) Cara Gormally, Peggy Brickman, and Mary Lutz. 2012.  Developing a test of scientific literacy skills (TOSLS): Measuring undergraduates’ evaluation of scientific information and arguments.  _CBE—Life Sciences Education_ 11, 4 (2012), 364–377.
  * Guo et al. (2024) Eddie Guo, Mehul Gupta, Jiawen Deng, Ye-Jean Park, Michael Paget, and Christopher Naugler. 2024.  Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study.  _Journal of Medical Internet Research_ 26, 1 (Jan. 2024), e48996.  https://doi.org/10.2196/48996
  * Hou et al. (2023) Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2023.  Large Language Models for Software Engineering: A Systematic Literature Review.  https://doi.org/10.48550/arXiv.2308.10620 arXiv:2308.10620 [cs]
  * Huotala et al. (2024) Aleksi Huotala, Miikka Kuutila, Paul Ralph, and Mika Mäntylä. 2024.  Dataset for paper: The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews.  https://doi.org/10.5281/zenodo.11028876
  * Khraisha et al. (2024) Qusai Khraisha, Sophie Put, Johanna Kappenberg, Azza Warraitch, and Kristin Hadfield. 2024.  Can Large Language Models Replace Humans in Systematic Reviews? Evaluating GPT-4’s Efficacy in Screening and Extracting Data from Peer-Reviewed and Grey Literature in Multiple Languages.  _Research Synthesis Methods_ (March 2024).  https://doi.org/10.1002/jrsm.1715
  * Kincaid et al. (1975) J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975.  _Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel_.  Technical Report.
  * Kitchenham and Charters (2007) Barbara Kitchenham and Stuart Charters. 2007.  Guidelines for performing systematic literature reviews in software engineering.  _Technical Report EBSE-2007-01_ (2007).
  * Kojima et al. (2022) Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large Language Models are Zero-Shot Reasoners. In _Advances in Neural Information Processing Systems_ , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 22199–22213.  https://proceedings.neurips.cc/paper_files/paper/2022/file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf
  * Kuutila et al. (2020) Miikka Kuutila, Mika Mäntylä, Umar Farooq, and Maëlick Claes. 2020.  Time Pressure in Software Engineering: A Systematic Review.  _Information and Software Technology_ 121 (May 2020), 106257.  https://doi.org/10.1016/j.infsof.2020.106257
  * Marshall and Wallace (2019) Iain J. Marshall and Byron C. Wallace. 2019.  Toward systematic review automation: a practical guide to using machine learning tools in research synthesis.  _Systematic Reviews_ 8, 1 (July 2019), 163.  https://doi.org/10.1186/s13643-019-1074-9
  * Mc Laughlin (1969) G Harry Mc Laughlin. 1969.  SMOG grading-a new readability formula.  _Journal of reading_ 12, 8 (1969), 639–646.
  * O’Mara-Eves et al. (2015) Alison O’Mara-Eves, James Thomas, John McNaught, Makoto Miwa, and Sophia Ananiadou. 2015.  Using text mining for study identification in systematic reviews: a systematic review of current approaches.  _Systematic reviews_ 4, 1 (2015), 1–22.
  * Ralph and Baltes (2022) Paul Ralph and Sebastian Baltes. 2022.  Paving the way for mature secondary research: the seven types of literature review. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. ACM, New York, NY, 1632–1636.
  * Rathbone et al. (2015) John Rathbone, Tammy Hoffmann, and Paul Glasziou. 2015.  Faster Title and Abstract Screening? Evaluating Abstrackr, a Semi-Automated Online Screening Program for Systematic Reviewers.  _Systematic Reviews_ 4, 1 (June 2015), 80.  https://doi.org/10.1186/s13643-015-0067-6
  * Robinson et al. (2023) Ambrose Robinson, William Thorne, Ben P. Wu, Abdullah Pandor, Munira Essat, Mark Stevenson, and Xingyi Song. 2023.  Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation.  https://doi.org/10.48550/arXiv.2308.06610 arXiv:2308.06610 [cs]
  * Salman et al. (2015) Iflaah Salman, Ayse Tosun Misirli, and Natalia Juristo. 2015.  Are students representatives of professionals in software engineering experiments?. In _Proceedings of the 37th International Conference on Software Engineering-Volume 1_. 666–676.
  * Štajner (2021) Sanja Štajner. 2021.  Automatic text simplification for social good: Progress and challenges.  _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ (2021), 2637–2652.
  * van Dinter et al. (2021) Raymon van Dinter, Bedir Tekinerdogan, and Cagatay Catal. 2021.  Automation of systematic literature reviews: A systematic literature review.  _Information and Software Technology_ 136 (Aug. 2021), 106589.  https://doi.org/10.1016/j.infsof.2021.106589
  * Wang et al. (2024) Shuai Wang, Harrisen Scells, Shengyao Zhuang, Martin Potthast, Bevan Koopman, and Guido Zuccon. 2024.  Zero-Shot Generative Large Language Models for Systematic Review Screening Automation.  https://doi.org/10.48550/arXiv.2401.06320 arXiv:2401.06320 [cs]
  * Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023.  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.  arXiv:2201.11903 [cs]
  * Wilkins (2023) David Wilkins. 2023.  Automated Title and Abstract Screening for Scoping Reviews Using the GPT-4 Large Language Model.  https://doi.org/10.48550/arxiv.2311.07918 arXiv:2311.07918 [cs]
  * Wohlin et al. (2022) Claes Wohlin, Marcos Kalinowski, Katia Romero Felizardo, and Emilia Mendes. 2022.  Successful combination of database search and snowballing for identification of primary studies in systematic literature studies.  _Information and Software Technology_ 147 (2022), 106908.  https://doi.org/10.1016/j.infsof.2022.106908
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023.  A Survey of Large Language Models.  https://doi.org/10.48550/arXiv.2303.18223 arXiv:2303.18223 [cs]


# Real-Time Summarization of Twitter

Yixin Jin  University of Michigan, Ann Arbor
Ann Arbor, MI, 48109, USA
<EMAIL_ADDRESS>Meiqi Wang  Brandeis University
Waltham, MA, 02453, USA
<EMAIL_ADDRESS>Meng Li  Columbia University
New York, NY, 10027, USA
<EMAIL_ADDRESS>Wenjing Zhou  University of Michigan, Ann Arbor
Ann Arbor, MI, 48109, USA
<EMAIL_ADDRESS>Yi Shen  University of Michigan, Ann Arbor
Ann Arbor, MI, 48109, USA
<EMAIL_ADDRESS>Hao Liu  Northeastern University
Shenyang, China
<EMAIL_ADDRESS>
###### Abstract

In this paper, we describe our approaches to TREC Real-Time Summarization of
Twitter. We focus on real time push notification scenario, which requires a
system monitors the stream of sampled tweets and returns the tweets relevant
and novel to given interest profiles. Dirichlet score with and with very
little smoothing (baseline) are employed to classify whether a tweet is
relevant to a given interest profile. Using metrics including Mean Average
Precision (MAP, cumulative gain (CG) and discount cumulative gain (DCG), the
experiment indicates that our approach has a good performance. It is also
desired to remove the redundant tweets from the pushing queue. Due to the
precision limit, we only describe the algorithm in this paper.

###### Index Terms:

Real-time, Twitter, Dirichlet Prior, Redundancy Removal

##  I Introduction

With the explosion of Internet and microblog, the information that one can
access is enormous and is expected to grow faster and faster. In this
circumstance, the need of information filtering is increasing [7]. For
example, a stock trader might want to know about the news or events about
companies he has stock with and wish to receive notifications whenever
something new happened for those companies. Other than the fact-checking
studies[5, 9, 11, 20, 22, 24, 25] we might need in these cases, this
information need is also considered in TREC Real-Time Summarization (RST)
Track

There are two different scenarios described in Real-Time Summarization (RST)
Track [13]. One is Push Notification: Given user’s interest profile and
sampled tweets stream, tweets that are identified as relevant and novel be
pushed to the user in a timely manner. The other one is Email Digest:
Everyday, tweets with top relevance that day are digested in one email and
pushed to the user.

We focus on the first scenario (push notification) in our project. To fulfill
such kind of information need, our goal is to design a filtering system which
is able to monitor the stream of tweets and find the tweets relevant to a
given interest profile. On the other hand, it is desired to remove the
redundant tweets. Some relevant tweets may contain the exact same information,
and we want to keep only one tweet of those in our push queue according to the
time order [14]. Under the light of above discussion, our filtering system is
designed to consist of two parts: classification of relevance and remove of
redundancy. Given a interest profile, we will first convert it into a query.
When a new tweet comes in, we will first calculate its similarity score with
the query and classify it as relevant tweet or irrelevant tweet. Then if this
new tweet is classified as a relevant tweet, we will compare it with relevant
tweets recovered before, and classify it as novel tweet or redundant tweet.
And we shall only keep novel tweets and push them to the users.

##  II Materials and Methods

###  II-A Related Work

####  II-A1 Smoothing in Language Model

Tweets are usually short (140 words limit), as a result the language model [8]
estimated by maximum likelihood is limited. We need to assign positive
probabilities for some words not observed. So we need smoothing. Zhai[6]
suggests smoothing plays two roles in the ranking function: avoiding assigning
zero probabilities to words that have not occurred in a document and
accommodating generation of common words in a query. The probabilities of
unseen words are usually assigned by a reference model.

Using Dirichlet Prior, the document language model
$p(w|d)=\frac{|d|}{|d|+\mu}\frac{c(w,d)}{|d|}+\frac{\mu}{|d|+\mu}p(w|REF)$
where $|d|$ is the length of the document, $\mu$ is a smoothing parameter,
$\frac{c(w,d)}{|d|}$ is the language model estimated by maximum likelihood and
$p(w|REF)$ is the reference language model.

Another smoothing method is linear interpolation, or Jelinek-Mercer smoothing.
$p(w|d)=(1-\lambda)\frac{c(w,d)}{|d|}+\lambda p(w|REF)$.

Comparing with the baseline with little smoothing, our results also show the
importance of smoothing.

####  II-A2 Clustering

To remove the redundant tweets, clustering algorithms are needed. Clustering
algorithms are unsupervised. Common clustering methods are k-means and
agglomerative hierarchical clustering [18, 29].

K-means minimizes the within class scatter

$W(C)=\sum_{l=1}^{k}\sum_{i:C(i)=l}\|x_{i}-\bar{x_{l}}\|^{2}$ (1)

where k is the number of clusters,
$C:\\{1,2,\ldots,n\\}\rightarrow\\{1,2,\ldots,k\\}$ is a function mapping data
points to clusters, $x_{i}$ is the data point and
$\bar{x_{l}}=\frac{1}{n_{l}}\sum_{j:C(j)=l}x_{j}$ is the center of the l-th
cluster, where $n_{l}$ is the number of data points in the l-th cluster. This
is a combinatorial optimization problem, so we optimize the function in an
iterative way. We start with $k$ randomly selected data points and assume they
are the centroids of the clusters. Then we assign every data point to a
cluster whose centroid is the closest to the data point. Next we recompute the
centroid for each cluster. We repeat this process until the objective function
converges. K-means is sensitive to initialization. In text clustering, we may
not always get meaningful clusters.

In agglomerative hierarchical clustering, we group similar data points
together in a bottom-up fashion and stop when some stopping criterion is met.

However, those algorithms need all the data in order to do clustering while
our task is real-time [15]. Our redundancy removal method can handle tweets
coming in a stream.

###  II-B Data and Preprocessing

In order to evaluate our filtering system in the real time environment, we
need the queries (interest profiles), a stream of tweets and the corresponding
ground truth [17].

A collection of the interest profiles (queries) are provided by TREC 2016
111http://trecrts.github.io/TREC2016-RTS-topics.json and we used the narrative
part in the interest profile for the queries.

The ground truth can also be found on the TREC 2016 website
222http://trec.nist.gov/data/rts/rts2016-qrels.txt, in which relevance of
sampled tweets (tweet id without tweet content) is given (0 for irrelevant, 1
for relevant, 2 for redundant). Furthermore, for each interest profiles, the
relevant tweets are also clustered according to content
333http://trec.nist.gov/data/rts/rts2016-batch-clusters.json, and tweets in
the same cluster are considered to be equivalent and contain the same
information.

For tweets, Foreseer Group sampled tweets during the time span of the
evaluation. The data from the Foreseer Group is really large (22G/day after
compressing, 10% sample of the original twitter stream) while the tweets for
TREC is 1% sample of the original twitter stream. We process the data to
search for the intersection of the tweets with ground truth available and the
tweets from Foreseer Group. Then we update the redundant labels in the ground
truth as some relevant tweets may be missing. As a result, there are 27
queries in total, with about 1000 tweets per query on average.

###  II-C Methods

In this section, we will discuss the methods we use to find the relevant
tweets and remove the redundant tweets from all the relevant. The procedure we
are going to present is summarized in Algorithm 1.

####  II-C1 Relevance

As mentioned, given a query $q$ and a tweet (document) $d$, we will determine
whether the tweet is relevant to query by calculating the similarity score
with Dirichlet Smoothing [16], which is defined by

$\small score(d,q)=\sum_{w\in d,w\in q}c(w,q)\log(1+\frac{c(w,d)}{\mu\cdot
p(w|C)})+|q|\log\frac{\mu}{|d|+\mu}$ (2)

where $\mu$ is a parameter and the reference model $p(w|C)$ is trained from
the tweets on August 1 from Foreseer group. The resulting vocabulary size
about $10^{7}$. The assumption we make here is that the language model of
tweet from August 2nd to August 11th is similar to the language model on
August 1. If the similarity score is above a threshold $t$, then we will
classify the tweet as relevant.

In order to study the effect of smoothing, we will base our baseline method on
the same similarity score but without smoothing, i.e. $\mu=0$. Because of
computing difficulty (divide by 0 and log of 0), our practical baseline method
is calculating the score with very little smoothing.

####  II-C2 Redundancy

With the consideration of novelty, we shall only push one tweet from a
cluster, which means that the filtering system we build is expected to remove
the redundant tweets among all the relevant [10]. If a tweet is classified to
be a relevant tweet, we can then calculate the cosine similarity between the
new tweet and the previous novel tweets from each cluster. If the maximum
cosine similarity is less than a threshold $\theta$, then the tweet is
regarded to be novel, and we shall keep this tweet. If the maximum cosine
similarity is above the threshold $\theta$, the tweet is considered to belong
to the same cluster as an existing novel tweet and is redundant. For the
consideration of efficiency, we don’t keep redundant tweets. So for each
relevant tweet, the number of cosine similarities we calculate is the number
of current surviving tweets (which are considered to be relevant and novel).

However, in practice, we find the precision of our approach is already low
(See Figure 1, lower left plot). So we think that the evaluation of redundancy
removal is not practical. The relevant tweets we retrieved form clusters and
we keep only one tweet from each cluster, while the irrelevant tweets we
retrieved are unlikely to be clustered. This will further reduce the precision
and users will have a terrible experience.

Algorithm 1 Relevance and Redundancy Removal, Input: twitter stream, query
$q$, threshold for relevance $t$, threshold for redundancy $\theta$ Output:
relevant, novel tweets

For a tweet $d$, calculate $score(d,q)$

If $score(d,q)<t$, the tweet is irrelevant, drop $d$. Else, calculate if it’s
redundant

Calculate the cosine similarity $cos(d,d_{i})$ where $d_{i}$ are one tweet
from each current existing clusters

If $\max_{i}cos(d,d_{i})<\theta$, $d$ is novel, we push and save $d$. Else,
$d$ belong to a current existing cluster and is redundant. Drop $d$

###  II-D Experiment

####  II-D1 Setting

In order to simulate real time environment, for each query (interest profile),
we organize the tweets (whose content and ground truth are both available) in
the time order, and feed our filtering system one by one. In our method, the
Dirichlet smoothing parameter is set to be $\mu=2500$ (see eq. (2)). As
mentioned, the practical baseline model we adapt has very little smoothing,
$\mu=0.000000001$.

####  II-D2 Threshold $t$

In this subsection, we discuss the effect of threshold $t$ on precision and
recall. The similarity score between the query $q$ and tweet $d$ is
independent of $t$, thus the method has different performance for different
threshold $t$. As we can observe from Figure 1, for reasonable threshold, the
precision is always low, which matches performance of other approaches for
this problem [2]. The reason for this phenomenon is that, among the huge
stream of tweets, the fraction of relevant tweets is close to zero, actually
for some query, there is no tweet is known to be relevant.

For our method, when the threshold is in 3-5, the relationship between
precision and recall is nearly linear. The precision and recall are not so
sensitive to threshold either. So we set the threshold to be 4.5 here, as
users may get annoying when there are too many irrelevant pushes.

For the baseline, when threshold is in the range of [-1000,-750], the
precision is not improving much while the recall is degrading rapidly. The
threshold is around the turning point of the precision-recall curve: (P=0.055,
R=0.55), where the threshold is about -850.

##  III Results & Discussion

###  III-A Results

In order to evaluate our method and compare it with the baseline method in
detail, we calculate three common information retrieval metrics for both our
method and the baseline. The first one is mean average precision (mAP) [26,
27, 30, 31], which provides a single-figure measure of quality across recall
levels. Among evaluation measures, mAP has been shown to have especially good
discrimination and stability. For a single query (interest profile), mAP is
the average of the precision values obtained for the set of top $k$ documents
existing after each relevant document is retrieved, and this value is then
averaged over all queries. The other two metrics we use are cumulative gain at
30 (CG@30) and discounted cumulative gain (DCG@30, log discount, base $r=2$).
DCG is a measure of ranking quality often used to measure the effectiveness of
web search engine algorithms or related applications [28]. Using a graded
relevance scale of documents in a search-engine result set, DCG measures the
usefulness, or gain, of a document based on its position in the result list.
The gain is accumulated from the top of the result list to the bottom, with
the gain of each result discounted at lower ranks.

Table I reports the three metrics for our method and the baseline method. As
can be seen from the table, our method outperforms the baseline method with
respect to all three evaluation metrics by a large margin. The superiority of
our methods can also be seen from the precision and recall curves (See Figure
1). It is clear that the precision-recall curve of our filtering system has a
larger area under the curve than that of the baseline method. In addition,
from the comparison of the threshold and precision/recall figures (See Figure
1, left panels), we can see that the values for the precision and recall of
our filtering system are better than those of the baseline method.

To provide a more comprehensive analysis of the performance metrics, we
include 95% confidence intervals for the reported improvements in mAP, CG@30,
and DCG@30. We also performed paired t-tests to statistically validate the
superiority of our method over the baseline, with p-values below 0.01
indicating statistically significant improvements.

TABLE I: Results comparison between our method and the baseline, where the three metrics are calculated and averaged over all 27 queries. Method | mAP | CG@30 | DCG@30
---|---|---|---
Baseline | 0.09 ± 0.02 | 0.88 ± 0.05 | 2.81 ± 0.20
Our method | 0.17 ± 0.03 | 1.78 ± 0.07 | 5.42 ± 0.30

Figure 1: Precision and Recall of different thresholds for filtering system
with and without smoothing. Top left panel: precision/recall versus threshold
for baseline. Top right panel: recall versus precision for baseline. Bottom
left panel: precision/recall versus threshold for filtering system with
smoothing $\mu=2500$. Bottom right panel: recall versus precision for
filtering system with smoothing $\mu=2500$.

###  III-B Discussion

####  III-B1 Time Consumption

When we calculate the similarity score in equation (2), we use vector
representation (CountVectorizer in sklearn) of documents, and then the
similarity score can be computed through an inner product form. As the
vocabulary size is large ($10^{7}$), it takes about 1 second to calculate the
similarity score for each incoming tweet. Considering the huge amount of
tweets, this may not withstand the 1% sample of the Twitter stream during the
actual TREC competition.

The main reason behind this long time consumption is that, after smoothing,
each query and tweet is converted to a long and dense vector (recall that our
vocabulary size is about $10^{7}$). One modification is to truncate our
vocabulary size. However, notice that the original queries and tweets vector
are quite sparse. We hope that there are advanced computation approaches that
can reduce the time consumption significantly.

Furthermore, while in this project we processed each query in series, in the
actual TREC competition it might be better to process each query in a parallel
manner.

####  III-B2 Future Work

For future work, there are several directions we can pursue:

  * •

Email Digestion Scenario: As we just need to digest all tweets into one email
every day, this scenario can be categorized as a classical document ranking
question. For the redundant removal, as we just need to cluster all tweets in
a day that we regard as relevant, standard methods including K-means and
agglomerative hierarchical clustering are expected to have good performance.
This is because, in this scenario, redundancy removal does not need to be
performed in real-time and we already have all the data at the end of the day.

  * •

Precision Improvement: Improve the precision of the retrieved documents such
that it will be practical to remove the redundant tweets. The method might use
other smoothing methods and similarity scores, such as KL divergence [4] and
JS divergence [1]. The threshold might be learned or updated every day [1].
The use of emojis can also be adapted. Furthermore, this real-time
summarization can be modeled as a reinforcement learning problem [3, 12, 17].
When the precision is high enough, we will implement the redundancy removal
algorithm.

  * •

Scalability: Address the challenge of scaling up for higher tweet frequencies
or larger user bases by exploring distributed computing frameworks such as
Apache Spark and optimized data structures for fast retrieval. Implementing
real-time processing pipelines and evaluating system performance under varying
tweet volumes and different user interest profiles will also be crucial.

By addressing these aspects, we aim to further improve our method and make it
more practical and scalable for real-world applications.

##  IV Conclusion

The research presented in this paper effectively demonstrates the superiority
of a new method over a baseline approach in information retrieval tasks,
showcasing significant improvements across all metrics such as mean average
precision (mAP), cumulative gain at 30 (CG@30), and discounted cumulative gain
at 30 (DCG@30). The detailed results, supported by precision-recall curves,
confirm the new method’s enhanced ability to retrieve relevant documents more
accurately. Conclusions drawn highlight the efficacy of the smoothing method
used, which substantially boosts retrieval quality, and point to the need for
efficiency improvements, given the high computational demands noted with large
vocabulary sizes. This research holds substantial significance for both the
academic community and practitioners by providing a proven approach that could
enhance real-time data processing applications like social media analytics,
improving both the relevance and precision of information retrieved.
Additionally, it sets a clear path for future research, suggesting further
exploration into optimization techniques and machine learning applications to
refine retrieval processes.

## References

  * [1] Bei, C. and Hu, P. (2016) CCNU at TREC 2016 Real-Time Summarization Track. In: Proceedings of the 25th Text REtrieval Conference (TREC). Gaithersburg.
  * [2] Lin, J., Roegiest, A., Tan, L., McCreadie, R., Voorhees, E. M., & Diaz, F. (2016) Overview of the TREC 2016 Real-Time Summarization Track. In: Proceedings of the 25th Text REtrieval Conference (TREC). Gaithersburg.
  * [3] Tan, H., Lu, Z. and Li, W. (2017) Neural network based reinforcement learning for real-time pushing on text stream. In: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. Tokyo. pp. 913-916.
  * [4] Yao, L., Lv, C., Fan, F., Yang, J., & Zhao, D. (2016) PKUICST at TREC 2016 Real-Time Summarization Track: Push Notifications and Email Digest. In: Proceedings of the 25th Text REtrieval Conference (TREC). Gaithersburg.
  * [5] Peng, X., Xu, Q., Feng, Z., Zhao, H., Tan, L., Zhou, Y., … & Zheng, Y. (2024) Automatic News Generation and Fact-Checking System Based on Language Processing. arXiv preprint arXiv:2405.10492.
  * [6] Zhai, C., & Lafferty, J. (2001) The dual role of smoothing in the language modeling approach. In: Proceedings of the Workshop on Language Models for Information Retrieval (LMIR). Pittsburgh. pp. 31-36.
  * [7] Zhu, Z. & Zhou, W. Taming heavy-tailed features by shrinkage. International Conference On Artificial Intelligence And Statistics. pp. 3268-3276 (2021)
  * [8] Xin, Y., Du, J., Wang, Q., Lin, Z. & Yan, K. VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding. Proceedings Of The AAAI Conference On Artificial Intelligence. 38, 16085-16093 (2024)
  * [9] Read, A., Zhou, W., Saini, S., Zhu, J. & Waljee, A. Prediction of Gastrointestinal Tract Cancers Using Longitudinal Electronic Health Record Data. Cancers. 15, 1399 (2023)
  * [10] Xin, Y., Du, J., Wang, Q., Yan, K. & Ding, S. MmAP: Multi-modal Alignment Prompt for Cross-domain Multi-task Learning. Proceedings Of The AAAI Conference On Artificial Intelligence. 38, 16076-16084 (2024)
  * [11] Read, A., Zhou, W., Saini, S., Zhu, J. & Waljee, A. PREDICTION OF GASTROINTESTINAL TRACT CANCERS USING LONGITUDINAL ELECTRONIC HEALTH RECORD DATA. GASTROENTEROLOGY. 162, S1045-S1045 (2022)
  * [12] Liu, H., Shen, Y., Zhou, W., Zou, Y., Zhou, C. & He, S. Adaptive speed planning for unmanned vehicle based on deep reinforcement learning. ArXiv Preprint ArXiv:2404.17379. (2024)
  * [13] Xin, Y., Luo, S., Jin, P., Du, Y. & Wang, C. Self-Training with Label-Feature-Consistency for Domain Adaptation. International Conference On Database Systems For Advanced Applications. pp. 84-99 (2023)
  * [14] Yang, Q., Li, P., Shen, X., Ding, Z., Zhou, W., Nian, Y. & Xu, X. A comparative study on enhancing prediction in social network advertisement through data augmentation. ArXiv Preprint ArXiv:2404.13812. (2024)
  * [15] Chen, S., Kong, N., Sun, X., Meng, H. & Li, M. Claims data-driven modeling of hospital time-to-readmission risk with latent heterogeneity. Health Care Management Science. 22 pp. 156-179 (2019)
  * [16] Chen, S., Lu, L., Zhang, Q. & Li, M. Optimal binomial reliability demonstration tests design under acceptance decision uncertainty. Quality Engineering. 32, 492-508 (2020)
  * [17] Shen, Y., Liu, H., Liu, X., Zhou, W., Zhou, C. & Chen, Y. Localization through particle filter powered neural network estimated monocular camera poses. ArXiv Preprint ArXiv:2404.17685. (2024)
  * [18] Chen, S., Lu, L., Xiang, Y., Lu, Q. & Li, M. A data heterogeneity modeling and quantification approach for field pre-assessment of chloride-induced corrosion in aging infrastructures. Reliability Engineering & System Safety. 171 pp. 123-135 (2018)
  * [19] Li, P., Yang, Q., Geng, X., Zhou, W., Ding, Z. & Nian, Y. Exploring diverse methods in visual question answering. ArXiv Preprint ArXiv:2404.13565. (2024)
  * [20] Lyu W, Zheng S, Ma T, et al. A Study of the Attention Abnormality in Trojaned BERTs[C]//Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022: 4727-4741.
  * [21] Li S, Lin R, Pei S. Multi-modal preference alignment remedies regression of visual instruction tuning on language model[J]. arXiv preprint arXiv:2402.10884, 2024.
  * [22] Lyu W, Zheng S, Pang L, et al. Attention-Enhancing Backdoor Attacks Against BERT-based Models[C]//Findings of the Association for Computational Linguistics: EMNLP 2023. 2023: 10672-10690.
  * [23] Li S, Tajbakhsh N. Scigraphqa: A large-scale synthetic multi-turn question-answering dataset for scientific graphs[J]. arXiv preprint arXiv:2308.03349, 2023.
  * [24] Lyu W, Dong X, Wong R, et al. A multimodal transformer: Fusing clinical notes with structured EHR data for interpretable in-hospital mortality prediction[C]//AMIA Annual Symposium Proceedings. American Medical Informatics Association, 2022, 2022: 719.
  * [25] Lyu W, Zheng S, Ling H, et al. Backdoor Attacks Against Transformers with Attention Enhancement[C]//ICLR 2023 Workshop on Backdoor Attacks and Defenses in Machine Learning. 2023.
  * [26] Gao Y, Bu X, Hu Y, et al. Solution for large-scale hierarchical object detection datasets with incomplete annotation and data imbalance[J]. arXiv preprint arXiv:1810.06208, 2018.
  * [27] Feng W, Bu X, Zhang C, et al. Beyond bounding box: Multimodal knowledge learning for object detection[J]. arXiv preprint arXiv:2205.04072, 2022.
  * [28] Chen S, Lin C, Guan W, et al. Visual Encoding and Debiasing for CTR Prediction[J]. arXiv preprint arXiv:2205.04168, 2022.
  * [29] Bu X, Pei M, Jia Y. Attention Estimation for Input Switch in Scalable Multi-display Environments[C]//Neural Information Processing: 23rd International Conference, ICONIP 2016, Kyoto, Japan, October 16–21, 2016, Proceedings, Part IV 23. Springer International Publishing, 2016: 329-336.
  * [30] Bu X, Peng J, Yan J, et al. Gaia: A transfer learning system of object detection that fits your needs[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 274-283.
  * [31] Peng J, Chang Q, Yin H, et al. GAIA-Universe: Everything is Super-Netify[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(10): 11856-11868.
  * [32] Jiang B, Tan Z, Nirmal A, et al. Disinformation detection: An evolving challenge in the age of llms[C]//Proceedings of the 2024 SIAM International Conference on Data Mining (SDM). Society for Industrial and Applied Mathematics, 2024: 427-435.
  * [33] Tan Z, Beigi A, Wang S, et al. Large Language Models for Data Annotation: A Survey[J]. arXiv preprint arXiv:2402.13446, 2024.
  * [34] Yuan B, Chen Y, Tan Z, et al. Label Distribution Learning-Enhanced Dual-KNN for Text Classification[C]//Proceedings of the 2024 SIAM International Conference on Data Mining (SDM). Society for Industrial and Applied Mathematics, 2024: 400-408.