# MIPT-NSU-UTMN at SemEval-2021 Task 5: Ensembling Learning with Pre-trained
Language Models for Toxic Spans Detection

Mikhail Kotyushev
Moscow Institute of
Physics and Technology
Moscow, Russia
<EMAIL_ADDRESS>
&Anna Glazkova
University of
Tyumen
Tyumen, Russia
<EMAIL_ADDRESS>
&Dmitry Morozov
Novosibirsk State
University
Novosibirsk, Russia
<EMAIL_ADDRESS>

###### Abstract

This paper describes our system for SemEval-2021 Task 5 on Toxic Spans
Detection. We developed ensemble models using BERT-based neural architectures
and post-processing to combine tokens into spans. We evaluated several pre-
trained language models using various ensemble techniques for toxic span
identification and achieved sizable improvements over our baseline fine-tuned
BERT models. Finally, our system obtained a F1-score of 67.55% on test data.

##  1 Introduction

Toxic speech has become a rising issue for social media communities. Abusive
content is very diverse and therefore offensive language and toxic speech
detection is not a trivial issue. Besides, social media moderation of lengthy
comments and posts is often a time-consuming process. In this regard, the task
of detecting toxic spans in social media texts deserves close attention.

This work is based on the participation of our team, named MIPT-NSU-UTMN, in
SemEval 2021 Task 5, “Toxic Spans Detection” (Pavlopoulos et al., 2021).
Organizers of the shared task provided participants with the trial, train, and
test sets of English social media comments annotated at the span level
indicating the presence or absence of text toxicity. We formulated the task as
a token classification problem and investigated several BERT-based models
using two-step knowledge transfer. We found that preliminary fine-tuning of
the model on data that is close to the target domain improves the quality of
the token classification. The source code of our models is available at
https://github.com/morozowdmitry/semeval21.

The paper is organized as follows. A brief review of related work is given in
Section 2. The definition of the task has been summarized in Section 3. The
proposed methods and experimental settings have been elaborated in Section 4.
Section 5 contains the results and error analysis respectively. Section 6 is a
conclusion.

##  2 Related Work

Computational approaches to tackle text toxicity have recently gained a lot of
interest due to the widespread use of social media. Since moderation is
crucial to promoting healthy online discussions, research on toxicity
detection has been attracting much attention. Our work is also related to hate
speech and abusive language detection (Fortuna et al., 2020).

The toxic speech detection task is usually framed as a supervised learning
problem. Moreover, fairly generic features, such as bag of words (Harris,
1954) or word embeddings (Mikolov et al., 2013), systematically yield
reasonable classification performance (Fortuna and Nunes, 2018; Schmidt and
Wiegand, 2017). To better understand the mechanisms of toxic speech detection,
some scholars (Waseem et al., 2017; Lee et al., 2018; Karan and Šnajder, 2018;
Swamy et al., 2019) compared different techniques for abusive language
analysis. Neural architectures and deep learning methods achieved high results
in this domain. Thus, Pavlopoulos et al. (2017a, b) explored the possibilities
of deep learning and deep attention mechanisms for abusive comment moderation.
Park and Fung (2017) proposed an approach to performing classification on
abusive language based on convolutional neural networks (CNN). Chakrabarty et
al. (2019) used Bidirectional Long-Short Term Memory network. Castelle (2018)
experimented with CNN and Gated Recurrent Units. Some recent studies (Mozafari
et al., 2019; Risch et al., 2019; Liu et al., 2019a; Nikolov and Radivchev,
2019) utilized pre-trained language models such as Bidirectional Encoder
Representations from Transformers (BERT) (Devlin et al., 2018) to detect
offensive or abusive language.

In recent years, the task of detecting and analyzing abusive, toxic, or
offensive language has attracted the attention of more and more researchers.
The shared tasks based on carefully curated resources, such as those organized
at the SemEval (Zampieri et al., 2019; Basile et al., 2019), GermEval (Wiegand
et al., 2018), EVALITA (Bosco et al., 2018), and OSACT (Mubarak et al., 2020)
events, have significantly contributed to the progress of the field and to the
enrichment of linguistic resources. In addition to the corpora collected for
these shared tasks, Rosenthal et al. (2020) released a large-scale dataset for
offensive language identification. Ibrohim and Budi (2018); Leite et al.
(2020); Pitenis et al. (2020); Komalova et al. (2021) presented various
datasets for abusive speech detection in non-English languages. Most of these
datasets classify whole texts or documents, and do not identify the spans that
make a text toxic.

##  3 Shared Task

The task focuses on the evaluation of systems that detect the spans that make
a text toxic, when detecting such spans is possible. The goal of the task is
to define a sequence of words (character offsets) that attribute to the
toxicity of the text, for example:

  * •

Input. “This is a stupid example, so thank you for nothing a!@#!@”.

  * •

Output. [10,11,12,13,14,15,51,52,53,54,55, 56].

The sources of data were various posts (comments) from publicly available
datasets. The provided dataset contains 10,629 posts split into training
(7939), trial (690), and test (2000) subsets.

Inspired by Da San Martino et al. (2019), the organizers proposed to employ
the F1-score for evaluating the responses of a system participating in the
shared task. Let system $A_{i}$ return a set $S^{t}_{A_{i}}$ of character
offsets, for parts of the post found to be toxic. Let $G^{t}$ be the character
offsets of the ground truth annotations of $t$. The F1-score of system $A_{i}$
is calculated with respect to the ground truth $G$ for post $t$ as follows,
where $|\cdot|$ denotes set cardinality.

$\textit{F}^{t}_{1}(A_{i},G)=\frac{2\cdot P^{t}(A_{i},G)\cdot
R^{t}(A_{i},G)}{P^{t}(A_{i},G)+R^{t}(A_{i},G)},$
$\textit{P}^{t}(A_{i},G)=\frac{S^{t}_{A_{i}}\cap S^{t}_{G}}{S^{t}_{A_{i}}},$
$\textit{R}^{t}(A_{i},G)=\frac{S^{t}_{A_{i}}\cap S^{t}_{G}}{S^{t}_{G}}.$

The final F1-score is an average $\textit{F}^{t}_{1}(A_{i},G)$ over all the
posts $t$ of an evaluation dataset $T$ to obtain a single score for system
$A_{i}$.

##  4 Methodology

The stated problem was modified from char-level to token-level binary-
classification. The proposed solution utilizes a pre-trained language model
with a classification head to classify tokens. Different configurations of
BERT pre-trained as masked language models were considered as a backbone.

Due to the lack of available token-level labeled public datasets for toxic
comment and the relatively small size and sparsity of dataset provided by the
competition, the following training pipeline was proposed to enhance knowledge
transfer. First, fine-tune pre-trained BERT on a larger-scale task of toxic
comment classification, using the Jigsaw
dataset111https://www.kaggle.com/c/jigsaw-toxic-comment-classification-
challenge from which the competition data were constructed. Second, fine-tune
obtained model to solve the actual toxic tokens classification problem. The
exact training parameters are to be found below.

For the first step:

  * •

remove texts occurred in spans dataset from classification dataset to prevent
data leakage (so as spans dataset is sampled from classification dataset);

  * •

4 epochs, 200 tokens max length, 64 batch size, 10 gradient accumulation,
mixed-precision FP16;

  * •

default AdamW (Loshchilov and Hutter, 2017) with lr = 4e-5, Layer-wise
Decreasing Layer Rate (Sun et al., 2019) with decay $\eta=0.95$ and cosine
learning rate (LR) schedule with T = 4 epochs and constant LR after epoch 3;

  * •

selected bert-base-uncased as best performance / speed ratio;

  * •

the best model on validation selection each 0.1 epoch by AUC.

For the second step:

  * •

hold-out $\approx 14\%$ of data to train ensemble of models later;

  * •

out-of-5-fold training on the residual $\approx 86\%$ of data;

  * •

4 epochs, 512 tokens max length, 16 batch size, 10 gradient accumulation,
mixed precision FP16;

  * •

default AdamW with lr = 4e-5, Layer-wise Decreasing Layer Rate with decay
$\eta=0.95$ and cosine LR schedule with T = 4 epochs;

  * •

the best model on validation selection each 0.1 epoch by F1-score.

The final solution contains $N\times K$ models, where $N$ is the number of
different backbone BERT architectures, $K$ is the number of folds (5 in the
current experiments). Obtained models are further to be ensembled using
different strategies with validation on the single hold-out dataset:

  * •

hard voting: final spans are selected as at least one model (spans union), as
all the models (spans intersection) or as some intermediate methods with at
least $m$ models;

  * •

soft voting: final probability is calculated as a weighted sum of models
probabilities;

  * •

train meta classifier.

##  5 Experiments and Results

Three pre-trained backbone BERT architectures were considered: BERT base
uncased, BERT large uncased (Devlin et al., 2018), and BERT base pre-trained
for Hate Speech Detection (Aluru et al., 2020). First step setup and results:

  * •

select subset of Jigsaw toxic classification data: all the targets with
toxicity score $\geq$ 0.5 ($L=135168$ objects) as class 1 and randomly sampled
$3*L$ objects with toxicity score $<$ 0.5 as class 0;

  * •

stratified 80% train, 20% validation;

  * •

0.968 AUC bert-base, 0.968 AUC bert-large, 0.942 AUC dehate bert.

So as models except BERT base uncased did not show compatible performance for
token classification (and later for tests on the fold 0 didn’t show good
F1-score for the actual task as well), later experiments were continued only
for BERT base uncased pre-trained model fine-tuned on token classification.

For step two results are following:

  * •

train + trial, 8621 comments;

  * •

average F1-score over 5 folds is 0.6714.

The experiments were conducted with Huggingface transformers library (Wolf et
al., 2019).

Many patterns in our results are expected, but some stand out. In general, our
model is good at detecting obscene language and utterances that demean honor
and dignity or denote low moral character. We noticed that our model is not
very good at identifying the posts that have no toxic span annotations.
According to the corpus description, in some toxic posts, the core message is
conveyed may be inherently toxic. Thus, a sarcastic post can indirectly claim
that people of a particular origin are inferior. Hence, it was difficult to
attribute the toxicity of those posts to particular spans. In such cases, the
corresponding posts were labeled as not containing toxic spans. Among our
results, there are many examples where the model detected spans in not
annotated posts, for example:

  * •

“uhhh Hillary Clinton is a serial killer and thief”: [] (true annotation),
[26, 27, 28, 29, 30, 31, 33, 34, 35, 36, 37, 38, 44, 45, 46, 47, 48] (our
annotation, “uhhh Hillary Clinton is a serial killer and thief”);

  * •

“This goes way beyond just being an asshole skipper, dude must have some
serious mental issues”: [] (true annotation), [35, 36, 37, 38, 39, 40, 41]
(our annotation, “This goes way beyond just being an asshole skipper, dude
must have some serious mental issues”).

In addition, some texts in the dataset raise questions of the annotation
credibility, for example:

  * •

“How the hell is this news? Am I supposed to be shocked that the Crown Prince
of Bahrain or one of the world’s biggest celebrity superstars get’s better
access to the State Department then I do? During which administration has this
ever not been true? The media’s desperation to keep this election close is far
past ridiculous” (training set, the toxic span annotation is underlined);

  * •

“Yup. NVN used the Press. The Press was USED. Used like their sister on prom
night! Idiots. All faux-erudite, not realizing they were being played”
(training set, the original annotation is underlined);

  * •

“And you are a complete moron who obviously doesn’t know the meaning of the
word narcissist. By the way your bias is showing” (test set, the original
annotation is underlined, the annotation of our model is highlighted in bold).

The final result of our model is presented in Table 1. As can be seen from the
table, the systems of the participants produce close results. Our system
achieved 67.55% of F1-score on the test set of this shared task that attracted
91 submitted teams in total. This value exceeded the average result by almost
10%.

Table 1: Results on the test set. Rank | Team | F1-score
---|---|---
1 | HITSZ-HLT | 0.7083
26 | UAntwerp | 0.67552
27 | MIPT-NSU-UTMN | 0.67551
28 | NLRG | 0.67532
| Avg result | 0.57805

##  6 Conclusion

This paper introduces our BERT-based model for toxic spans detection. As
expected, pre-training of the BERT model using an additional domain-specific
dataset improves further toxic spans detection performance. Experimenting with
different fine-tuning approaches has shown that our BERT-based model benefits
from the two-step knowledge transfer technique. An ensemble with spans
intersection obtained our best result on the test data.

In our future work, we will evaluate various language models, such as
distilled versions of BERT (Sanh et al., 2019; Jiao et al., 2020) and RoBERTa
(Liu et al., 2019b).

## References

  * Aluru et al. (2020) Sai Saket Aluru, Binny Mathew, Punyajoy Saha, and Animesh Mukherjee. 2020.  Deep learning models for multilingual hate speech detection.  _arXiv preprint arXiv:2004.06465_.
  * Basile et al. (2019) Valerio Basile, Cristina Bosco, Elisabetta Fersini, Nozza Debora, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, Manuela Sanguinetti, et al. 2019.  Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter.  In _13th International Workshop on Semantic Evaluation_ , pages 54–63. Association for Computational Linguistics.
  * Bosco et al. (2018) Cristina Bosco, Dell’Orletta Felice, Fabio Poletto, Manuela Sanguinetti, and Tesconi Maurizio. 2018.  Overview of the evalita 2018 hate speech detection task.  In _EVALITA 2018-Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian_ , volume 2263, pages 1–9. CEUR.
  * Castelle (2018) Michael Castelle. 2018.  The linguistic ideologies of deep abusive language classification.  In _Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)_ , pages 160–170.
  * Chakrabarty et al. (2019) Tuhin Chakrabarty, Kilol Gupta, and Smaranda Muresan. 2019.  Pay “attention” to your context when classifying abusive language.  In _Proceedings of the Third Workshop on Abusive Language Online_ , pages 70–79.
  * Da San Martino et al. (2019) Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeno, Rostislav Petrov, and Preslav Nakov. 2019.  Fine-grained analysis of propaganda in news article.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5640–5650.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: pre-training of deep bidirectional transformers for language understanding.  _CoRR_ , abs/1810.04805.
  * Fortuna and Nunes (2018) Paula Fortuna and Sérgio Nunes. 2018.  A survey on automatic detection of hate speech in text.  _ACM Computing Surveys (CSUR)_ , 51(4):1–30.
  * Fortuna et al. (2020) Paula Fortuna, Juan Soler, and Leo Wanner. 2020.  Toxic, hateful, offensive or abusive? what are we really classifying? an empirical analysis of hate speech datasets.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 6786–6794.
  * Harris (1954) Zellig S Harris. 1954.  Distributional structure.  _Word_ , 10(2-3):146–162.
  * Ibrohim and Budi (2018) Muhammad Okky Ibrohim and Indra Budi. 2018.  A dataset and preliminaries study for abusive language detection in indonesian social media.  _Procedia Computer Science_ , 135:222–229.
  * Jiao et al. (2020) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020.  Tinybert: Distilling bert for natural language understanding.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , pages 4163–4174.
  * Karan and Šnajder (2018) Mladen Karan and Jan Šnajder. 2018.  Cross-domain detection of abusive language online.  In _Proceedings of the 2nd workshop on abusive language online (ALW2)_ , pages 132–137.
  * Komalova et al. (2021) Liliya Komalova, Tatiana Goloshchapova, Leonid Motovskikh, Rostislav Epifanov, Dmitry Morozov, and Anna Glazkova. 2021.  Mca workshop - toxic comments.  Mendeley Data, V1.
  * Lee et al. (2018) Younghun Lee, Seunghyun Yoon, and Kyomin Jung. 2018.  Comparative studies of detecting abusive language on twitter.  In _Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)_ , pages 101–106.
  * Leite et al. (2020) JA Leite, DF Silva, K Bontcheva, and C Scarton. 2020.  Toxic language detection in social media for brazilian portuguese: new dataset and multilingual analysis.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pages 914–924. Association for Computational Linguistics (ACL).
  * Liu et al. (2019a) Ping Liu, Wen Li, and Liang Zou. 2019a.  Nuli at semeval-2019 task 6: Transfer learning for offensive language detection using bidirectional transformers.  In _Proceedings of the 13th international workshop on semantic evaluation_ , pages 87–91.
  * Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Mozafari et al. (2019) Marzieh Mozafari, Reza Farahbakhsh, and Noel Crespi. 2019.  A bert-based transfer learning approach for hate speech detection in online social media.  In _International Conference on Complex Networks and Their Applications_ , pages 928–940. Springer.
  * Mubarak et al. (2020) Hamdy Mubarak, Kareem Darwish, Walid Magdy, Tamer Elsayed, and Hend Al-Khalifa. 2020\.  Overview of osact4 arabic offensive language detection shared task.  In _Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection_ , pages 48–52.
  * Nikolov and Radivchev (2019) Alex Nikolov and Victor Radivchev. 2019.  Nikolov-radivchev at semeval-2019 task 6: Offensive tweet classification with bert and ensembles.  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 691–695.
  * Park and Fung (2017) Ji Ho Park and Pascale Fung. 2017.  One-step and two-step classification for abusive language detection on twitter.  In _Proceedings of the First Workshop on Abusive Language Online_ , pages 41–45.
  * Pavlopoulos et al. (2021) John Pavlopoulos, Léo Laugier, Jeffrey Sorensen, and Ion Androutsopoulos. 2021\.  Semeval-2021 task 5: Toxic spans detection (to appear).  In _Proceedings of the 15th International Workshop on Semantic Evaluation_.
  * Pavlopoulos et al. (2017a) John Pavlopoulos, Prodromos Malakasiotis, and Ion Androutsopoulos. 2017a.  Deep learning for user comment moderation.  In _Proceedings of the First Workshop on Abusive Language Online_ , pages 25–35.
  * Pavlopoulos et al. (2017b) John Pavlopoulos, Prodromos Malakasiotis, and Ion Androutsopoulos. 2017b.  Deeper attention to abusive user content moderation.  In _Proceedings of the 2017 conference on empirical methods in natural language processing_ , pages 1125–1135.
  * Pitenis et al. (2020) Zesis Pitenis, Marcos Zampieri, and Tharindu Ranasinghe. 2020.  Offensive language identification in greek.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 5113–5119.
  * Risch et al. (2019) Julian Risch, Anke Stoll, Marc Ziegele, and Ralf Krestel. 2019.  hpidedis at germeval 2019: Offensive language identification using a german bert model.  In _KONVENS_.
  * Rosenthal et al. (2020) Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Marcos Zampieri, and Preslav Nakov. 2020.  A large-scale semi-supervised dataset for offensive language identification.  _arXiv preprint arXiv:2004.14454_.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Schmidt and Wiegand (2017) Anna Schmidt and Michael Wiegand. 2017.  A survey on hate speech detection using natural language processing.  In _Proceedings of the fifth international workshop on natural language processing for social media_ , pages 1–10.
  * Sun et al. (2019) Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019.  How to fine-tune BERT for text classification? _CoRR_ , abs/1905.05583.
  * Swamy et al. (2019) Steve Durairaj Swamy, Anupam Jamatia, and Björn Gambäck. 2019.  Studying generalisability across abusive language detection datasets.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 940–950.
  * Waseem et al. (2017) Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017.  Understanding abuse: A typology of abusive language detection subtasks.  In _Proceedings of the First Workshop on Abusive Language Online_ , pages 78–84.
  * Wiegand et al. (2018) Michael Wiegand, Melanie Siegel, and Josef Ruppenhofer. 2018.  Overview of the germeval 2018 shared task on the identification of offensive language.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Zampieri et al. (2019) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019.  Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval).  _arXiv preprint arXiv:1903.08983_.


# LLM-Collaboration on Automatic Science Journalism
for the General Audience

Gongyao Jiang  Xinran Shi
The Hong Kong University of
Science and Technology (Guangzhou)
<EMAIL_ADDRESS>
&Qiong Luo
The Hong Kong University of
Science and Technology (Guangzhou) /
The Hong Kong University of
Science and Technology
<EMAIL_ADDRESS>
Corresponding Author

###### Abstract

Science journalism reports current scientific discoveries to non-specialists,
aiming to enable public comprehension of the state of the art. However, this
task can be challenging as the audience often lacks specific knowledge about
the presented research. To address this challenge, we propose a framework that
integrates three LLMs mimicking the real-world writing-reading-feedback-
revision workflow, with one LLM acting as the journalist, a smaller LLM as the
general public reader, and the third LLM as an editor. The journalist’s
writing is iteratively refined by feedback from the reader and suggestions
from the editor. Our experiments demonstrate that by leveraging the
collaboration of two 7B and one 1.8B open-source LLMs, we can generate
articles that are more accessible than those generated by existing methods,
including advanced models such as GPT-4.

LLM-Collaboration on Automatic Science Journalism
for the General Audience

Gongyao Jiang and Xinran Shi The Hong Kong University of Science and
Technology (Guangzhou<EMAIL_ADDRESS>Qiong Luo††thanks: Corresponding
Author The Hong Kong University of Science and Technology (Guangzhou) / The
Hong Kong University of Science and Technology<EMAIL_ADDRESS>

##  1 Introduction

Science journalism creates journalistic content that covers a wide range of
scientific research, enhancing the public’s understanding of science (Göpfert,
2008; Allan, 2011; Angler, 2017). However, with rapid advances in various
disciplines, science journalism struggles to keep pace with the exponential
growth of knowledge. In response, automatic science journalism (ASJ) has been
proposed to expedite the filtering, learning, and communication of scientific
knowledge (Dangovski et al., 2021).

The essence of ASJ lies in elucidating complex technical content for readers,
thereby facilitating their comprehension of advanced research (Cardenas et
al., 2023). However, ASJ-generated content can be challenging for the general
audience who lack in-depth knowledge of specific fields. As depicted in Figure
1, the degree to which content is embraced varies among readers with different
levels of domain knowledge (August et al., 2024), underscoring the need for
high readability to broaden a diverse readership. Some researchers have
developed parallel corpora (Dangovski et al., 2021; Goldsack et al., 2022;
Cardenas et al., 2023), where the target content is extracted from online
scientific news or journals. However, these press releases often remain
technical, likely because they are originally tailored for professional
researchers rather than the general audience. Models trained on such content
struggle to generate materials easily understandable for a broader audience.

Figure 1: Reader experience varies with content technicality. Science
journalism for the general audience demands high accessibility. Figure 2:
Overview of our LLM-Collaboration framework, LLM-CLBR.

Large language models (LLMs) have shown impressive proficiency in instruction
adherence and content generation (Achiam et al., 2023; Bai et al., 2023),
thereby making them potential tools for ASJ. Furthermore, LLMs have exhibited
social intelligence (Park et al., 2023), enabling them to play realistic roles
and collaborate in real-world tasks (Qian et al., 2023; Talebirad and Nadiri,
2023). Motivated by these observations, we propose a novel framework that
leverages LLMs as communicative agents to collectively accomplish the ASJ
task.

Our goal is to automatically generate a popular science article based on a
technical paper and make it accessible to the general public. In the real
world, a journalist typically receives revision suggestions from a
professional editor; to make it accessible to the general public, the
journalist could pass their article to friends without domain expertise on the
topic to get feedback for revision. Therefore, we design our framework to
simulate such real-world scenarios, in which three LLMs collaborate to go
through a four-step iterative process, which includes writing, reading,
feedback, and revision, to generate highly accessible popular science
articles. The questions for our framework are then (1) how the reader gives
feedback; (2) how the editor gives suggestions based on the reader feedback;
(3) how the journalist takes suggestions to revise the generated article; and
(4) whether this iterative process improves the quality, especially
accessibility, of the generated article.

As depicted in Figure 2, we have an LLM serve as the journalist writing for
readers who lack domain knowledge of the given paper. We have another LLM,
smaller than the journalist LLM, which acts as a general reader, to read the
generated article and take notes to give reading feedback. As a less
proficient model, the reader LLM needs material that is easily understandable
to take comprehensive notes on. Therefore, the more accessible the
explanations in the written article are, the greater the clarity and accuracy
of the notes will exhibit.

LLMs have shown the capability of evaluating the quality of text (Chan et al.,
2023; Zheng et al., 2024a; Desmond et al., 2024). Therefore, we let an editor
(the third LLM) evaluate the correctness and comprehensiveness of the reader’s
notes and then provide suggestions for the revision of the journalist’s
article. The journalist then revises the previous version of the article based
on the suggestions. By this iterative and tuning-free process, the popular
science article is enhanced continuously and made more accessible to a general
audience. To the best of our knowledge, our work is the first comprehensive
study on LLMs for ASJ.

To assess the proposed method for ASJ, we employ both automatic metrics and
human evaluation on measures including readability, information conveyance,
authenticity, and interestingness of our generated articles. Compared with
other methods, including those with fine-tuning and prompting on various LLMs,
our proposed method achieves the highest readability while remaining
competitive on the other measures. We also provide a detailed analysis,
including ablation studies of removing the editor LLM, removing the reader
LLM, or removing both, as well as trend analysis and case studies, to offer a
comprehensive understanding of LLMs in the ASJ task.

In brief, we make the following contributions:

  * •

A novel ASJ framework with collaborative LLMs, generating articles of high
readability.

  * •

Comprehensive experiments, analyses, and recommendations for LLM usage in ASJ.

##  2 Methodology

Following Dangovski et al. (2021); Goldsack et al. (2022); Cardenas et al.
(2023), ASJ aims to automatically distill a scientific paper into an article
accessible to a broader audience. Our ASJ framework employs an iterative
workflow of writing, reading, suggestion-making, and revision, as illustrated
in Figure 2. All prompts for each LLM agent are listed in Appendix A.

###  2.1 The LLM Journalist

LLMs have shown strong writing abilities (Yuan et al., 2022; Wasi et al.,
2024). Thus, they are promising tools for rewriting a provided paper into a
more accessible version. Following established strategies (Zheng et al.,
2024a; Zhang et al., 2024), we start with prompting an LLM to assume the role
of a journalist. Subsequently, the LLM is prompted that, given the paper, its
task is to compose an article for the general public.

###  2.2 The LLM Reader

In our preliminary attempts, we asked the journalist LLM to directly assess
the readability of the content. However, the results are unsatisfactory,
probably due to the gap between human and model perceptions of reading
difficulty. As illustrated in the two text boxes at the bottom of Figure 3,
LLMs may regard both pieces of writing are of a similar level of readability,
as they all incorporate essential information, even if the terminology “low-
cost paper-based microfluidic diagnostic tests” on the left side is not
clearly explained. However, in the eyes of a human reader, the content on the
right is perceived as more accessible.111We briefly document other failed
attempts in Appendix B for reader information.

To address this readability assessment problem, we design a separate reader
LLM to read the content and generate reading notes. Our idea is inspired by
the accumulation of errors, a common phenomenon in pipeline systems (Caselli
et al., 2015; Wu et al., 2018; Jiang et al., 2023b; Dziri et al., 2023).
Specifically, we can utilize the propagation from the textual readability of
the journalist’s article to the reading comprehensibility of the reader in the
writing-reading pipeline, to induce the readability of the generated article
to become explicit.

Different from the LLM journalist, the reader LLM is of a smaller scale and
simulates a general reader with limited domain knowledge. Once presented with
the article crafted by the journalist, the reader LLM is employed to read the
article and take notes. Specifically, we instruct the reader LLM to explain
key terms in the article by extracting the explanations, if present, directly
from the article or offering explanations for these terms, otherwise.

Figure 3: Accessible content helps the reader take comprehensive notes.

Intuitively, if the article is more accessible, the reading notes will be more
comprehensive. For instance, in Figure 3, if the piece (left) lacks a detailed
explanation of the term “paper-based microfluidic diagnostic test”, the reader
will only note this term as being “combined with deep learning”. If the
article (right) explains the usage and advantages of this term in plain and
readable language, the reader LLM can grasp this knowledge. Through this
readability propagation from the journalist’s article to the comprehensiveness
of the reader’s notes, the editor LLM can better recognize issues in the
journalist’s writing and then provide suitable suggestions for modifications.

###  2.3 Automated Suggestions and Revisions

LLMs have demonstrated strong capabilities in serving as evaluators, widely
utilized in various generative tasks (Chan et al., 2023; Zheng et al., 2024a;
Desmond et al., 2024). Therefore, we employ an LLM as a senior editor for
automated evaluation of reader comprehension and providing recommendations for
article enhancement. Given the article from the journalist and notes from the
reader, an LLM editor is tasked with assessing the quality of the reader’s
notes and identifying issues in the journalist’s writing, which may lead to
reading obstacles.

Next, the editor offers advice for the journalist’s content development. For
example, in Figure 2, the editor finds that the reader’s understanding of the
term “blockchain” is limited, possibly due to an insufficient explanation in
the reading material. To address this perceived issue, the editor suggests
that the article should “explain technical terms”. These suggestions are then
incorporated into the instructions that will guide the journalist in revising
the article.

Subsequently, with the strong ability to follow instructions, the journalist
LLM rewrites the article according to the suggestions. Then, the revised piece
is fed to the reader for reading and taking notes to continue the process. By
an iterative cycle encompassing writing, note-taking, suggesting
modifications, and revision among three LLMs, the article tailored for the
general readership undergoes steady enhancement. We further analyze this
process in Section 4.

##  3 Experiments

###  3.1 Settings

Datasets. We use three publicly available corpora in different disciplines as
benchmarks, namely _SCITech_ , _eLife_ , and _PLOS_. Appendix C presents a
brief introduction and some statistics of these datasets. The same as the
previous studies, in _SCITech_ , we use 1431 instances for training and
validation and the remaining 1000 for testing. We also separate each of
_eLife_ and _PLOS_ datasets into training, validation, and testing splits at a
ratio of 90%/5%/5%.

| SCITech | eLife | PLOS |
---|---|---|---|---
Approach | CLI$\downarrow$ | FKGL$\downarrow$ | DCRS$\downarrow$ | CLI$\downarrow$ | FKGL$\downarrow$ | DCRS$\downarrow$ | CLI$\downarrow$ | FKGL$\downarrow$ | DCRS$\downarrow$ | Avg.
LLaMA-2-7B | 15.13 | 13.79 | 10.38 | 15.16 | 14.03 | 10.50 | 15.36 | 14.28 | 10.54 | 13.24
Gemma-7B | 14.93 | 13.75 | 10.52 | 15.01 | 12.08 | 11.03 | 15.52 | 12.29 | 10.92 | 12.89
Mistral-7B | 14.90 | 13.54 | 10.82 | 14.61 | 11.72 | 10.85 | 15.38 | 11.98 | 11.21 | 12.78
Qwen-1.5-7B | 14.77 | 13.50 | 10.72 | 14.72 | 11.83 | 10.92 | 15.06 | 11.94 | 11.09 | 12.73
LLaMA-3-8B | 14.84 | 13.18 | 10.41 | 14.55 | 11.65 | 10.49 | 15.18 | 12.01 | 10.88 | 12.58
Mixtral-8x7B | 13.98 | 13.25 | 10.36 | 14.21 | 12.01 | 10.28 | 15.34 | 11.58 | 10.98 | 12.44
Qwen-1.5-72B | 13.78 | 13.10 | 10.25 | 14.17 | 12.09 | 10.35 | 15.18 | 11.75 | 10.62 | 12.37
GPT-3.5-Turbo | 14.98 | 13.62 | 10.81 | 14.35 | 11.87 | 10.98 | 15.11 | 11.92 | 10.87 | 12.72
GPT-4 | 13.48 | 12.13 | 10.14 | 13.96 | 10.87 | 10.11 | 14.86 | 11.78 | 10.47 | 11.98
BART-FT | 13.43 | 15.22 | 10.66 | 12.32 | 10.65 | 9.19 | 15.61 | 14.24 | 10.51 | 12.43
Qwen-1.5-7B-FT | 13.37 | 14.79 | 10.48 | 12.15 | 10.63 | 9.12 | 15.54 | 13.95 | 10.58 | 12.29
LLM-WS-CLBR | 12.94 | 13.33 | 10.33 | 12.04 | 9.85 | 9.04 | 13.15 | 11.48 | 10.17 | 11.37
LLM-CLBR | 12.69 | 10.16 | 9.79 | 11.60 | 10.10 | 9.46 | 12.74 | 10.00 | 9.69 | 10.69
Reader: 1.8B $\rightarrow$ 7B | 12.81 | 10.35 | 9.68 | 11.82 | 10.01 | 9.51 | 12.67 | 9.93 | 9.78 | 10.73
$-$ Reading Notes | 13.21 | 10.63 | 10.33 | 12.22 | 10.78 | 10.02 | 13.35 | 10.59 | 10.25 | 11.26
$-$ Suggestions | 13.25 | 10.69 | 10.39 | 12.17 | 10.83 | 10.08 | 13.31 | 10.74 | 10.42 | 11.32
$-$ Collaboration | 13.50 | 11.01 | 10.71 | 12.47 | 10.99 | 10.41 | 13.65 | 10.91 | 10.70 | 11.59
Paper Abstracts | 16.67 | 15.27 | 11.39 | 17.53 | 15.35 | 11.87 | 16.38 | 14.98 | 11.10 | 14.50
Plain Summaries | 14.23 | 14.79 | 11.13 | 12.52 | 10.91 | 8.94 | 15.90 | 14.76 | 10.91 | 12.68
Table 1: The results of automated evaluation. We tested various methods, including the open-source LLM prompting, the closed LLM prompting, fine-tuning, and collaboration of LLMs (ours). Approach | Read. | Info. | Auth. | Intr.
---|---|---|---|---
Within Field
Plain Summaries | 2.95 | 2.90 | 3.35 | 2.70
Qwen1.5-7B | 3.50 | 3.35 | 3.40 | 3.10
GPT-4 | 3.80 | 3.75 | 3.80 | 3.40
LLM-CLBR | 3.95 | 3.60 | 3.70 | 3.55
Outside Field
Plain Summaries | 2.75 | 2.85 | 3.25 | 2.65
Qwen1.5-7B | 3.35 | 3.10 | 3.30 | 3.10
GPT-4 | 3.40 | 3.55 | 3.70 | 3.15
LLM-CLBR | 3.65 | 3.40 | 3.55 | 3.20
Table 2: Results of human evaluation, where ‘Read.’ indicates ‘Readability,’
‘Info.’ denotes ‘Information Conveyance,’ ‘Auth.’ represents ‘Authenticity,’
and ‘Intr.’ signifies ‘Interestingness’.

Methods For Comparison.

  * •

BART. Goldsack et al. (2022); Cardenas et al. (2023) used BART (Lewis et al.,
2020) for ASJ, showing strong performance.

  * •

LLMs. We test the performance of various LLMs, including both open-source and
closed LLMs, i.e., LLaMA-2-7B (Touvron et al., 2023), Gemma-7B (Google, 2024),
Mistral (7B, 8x7B, Jiang et al., 2023a), Qwen1.5 (7B, 72B, Bai et al., 2023),
LLaMA-3-8B (Meta, 2024), GPT-3.5-Turbo-1106 (OpenAI, 2023), and
GPT-4-1106-preview (Achiam et al., 2023). We prompt these LLMs and also fine-
tune the Qwen1.5-7B model for ASJ.

  * •

Our Methods. We test two versions of collaborating LLMs. One is
2$\times$Qwen1.5-7B+Qwen1.5-1.8B (LLM-CLBR, CLBR for ‘collaboration’). Another
is this combination where the journalist is replaced by a fine-tuned version
as a warm start (LLM-WS-CLBR).

Automatic Evaluation. Following Goldsack et al. (2022); Cardenas et al.
(2023), we use Coleman-Liau Index (CLI), Flesch-Kincaid Grade Level (FKGL) and
Dale-Chall Readability Score (DCRS) to automatically assess the readability.
CLI considers the count of sentences, words, and characters, while FKGL is
based on the number of sentences, words, and syllables. DCRS assesses
readability by analyzing the average sentence length and the presence of
familiar words from a list of the most commonly used words.

Human Evaluation. Automatically assessing the authenticity and informativeness
of content has been a challenging task. Cardenas et al. (2023) used QuestEval
(Scialom et al., 2021) to assess the faithfulness of ASJ-generated content,
yet the results exhibited significant variances. Therefore, human evaluation
remains the main method for such assessments. We enlist four human
participants for evaluation. All of them are including master’s students or
holders of master’s degrees, two from computer science and two from biomedical
science. Specifically, we sample 10 pairs of original papers and generated
articles in computer science from SCITech, as well as 10 pairs in biomedical
science pairs, 5 pairs from eLife and 5 pairs from PLOS.

We choose four representative methods for human evaluation: (1) plain
summaries by human writers, (2) Qwen1.5-7B generation, (3) GPT-4 generation,
and (4) generation by our LLM-Collaboration method. The human evaluation
encompasses multiple dimensions, namely Readability, Information Conveyance,
Authenticity, and Interestingness. Participants are tasked with evaluating the
articles using a 1-5 Likert scale (Likert, 1932), grounded on specific
questions. Each participant is assigned to assess all articles both in the
field they are familiar with and those they are not familiar with, to provide
a genuine evaluation from readers within the specific discipline and general
readers. Appendix D shows the details of these measures and the questionnaire
form.

Hyperparameters. We list hyperparameters in Appendix C for brevity.

###  3.2 Automatic Evaluation

The results in Table 1 show the comparison of different methods. Recent LLMs
of similar scales have shown comparable performance, surpassing the LLaMA-2
introduced in 2023. Larger LLMs such as Mixstral-8x7B and Qwen1.5-72B show
even better performance, indicating that performance improves as the model
scale increases. Additionally, the formidable LLM GPT-4 outperforms all the
other single LLMs. These findings demonstrate their performance on ASJ
consistent with the capacity of LLMs. The fine-tuning methods exhibit
competitive performance, slightly superior to prompting the open-source LLMs.
Fine-tuning a model with a larger scale (Qwen1.5-7B) outperforms fine-tuning
BART, in line with prompting.

Our collaboration with LLMs has demonstrated significant improvement over
previous methods. Interestingly, it can be observed that fine-tuning the
journalist LLM to warm up does not lead to any noticeable improvement. One
possible reason may be the decay in the ability to follow instructions after
this specialized training. Nonetheless, these empirical results show the
effectiveness of our framework that integrates multiple LLMs.

(a) SCITech

(b) eLife

(c) PLOS

Figure 4: Performance improvement over iterations, with the 0th iteration
indicating the initial writing. Scientific Paper. In infectious disease
diagnosis, results need to be communicated rapidly to healthcare professionals
once testing has been completed so that care pathways can be implemented. This
represents a particular challenge when testing in remote, low-resource rural
communities, in which such diseases often create the largest burden. Here, we
report a smartphone-based end-to-end platform for multiplexed DNA diagnosis of
malaria. The approach uses a low-cost paper-based microfluidic diagnostic
test, which is combined with deep learning algorithms for local decision
support and blockchain technology for secure data connectivity and management.
We validated the approach via field tests in rural Uganda, where it correctly
identified more than 98% of tested cases. Our platform also provides secure
geotagged diagnostic information, which creates the possibility of integrating
infectious disease data within surveillance frameworks.
---
Press Release. Researchers at the U.K.ś University of Glasgow and Ugandaś
Ministry of Health have developed easy-to-administer blood tests that could
help curtail the spread of malaria in sub-Saharan Africa. These "origami"
tests can be conducted in the field, with a blood sample taken via fingerprick
placed in a channel made from sheets of folded wax paper. The blood is
directed into three small chambers of a loop-mediated isothermal amplification
(LAMP) machine, which tests for the presence of the mosquito-borne parasite
that causes malaria. A smartphone app uses deep learning to ensure the tests
are accurate and to monitor community transmission. The results are stored on
a blockchain-based ledger to maintain privacy, with anonymized data shared
with local authorities. In Uganda, the blood tests were shown to be 98%
accurate.
Article Written by Our Method. In the fight against malaria, timely diagnosis
is crucial, especially in rural communities where these diseases often hit
hardest. Researchers have come up with a brilliant solution using smartphones!
It’s a smartphone-based system that combines a low-cost test with deep
learning and blockchain technology. First, imagine a tiny paper device that
can detect malaria DNA right from blood samples. This microfluidic test is as
simple as using a piece of paper - no fancy lab equipment needed! Then, the
results are analyzed by smart algorithms on your phone. These "AI helpers"
give instant feedback to healthcare workers without delay. All this diagnostic
data is stored securely on blockchain. This means information is protected and
cannot be tampered with. In Uganda, field tests showed that this system was
almost perfect - it correctly identified 98% of cases! The best part? The
platform also records each diagnosis’s location through geotags. This opens up
the possibility to connect local disease data to bigger surveillance networks,
helping us track and combat malaria more efficiently. So next time you hear
about remote villages facing health challenges due to lack of fast diagnosis
tools, remember: there’s hope with these smartphone-powered innovations!
Table 3: An example of the original paper, excerpts from the manually written
press release, and the article generated by our method. We highlight
statements that we regard easy to understand and explanations for technical
details. 1st Iteration. 1\. The mobile lab on your phone is a portable, paper-
based test that uses deep learning algorithms to analyze blood samples for the
presence of malaria. 2\. The end-to-end system combines a smartphone app,
allowing healthcare workers to make informed decisions about treatment without
delay. 3\. The smartphone app uses deep learning algorithms to interpret the
test results, providing accurate diagnoses over 98% in field tests. 4\. The
platform ensures secure data management by using blockchain technology, which
records transparently and cannot be tampered with.
---
2nd Iteration. 1\. Smartphones and Blockchain: The article introduces the use
of smartphones and blockchain technology in diagnosing diseases like malaria,
particularly in remote areas where traditional diagnostic methods may not be
available or accessible. 2\. Malaria Diagnosis: The low-cost paper-based
microfluidic diagnostic test uses a piece of tiny paper device to detect
malaria DNA. It allows rapid detection with the support of AI systems. 3\.
Low-Cost Test: The AI-based test is referred to as a "smart decision support"
system, which uses advanced technology such as artificial intelligence to
interpret the results right on the user’s device. 4\. Artificial Intelligence
(AI): The AI acts like a smart detective, quickly identifying potential cases
with an accuracy rate of 98% in real-world tests, enabling healthcare workers
to make informed decisions about treatment without needing fancy labs or
waiting too long.
3rd Iteration. 1\. Smartphone: A device with advanced features such as
cameras, sensors, and processors that can perform various tasks, including
medical diagnosis. 2\. Blood test: A diagnostic procedure where blood is
collected from a finger or other body part through a small needle and analyzed
for specific signs of malaria (e.g., anemia or fever). 3\. AI as personal
health assistant: An artificial intelligence system designed to assist
individuals in managing their health by providing personalized recommendations
based on medical data analysis (e.g., detecting early signs of disease). 4\.
Secure digital diary: A digital record containing sensitive information about
an individual’s health status stored on a secure blockchain network to ensure
accuracy and prevent unauthorized access or manipulation. 5\. Blockchain
technology: A distributed ledger system that allows secure sharing of data
across multiple parties without the need for intermediaries or centralized
authorities (e.g., storing patient records in hospitals). In this case, it
serves as the secure digital ledger for tracking where diseases are spreading
in remote areas due to malaria prevalence.
Table 4: Notes taken from the reader, two technical terms and their
explanations are highlighted. The notes become more detailed and comprehensive
during the iterative process. 1st Iteration. 1\. Simplify technical terms 2\.
Break down processes 3\. Emphasize benefits 4\. Conclusion statement
---
2nd Iteration. 1\. Emphasize technology: Instead of listing specific platforms
(Ethereum or Hyperledger Fabric), mention that a secure digital ledger is used
without specifying which one. 2\. Simplify medical terms: For "Plasmodium
falciparum," provide a simpler explanation or use a common term like "malaria
parasite type." 3\. Break down process: Explain how smartphone analysis works
in more detail - perhaps by describing how it compares to traditional lab
methods. 4\. Connect to equity: Highlight how this technology addresses health
disparities by providing quick diagnosis in remote areas.
3rd Iteration. 1\. Emphasize simplicity: For accessibility, rephrase "low-cost
paper-based microfluidic diagnostic test" as "affordable, easy-to-use test
with a paper strip." 2\. Explain AI in simpler terms: Instead of "AI instantly
interprets results," say "The smartphone app quickly analyzes the data to give
a diagnosis." 3\. Break down data security: Highlight that information is
stored securely on a phone or cloud server with strong passwords or
encryption. 4\. Quantify success: Mention that 98% accuracy rate is
exceptional but could be framed as an impressive achievement ("This system
detected almost all cases correctly!"). 5\. Cite real-life impact: Share
examples of how this technology has made a difference in remote communities to
connect it emotionally with readers.
Table 5: Suggestions provided by the editor LLM are becoming increasingly
specific over the iterative process.

###  3.3 Human Evaluation

For a thorough evaluation, we carry out a human assessment on four
representative methods. We request that participants assess articles relevant
to their fields as well as those in their unfamiliar fields. The results are
reported in Table 2, categorized into within-field and outside-field articles.
Notably, participants assign lower ratings to articles outside their
expertise, possibly due to the inherent comfort and familiarity bias elevating
subjective scores. Despite the evident discrepancies arising from differing
reader familiarity contexts, the method comparisons and overarching trends
appear consistent across both settings.

Interestingly, all LLM-based methods outperform the plain summaries written by
humans, probably because these summaries remain technical, providing a poor
reading experience for a wide readership. The LLM-generated content, targeted
to popular science, should be easier for both in-domain and out-of-domain
readers to read. Furthermore, our LLM-collaboration approach surpasses the
single Qwen in all dimensions, demonstrating the effectiveness of LLM
collaboration. Most importantly, our LLM collaboration method achieves the
highest readability and interest in both within-field and outside-field
reading evaluation, with high information conveyance and authenticity close to
the most advanced model GPT-4. Collectively, these findings attest to the
potency and effectiveness of our proposed approach.

##  4 Analysis

###  4.1 Ablation Study

We conduct an ablation experiment to validate the effectiveness of each
component. The results have been included in Table 6 for clarity. In the
“Reader: 1.8B $\rightarrow$ 7B” setting, we substitute the 1.8B reader model
with the 7B version. This substitution leads to a minor performance
fluctuation. On the one hand, the 7B model’s strong ability gives it greater
tolerance for low readability of content, making it harder to highlight
writing issues in articles. On the other hand, it excels in instruction
following, enhancing task execution and reducing intermediate errors in the
workflow. The dynamics between gains and losses render the use of the 7B model
as a “reader” comparably advantageous at times and disadvantageous at others.
Nevertheless, we recommend using the 1.8B model due to its higher resource
efficiency and throughput.

For $-$Reading Notes, we eliminate the requirement for the reader LLM to read
the article and make notes. Instead, we ask the editor LLM to offer
suggestions directly. For $-$Suggestions, the editor’s advice is omitted, and
the journalist revises the article based on the reader LLM’s reading. For
$-$Collaboration, the journalist revises the article based on the previous
writing without any input from the reader or the editor. As depicted in Table
6, our approach exhibits a decrease in performance when each module is
removed, underscoring the significance of each module.

###  4.2 Trend Analysis

We provide an in-depth analysis of the performance trajectory over iteration
cycles. The entire writing-reading-suggesting-revising process is carried out
for five iterations, with readability score changes depicted in Figure 4. Our
findings reveal a successive and pronounced decline in all reading difficulty
metrics over the initial three iterations. This pattern demonstrates the
efficacy of our iterative revision methodology. Following the third iteration,
the performance levels off, indicating diminishing improvements from
subsequent suggesting and editing efforts.

###  4.3 Writing Case Analysis

To facilitate an intuitive comparison of our method, we present a case study
on one writing sample. As shown in Table 3, our method can generate articles
that are more readable, with concise and vivid expressions, along with
explanations for technical details. For instance, our generated article states
that “timely diagnosis is crucial” rather than “results need to be
communicated rapidly to healthcare professionals once testing has been
completed so that care pathways can be implemented”, making it more brief and
accessible for readers to grasp the research objective. Moreover, our
generated article details that the proposed system is a “smartphone-based
system that combines a low-cost test with deep learning and blockchain
technology” rather than “smartphone-based end-to-end platform for multiplexed
DNA diagnosis of malaria”, enhancing the comprehensibility for a broader
audience.

###  4.4 Case of Reading Notes

We also present a case study on the notes taken by the reader LLM during the
first three iterations. As evidenced in Table 4, the notes offer more detailed
explanations for technical jargon than those in preceding iterations. For
example, the technical terms “AI” and “blockchain” are more thoroughly
explained in the third iteration than in the first two iterations. This
phenomenon suggests that as the readability of writing improves, readers can
acquire deeper and more elaborate insights, aligning with human reading
behaviours.

###  4.5 Suggestions for Revision

We further examine the advice given by the LLM editor for the revision over
the iterative process, as exemplified in Table 5. In the initial stage, the
editor offers general advice. Following the revision, the advice becomes more
specific and detailed. Subsequent rounds of suggestions highlight specific
content in the article and recommend revisions based on various aspects. In
the 3rd iteration, the feedback from the editor becomes more detailed and
specific, with specific adjustments from the original version to the revised
content. This revision behaviour is similar to real-world science journalism,
showcasing the efficacy of our framework and the social intelligence of LLMs.

##  5 Related Work

Automatic Science Journalism. ASJ has gained increasing interest in recent
years. Dangovski et al. (2021) created a parallel corpus and provided a
sequence-to-sequence method to generate news summaries from scientific
articles. However, this dataset is not available to the public because of the
licensing restriction. Goldsack et al. (2022) released two corpora, primarily
focused on the biomedical and life science domains. Similarly, they employed a
standard sequence-to-sequence model for such tasks. Cardenas et al. (2023)
constructed a dataset in various scientific fields and integrated the
discourse structure of papers with their metadata to guide the generation.
These methods of fine-tuning on small models can provide a good match with the
reference, but there is still room for improvement in readability. In this
work, we present a novel approach that integrates LLMs acting as agents for
enhancing readability.

Large Language Models. With the emergence of the LLM, advanced models (Achiam
et al., 2023; Touvron et al., 2023; Anthropic, 2023; Jiang et al., 2023a; Bai
et al., 2023) have shown performance comparable to that of humans in a variety
of real-world tasks. Park et al. (2023) leveraged LLMs for social simulation,
showing communication and collaboration between LLM-based agents. Liu et al.
(2023) built an LLM-collaboration architecture for enhancing reasoning and
code generation tasks. Qian et al. (2023) used different LLM-based agents
throughout the development process for software engineering. This body of work
has demonstrated the strong collective intelligence of LLMs. Inspired by this
collection of work, we utilize LLMs as communicative agents, writing, reading,
and revising scientific articles to make content accessible to the general
audience through a process resembling real-world practice.

##  6 Conclusion

This study presents the first examination of LLM collaboration for automatic
science journalism aimed at general readers. Initially, an LLM functions as a
journalist by composing an explanatory article for the general public.
Subsequently, a smaller LLM, acting as a general audience, reads these
articles and takes notes, potentially helping reveal the low readability of
the article content. An LLM editor assesses the reader’s notes and offers
suggestions for improvement. Following the editor’s advice, the journalist
revises the manuscript, which is then passed to the reader again to continue
the iterative process. Extensive experiments are conducted to evaluate the
effectiveness of our framework, including both automatic metrics and human
evaluation. In comparison to directly prompting and fine-tuning various LLMs,
our method achieves the highest readability while maintaining high quality.
Additionally, we offer an in-depth analysis, which further validates our
method and provides insights into designing LLM-driven ASJ systems.

## Limitations

We identify the following four limitations of our work. First, following
previous work, we have defined ASJ as the process of transforming a single
paper into an article intended for a general audience. In practice, a popular
science article may encompass multiple studies. Therefore, an extension can be
to address the challenge of consolidating several papers into a single
article. Second, we have utilized some statistical indexes for automatic
assessment, but these statistical measures may miss semantic information. LLM-
driven evaluation could offer a solution. While there remains a gap between
LLMs and humans in evaluating text on readability and authenticity, efforts
could be made to minimize this gap. Third, given our exploratory approach in
utilizing LLMs for ASJ, we strategically chose abstracts as input to maintain
both simplicity and resource efficiency. Nevertheless, long-context ASJ is an
intriguing task potentially with new challenges to address. Lastly, all
components of our framework are powered by LLMs. In addition to our efforts to
make each LLM simulate humans, it will be interesting to incorporate genuine
human preferences to enhance the generated content.

## Ethics Statement

The experiments in this study were conducted on publicly available datasets.
Any information involving privacy was removed. All annotators have been
properly paid for their efforts.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, and Sam Altman et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Allan (2011) Stuart Allan. 2011.  Introduction: Science journalism in a digital age.  _Journalism_ , 12(7):771–777.
  * Angler (2017) Martin Angler. 2017.  _Science journalism: an introduction_.  Routledge.
  * Anthropic (2023) Anthropic. 2023.  Claude anthropic.  https://www.anthropic.com/claude.
  * August et al. (2024) Tal August, Kyle Lo, Noah A Smith, and Katharina Reinecke. 2024.  Know your audience: The benefits and pitfalls of generating plain language summaries beyond the" general" audience.  _arXiv preprint arXiv:2403.04979_.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_.
  * Cardenas et al. (2023) Ronald Cardenas, Bingsheng Yao, Dakuo Wang, and Yufang Hou. 2023.  ‘don’t get too technical with me’: A discourse structure-based framework for automatic science journalism.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 1186–1202.
  * Caselli et al. (2015) Tommaso Caselli, Piek Vossen, Marieke van Erp, Antske Fokkens, Filip Ilievski, Rubén Izquierdo, Minh Le, Roser Morante, and Marten Postma. 2015.  When it’s all piling up: investigating error propagation in an nlp pipeline.  In _WNACP@ NLDB_.
  * Chan et al. (2023) Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023.  Chateval: Towards better llm-based evaluators through multi-agent debate.  _arXiv preprint arXiv:2308.07201_.
  * Dangovski et al. (2021) Rumen Dangovski, Michelle Shen, Dawson Byrd, Li Jing, Desislava Tsvetkova, Preslav Nakov, and Marin Soljačić. 2021.  We can explain your research in layman’s terms: Towards automating science journalism at scale.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 12728–12737.
  * Desmond et al. (2024) Michael Desmond, Zahra Ashktorab, Qian Pan, Casey Dugan, and James M Johnson. 2024.  Evalullm: Llm assisted evaluation of generative outputs.  In _Companion Proceedings of the 29th International Conference on Intelligent User Interfaces_ , pages 30–32.
  * Dziri et al. (2023) Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Xiang Ren, Allyson Ettinger, Zaïd Harchaoui, and Yejin Choi. 2023.  Faith and fate: Limits of transformers on compositionality.  In _Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023_.
  * Goldsack et al. (2022) Tomas Goldsack, Zhihao Zhang, Chenghua Lin, and Carolina Scarton. 2022.  Making science simple: Corpora for the lay summarisation of scientific literature.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10589–10604.
  * Google (2024) Google. 2024.  Gemma: Introducing new state-of-the-art open models.
  * Göpfert (2008) Winfried Göpfert. 2008.  The strength of pr and the weakness of science journalism.  In _Journalism, science and society_ , pages 227–238. Routledge.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_.
  * Jiang et al. (2023a) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023a.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Jiang et al. (2023b) Gongyao Jiang, Shuang Liu, Meishan Zhang, and Min Zhang. 2023b.  A pilot study on dialogue-level dependency parsing for chinese.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 9526–9541.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Likert (1932) Rensis Likert. 1932.  A technique for the measurement of attitudes.  _Archives of psychology_.
  * Lin et al. (2023) Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. 2023.  Awq: Activation-aware weight quantization for llm compression and acceleration.  _arXiv preprint arXiv:2306.00978_.
  * Liu et al. (2023) Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. 2023.  Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization.  _arXiv preprint arXiv:2310.02170_.
  * Meta (2024) Meta. 2024.  Introducing Meta Llama 3: The most capable openly available LLM to date.
  * OpenAI (2023) OpenAI. 2023.  Introducing ChatGPT.
  * Park et al. (2023) Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023.  Generative agents: Interactive simulacra of human behavior.  In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_ , pages 1–22.
  * Qian et al. (2023) Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. 2023.  Communicative agents for software development.  _arXiv preprint arXiv:2307.07924_.
  * Scialom et al. (2021) Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021.  Questeval: Summarization asks for fact-based evaluation.  _Empirical Methods in Natural Language Processing_.
  * Talebirad and Nadiri (2023) Yashar Talebirad and Amirhossein Nadiri. 2023.  Multi-agent collaboration: Harnessing the power of intelligent llm agents.  _arXiv preprint arXiv:2306.03314_.
  * Tkachenko et al. (2020-2022) Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. 2020-2022.  Label Studio: Data labeling software.  Open source software available from https://github.com/heartexlabs/label-studio.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wasi et al. (2024) Azmine Toushik Wasi, Rafia Islam, and Raima Islam. 2024.  Llms as writing assistants: Exploring perspectives on sense of ownership and reasoning.  _arXiv preprint arXiv:2404.00027_.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Wu et al. (2018) Lijun Wu, Xu Tan, Di He, Fei Tian, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2018.  Beyond error propagation in neural machine translation: Characteristics of language also matter.  _arXiv preprint arXiv:1809.00120_.
  * Yang et al. (2023) Zeyuan Yang, Peng Li, and Yang Liu. 2023.  Failures pave the way: Enhancing large language models through tuning-free rule accumulation.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 1751–1777.
  * Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022.  React: Synergizing reasoning and acting in language models.  _arXiv preprint arXiv:2210.03629_.
  * Yuan et al. (2022) Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022.  Wordcraft: story writing with large language models.  In _27th International Conference on Intelligent User Interfaces_ , pages 841–852.
  * Zhang et al. (2024) Meishan Zhang, Gongyao Jiang, Shuang Liu, Jing Chen, and Min Zhang. 2024.  Llm–assisted data augmentation for chinese dialogue–level dependency parsing.  _Computational Linguistics_ , pages 1–24.
  * Zheng et al. (2024a) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024a.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _Advances in Neural Information Processing Systems_ , 36.
  * Zheng et al. (2024b) Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, and Yongqiang Ma. 2024b.  Llamafactory: Unified efficient fine-tuning of 100+ language models.  _Preprint_ , arXiv:2403.13372.

##  Appendix A Prompts for LLMs

We list all prompts in Table 7. All prompts follow a similar format. First, we
assign a role to each LLM agent by a sentence. We then specify the task and
background in one or two sentences. Next, we give each LLM step-by-step
instructions. After that, we input the rules to be followed for each LLM.
Finally, the format of the output is specified as “markdown” style to
facilitate the extraction and support the information flow among LLMs. In our
preliminary study, this pattern works well in prompting various LLMs, with
strong task completion and format adherence. Algorithm 1 presents the workflow
of our framework.

1:a scientific paper $\boldsymbol{x}$; journalist $\mathcal{J}$; reader
$\mathcal{R}$; editor $\mathcal{S}$; number of iterations $t$.

2:$\boldsymbol{p}_{0}\leftarrow\mathcal{J}(\boldsymbol{x})$ $\triangleright$
Initialization

3:for $i=1$ to $t$ do

4: $\boldsymbol{r}\leftarrow\mathcal{R}\left(\boldsymbol{p}_{i-1}\right)$
$\triangleright$ Reader’s notes

5: $\boldsymbol{s}\leftarrow\mathcal{S}\left(\boldsymbol{p_{i-1},r}\right)$
$\triangleright$ Editor’s suggestions

6:
$\boldsymbol{p}_{i}\leftarrow\mathcal{J}\left(\boldsymbol{x,p_{i-1},s}\right)$
$\triangleright$ Revision

7:end for

8:return $\boldsymbol{p}_{t}$

Algorithm 1 LLM-Collaboration for ASJ

##  Appendix B Failed Attempts

This appendix outlines our unsuccessful attempts. We hope that it will save
time for other researchers. First, as a commonly used mechanism in LLM agents,
reflection can support iterative enhancement by consolidating prior
experiences (Yao et al., 2022; Park et al., 2023; Yang et al., 2023). In our
pilot experiments, however, this approach did not succeed in refining the
journalist’s writing. One potential explanation could be that the varied
nature of journalistic content necessitates more targeted revision
instructions, whereas summarizing writing experiences results in general
writing guidance.

Within our framework, an LLM acts as an audience, reading the written article
and taking notes. How about having this audience perform the reading
comprehension task instead of taking notes? Intuitively, it can also
demonstrate the reader’s understanding, and then induce the content
complexity. Preliminary investigations, however, revealed that this approach
yields lesser efficacy compared to the note-taking strategy. One possible
reason might be that the quality of question generation greatly affects the
efficiency of the whole workflow. Besides, asking a fixed number of questions
narrows the expanse of textual exploration, thereby constricting the
comprehensive perception of content complexity.

##  Appendix C Dataset and Hyperparameters

Table 6 shows some statistics of the three datasets used in our experiments.
_SCITech(News)_ is released by Cardenas et al. (2023), who gathered press
releases from ACM Technews as well as their source articles from various
publishers, involving fields of computer science, engineering, astrophysics,
biology, and others. _eLife_ is an open-access journal that focuses on
biomedical and life sciences. Goldsack et al. (2022) collected some eLife
articles as well as digests written by expert editors based on both the
article itself and questions answered by the author. Similarly, _PLOS_ hosts
journals across areas of science and medicine. Some of these articles, also
collected by Goldsack et al. (2022), come with the author’s summary. For
resource saving, the original paper’s abstract serves as the scientific
content input.

Statistic | SCITech | eLife | PLOS
---|---|---|---
# pairs | 2431 | 4828 | 27525
# $\text{words}^{\text{ori}}$ | 216.8 | 166.3 | 268.3
# $\text{sentences}^{\text{ori}}$ | 5.7 | 6.8 | 10.2
# $\text{words}^{\text{pln}}$ | 176.1 | 347.6 | 175.6
# $\text{sentences}^{\text{pln}}$ | 7.9 | 15.7 | 7.8
Table 6: Statistics of benchmark datasets. Words and sentences are average
values. The “ori” superscript indicates abstracts of original papers, and
“pln” represents plain summaries written by authors or journalists.

Our local LLM service runs on a machine with eight GTX 4090 GPUs. We utilize
the Huggingface platform (Wolf et al., 2019) for downloading and loading
checkpoints. For rapid inference and memory efficiency, we utilize the vLLM
library222https://docs.vllm.ai to develop API services. We deploy agents from
the Qwen-1.5 series, for their good performance and diverse model scales Bai
et al. (2023). In particular, Qwen-1.5-7B is employed for the steps of
writing, providing suggestions, and revision, whereas Qwen-1.5-1.8B serves as
the reader for taking notes. To improve memory efficiency, we implement
activation-aware weight quantization (AWQ, Lin et al. 2023) for model
quantization. For fine-tuning, we utilize LoRA (Hu et al., 2021) with Llama-
Factory (Zheng et al., 2024b), adopting the default setting with the number of
the epochs set to 10. We use the default temperature setting and empirically
set top_p to 0.4, frequency penalty and repetition penalty to 1, ensuring the
stability of the LLMs’ output while retaining diversity. We iterate five times
and empirically select the output from the third iteration as the final
result. The maximum number of tokens in the model output is 4096.

##  Appendix D Details of Human Evaluation

We create a questionnaire for human evaluation using a 1-5 Likert scale, as
shown in Figure 5. All participants were informed that their assessments would
be used for research purposes. We utilize Label Studio (Tkachenko et al.,
2020-2022) to construct the annotation platform. Initially, participants
indicate their familiarity with the given topic. They are then tasked with
answering 4 questions related to Readability, Information Conveyance,
Authenticity, and Interestingness. Readability assesses how easily the article
can be read, serving as a supplementary and further validation of automated
evaluation. Information conveyance determines if the rewritten content
accurately and comprehensively conveys the information from the original
paper. Similarly, authenticity assesses the correctness of the content. A
high-quality article should contain minimal factual or common sense errors to
avoid misleading readers. The level of interest is also a crucial factor;
content of high appeal will attract more readers.

##  Appendix E Use of AI Assistants

We use ChatGPT for correcting grammar and improving expressions in this
manuscript.

Journalist. You are a science journalist for general audiences. Given a
paper’s summary, you are assigned to rewrite it into a short understandable
article for general audiences.
---
Follow the rules strictly:
\- Keep short yet informative.
\- The output format:
## Article
…
Reader. You are a general reader. Given a popular science article, please read
it carefully and take some notes.
Please take the following steps:
1\. First, extract all technical terms with their context from the article.
2\. Then, explain the technical terms based on their context.
Follow the rules strictly:
\- Extraction should mention the specific location of each technical term in
the article.
\- Explanation should be first extracted from the article; if not found, it
can be some common-sense or specialized knowledge.
\- Extraction and explanation should be in points, like "1…2…3…".
\- The output format:
### Extraction
1\. …
2\. …
…
### Explanation
1\. …
2\. …
Editor. You are a senior editor. Here are a scientific paper summary, and a
short popular science article. A general reader has read the science article
and takes some notes.
Please take the following steps:
1\. First, evaluate the **reader’s notes** based on these factors: content
accuracy, lexical and technical complexity, and information conveyance (from
the original content).
2\. Then, based on the above evaluation, list some brief yet informative
writing advice that may benefit the popular science article, to make the
article easier for general readers without specialized knowledge to read and
understand. Specifically, the advice should benefit these factors of the
article:
a) Content Accuracy: The factual correctness, scientific validity, and absence
of errors in the general popular science article.
b) Accessibility: Higher accessibility means less technical, more readable and
interestingly, etc. c) Information Conveyance: How effectively key information
from the original paper is transferred to the popular science article.
Follow the rules strictly:
\- Evaluation and advice sections should be in points, like "1…2…3…".
\- Each advice should not go beyond the fact of original paper, but can be
some common-sense or specialized knowledge.
\- Each advice should be targeted at one specific aspect of the article.
\- Don’t suggest visualization, references and links.
\- Suggest explanations rather than content additions.
\- The output format:
## Evaluation for reader’s notes
\- Content accuracy of reader’s notes: …
\- Lexical and technical complexity of reader’s notes: …
\- Information conveyance of reader’s notes: …
## Advice
1\. …
2\. …
Revision You are a science journalist for general audiences. Given the paper
summary and a short summary of the popular science article, you are assigned
to rewrite the popular science summary for general audiences, who have no
specialized knowledge on this field. There are some writing advice.
Please take the following steps:
1\. Choose and refine the most relevant and suitable advice for writing
improvement.
2\. Then, based on the refined advice and the paper summary, rewrite the
popular science article.
Follow the rules strictly:
\- Keep short yet informative.
\- Don’t include visualization, references and links.
\- Revision must not go beyond the fact of original paper, but can be with
some additional common-sense or professional knowledge for explanation.
\- The output format:
## Improvement
…
## Revised Article
…
Table 7: Prompts for each LLM agent. Figure 5: The questionnaire for
participants to evaluate articles.