# Hierarchical Catalogue Generation for Literature Review: A Benchmark

Kun Zhu  Harbin Institute of Technology<EMAIL_ADDRESS>,  Xiaocheng Feng
Harbin Institute of Technology<EMAIL_ADDRESS>,  Xiachong Feng  Harbin
Institute of Technology<EMAIL_ADDRESS>,  Yingsheng Wu  Harbin
Institute of Technology<EMAIL_ADDRESS>and  Bing Qin  Harbin Institute of
Technology<EMAIL_ADDRESS>
###### Abstract.

Multi-document scientific summarization can extract and organize important
information from an abundant collection of papers, arousing widespread
attention recently. However, existing efforts focus on producing lengthy
overviews lacking a clear and logical hierarchy. To alleviate this problem, we
present an atomic and challenging task named Hierarchical Catalogue Generation
for Literature Review (HiCatGLR), which aims to generate a hierarchical
catalogue for a review paper given various references. We carefully construct
a novel English Hierarchical Catalogues of Literature Reviews Dataset (HiCaD)
with 13.8k literature review catalogues and 120k reference papers, where we
benchmark diverse experiments via the end-to-end and pipeline methods. To
accurately assess the model performance, we design evaluation metrics for
similarity to ground truth from semantics and structure. Besides, our
extensive analyses verify the high quality of our dataset and the
effectiveness of our evaluation metrics. Furthermore, we discuss potential
directions for this task to motivate future research.

scientific document processing, multi-document summarization, datasets,
metrics

††ccs: Information systems Summarization

##  1\. Introduction

With the development of technology, today’s researchers can publish not only
in traditional venues, such as conferences and journals but also in e-preprint
libraries and mega-journals that offer fast publication times (Fire and
Guestrin, 2019). The reduced time required for the publication process and the
ease of access to content have further increased the speed at which
researchers can share and publish their research to some extent. The
phenomenon is better reflected in the occurrence of hot events. For example,
statistics show that more than 50,000 publications occurred in 2020 in
response to COVID-19 (Wang and Lo, 2021).

The rapid increase in scientific papers has overwhelmed researchers with
considerable reading. To meet the growing need of researchers to gain an
understanding of a particular field quickly, the automatic summarization of
scientific literature has received a lot of attention. There are many forms of
this scientific document summarization task, such as citation sentence
generation (Xing et al., 2020) (Luu et al., 2020) (Ge et al., 2021) (Wu et
al., 2021), related work generation (Hoang and Kan, 2010) (Hu and Wan, 2014)
(Li et al., 2022) (Wang et al., 2022), and survey generation (Mohammad et al.,
2009) (Jha et al., 2015) (Shuaiqi et al., 2022).

While some work has been done on scientific document summarization, these
efforts either generate short texts or long, disorganized texts, which do not
generate a real literature review with a hierarchical structure. A literature
review is a survey of research papers that provides an overview of a
particular topic, which provides a comprehensive look at what has been done on
the subject and by whom (Webster and Watson, 2002). A literature review is a
valuable tool for beginners who need an overview of knowledge in a field
(Booth et al., 2021). Producing a literature review is a demanding task, which
faces the challenge of understanding a large number of articles and giving a
structured and complete paper.

Figure 1. An example for a literature review and its catalogue.

The catalogue is highly logical and applicable and can be seen as the soul of
a review, representing the author’s understanding and organization of existing
research. The catalogue also helps the reader better understand a particular
field’s division and find interesting content. Figure 1 shows the review paper
titled ’A Survey of Domain Adaptation for Neural Machine Translation’ and a
part of its catalogue. In this paper, we propose a new, extremely challenging
task of generating a hierarchical catalogue for literature review. This task
is also a first step towards automatic review generation. There are three
challenges to this task:

  1. (1)

Obtaining a dataset suitable for the task;

  2. (2)

Generating a hierarchically structured text like a catalogue;

  3. (3)

Evaluating the quality of the generated catalogue.

Table 1. Comparison of our HiCaD dataset to other multi-document scientific
summarization tasks and their datasets. ”Pairs” means the number of examples.
”Refs” stands for the average number of input papers for each sample. ”Sents”
and ”Words” indicate the average number of words and sentences in input or
output calculated by concatenating all input sources. (Dataset ”Multi-
XScience” does not present the number of sentences of input and output in the
paper.) ”Form” is the form of the output text.

Datasets | Task | Pairs | Refs | #Sents | #Words | Form | #Sents | #Words
---|---|---|---|---|---|---|---|---
(input) | (input) | (output) | (output)
Multi-XScience | Related Work | 40,828 | 4.4 | - | 778.1 | Paragraph | - | 116.4
BigSurvey-MDS | Survey Introduction | 4,478 | 76.3 | 450.1 | 11,893.1 | Paragraph | 38.8 | 1,051.7
HiCaD | Survey Catalogue | 13,853 | 80.6 | 448.8 | 14,653.6 | Hierarchical Catalogue | 19.3 | 80.5

As there is currently no existing dataset for catalogue generation, the first
step in developing HiCatGLR is to construct a dataset to serve as a benchmark
for the approach. Since it is not possible to extract high-quality catalogues
from the pdf format of the papers, we gather survey papers’ catalogues by
html5 format111https://ar5iv.labs.arxiv.org/ and their references papers’
abstracts from Semantics Scholar222https://www.semanticscholar.org/. After
several filtering stages, plus one round of manual screening, we obtain the
final dataset with 13.8k pairs.

Taking the popular scientific dataset as an example, we present the
characteristics of the different multi-document scientific summarization tasks
in Table 1. Dataset Multi-Xscience was proposed by Lu et al. (2020), which
focuses on writing the related work section of a paper based on its abstract
and the average of 4.4 articles cited. Dataset BigSurvey-MDS is the first
large-scale multi-document scientific summarization dataset using review
papers’ introduction section as target (LIU et al., 2022). In contrast,
previous massive dataset work has targeted the section of related work. While
both BigSurvey and our HiCatGLR task have more than 70 references, resulting
in over 10,000 words of input, the output is still a standard text paragraph,
similar to Multi-Xscience. The natural difference between our task and other
tasks is that our output contains several small levels. This is the biggest
challenge during the generation process.

For the second subtask, we study the influence of one-step generation and
step-by-step generation on the results considering the structural nature of
the catalogue. According to the statistics of our dataset, each survey paper
has an average of 80.6 reference papers in the constructed dataset, resulting
in an average input length of 14654, much longer than the input limit of
typical language models. This more extended input can lead to a more dispersed
distribution of crucial content and makes existing neural models exponentially
more complex in terms of time and space. We carried out experiments using
transformer models with sparse attention and multi-encoder models that can
handle long texts.

The third subtask is designing a reasonable set of metrics to assess the
generated catalogues. We designed two types of evaluations to evaluate the
goodness of the generation synthetically. Catalogue Edit Distance Similarity
(CEDS) measures the similarity to ground truth. Catalogue Quality Estimate
(CQE) measures the degree of catalogue standardization from the rate of
catalogue template words in results.

We tested the two baseline methods and the other baselines with the proposed
evaluation metrics. The one-step end-to-end generation method generally works
better, while the step-by-step generation approach allows for more focus on
generating target-level headings. Through consistency tests with ROUGE and
human evaluation, we demonstrate the validity of CQE and CEDS.

In summary, our contribution is threefold:

  * •

We propose a new task, Hierarchical Catalogue Generation for Literature Review
(HiCatGLR), and build the first dataset for it.

  * •

We benchmark various methods on our dataset and find the step-by-step approach
can bring extra performance gains for headings at specific levels.

  * •

We design a series of evaluation metrics and validate their effectiveness.

##  2\. Related Work

Automated Scientific Document Processing (SDP) The growth rate of published
articles continued to increase, averaging between 5 and 6.5% in 2021 (An,
2021). The exponential increase in published
papers333https://arxiv.org/stats/monthly_submissions has placed a heavy burden
on researchers. As people try to navigate the ever-growing flood of data,
Automated Scientific Document Processing (SDP) using Natural Language
Processing (NLP) is one way that has proven useful. SDP can be broadly
classified into four categories: extraction (Tkaczyk et al., 2015)
(Viswanathan et al., 2021), analysis (Abu-Jbara et al., 2013) (Shpynov and
Nikolai, 2021), recommendation (Färber and Jatowt, 2020) (Bedi et al., 2022)
and generation (Hoang and Kan, 2010) (Mohammad et al., 2009) (Yasunaga and
Lafferty, 2019).

Survey generation Our work belongs to the multi-document summarization task of
generation, which aims to reduce the reading burden of researchers. Survey
generation is the most challenging task in multi-document scientific
summarization. The early work was mainly an extractive approach based on
various content selection ways. Mohammad et al. (2009) select the sentences in
the text of the citations and other forms of input, such as the abstracts or
full texts of the source articles with Trimmer, LexRank, C-LexRank, and C-RR.
Jha et al. (2015) propose a new HITS-based content model to rank input
sentences and picks the top sentences to 2,000 characters. The results
generated using the above unsupervised content selection models lack coherence
and have significant duplication problems. To be more precise, these results
do not have the characteristic that a literature review should be well
organized.

To make the generated results have a clear hierarchy, Hoang and Kan (2010)
additionally inputs an associated topic hierarchy tree that describes a target
paper’s topics to drive the creation of an extractive related work section.
Sun and Zhuge (2019) proposes a template-based framework for survey paper
automatic generation. It allows users to compose a template tree that consists
of two types of nodes, dimension node and topic node. Shuaiqi et al. (2022)
trains classifiers based on Bert (Devlin et al., 2018) to conduct category-
based alignment, where each sentence from academic papers is annotated into
five categories: background, objective, method, result, and other. Next, each
research topic’s sentences are summarized and concatenated together. However,
these template trees are either inflexible or require the users to have some
background knowledge to give, which could be more friendly to beginners. Also,
this defeats the original purpose of automatic summarization to aid reading.

Text structure Generation There are some efforts involving the automatic
generation of text structures. Xu et al. (2021) employs a hierarchical model
to learn structure among sentences in paragraphs, selecting informative
sentences to generate the survey. But this structure is not explicit. Tan et
al. (2021) generate long texts by producing domain-specific keywords and then
gradually refining them into complete passages. The keywords can be considered
a structure’s guide for the subsequent generation. Fu et al. (2022) generate
slides from a multi-modal document. In the process of summarizing and placing
an object, the model realizes the generation of slide titles, i.e. structures
of one target document. These efforts don’t really model the relationships
between multiple documents, which is far from the hierarchical structure of a
review paper. This paper focuses on the generation of a hierarchical catalogue
for literature review.

##  3\. Datasets: HiCaD

We now describe the HiCaD dataset, including the task definitions, data
source, and processing procedures. We also conduct descriptive statistics and
initial analysis.

###  3.1. Definitions

The input of this task is the combination of the title $t$ of the target
survey $S$ and representative information of reference articles
$\\{R^{1},R^{2},...,R^{n}\\}$, where $n$ is the number of reference articles
cited by $S$. Combining the cost of data collection and experimental effects,
we finally chose the reference’s abstract as representative information.

The target is the catalogue $C=\\{c^{1},...,c^{k}\\}$ of $S$ , where $k$ is
the number of catalogue items. $c^{i}$ is an item in the catalogue consisting
of a level mark $l_{m}^{i}$ and the context $c_{context}^{i}$.
$m\in\\{1,2,3\\}$, which represents the level of the catalogue item, and
$c_{context}^{i}$ contains $\\{w^{i}_{1},w^{i}_{2},...,w^{i}_{p}\\}$, where
$p$ is the number of words and $p\leq 350$. Abstracts that exceed the length
limit will be truncated. As shown in Figure 1, ”Domain Adaptation for NMT” is
the first-level heading, ”Model Centric” is the second-level heading, and
”Training Objective Centric” is the third-level heading. In our experiment, we
only keep up to the third-level headings and do not consider further lower
level-headings.

###  3.2. Dataset Construction

Our dataset is collected by combining information from two sources: arXiv.org
and Semantics Scholar. We first select papers444These survey papers’ metadata
are obtained from Kaggle up to the end of November 2022.
https://www.kaggle.com/datasets/Cornell-University/arxiv. including the words
”survey” and ”review” in the title as an initial filter, and then remove
papers in which the paper title includes ”book review” and ”comments”. We
finally obtained 24,292 papers that we considered to be review papers.

We used the crawler to get all 24,292 papers in PDF format according to arxiv-
id. We tried to obtain the catalogue directly from the PDF file using
pypdf555https://pypi.org/project/pypdf/. For a portion of PDF files where
catalogues cannot be extracted directly, we converted the PDF to
text666https://github.com/kermitt2/xpdf-4.00 or json (Lo et al.,
2020)777https://github.com/allenai/s2orc-doc2json file then extracted
catalogues from the converted file. This processing method produces low-
quality catalogues due to dropped structural information when converting, so
we used ar5iv.org to get the papers in HTML format. This website processes
articles from arXiv.org as responsive HTML5 web pages by converting from TeX
with LaTeXML. Some authors did not upload the LaTeX code, so we did not
consider their papers leaving us with 16,388 pieces.

We obtained the original catalogues for the target by cleaning up the HTML
files and then replaced the serial number from the heading using regex with
the level mark $<L_{i}>$. As for the source, we collated the list of reference
papers and kept only the valid papers where titles and abstracts existed. We
convert all words to lowercase for subsequent generation and evaluation.
Finally, after removing data with less than five catalogue items and less than
ten valid references, we obtained 13,853 source-target pairs and split them
into training (80%), validation (10%), and test (10%) sets.

###  3.3. Dataset Statistics and Analysis

To measure how abstractive our target catalogues are, we present the
proportion of novel n-grams in the target summaries that do not appear in the
source (Table 2). The abstractiveness of HiCaD is lower than that of
BigSurvey-MDS and Multi-XScience, which suggests that writing catalogues focus
on extracting keywords from references. This conclusion is in line with the
common sense that literature review is closer to reorganization than
innovation.

Table 2. The proportion of novel n-grams in target summaries across different
summarization datasets.

Datasets | % of novel n-grams in target summary
---|---
unigrams | bigrams | trigrams | 4-grams
Multi-XScience | 42.33 | 81.75 | 94.57 | 97.62
BigSurvey-MDS | 37.39 | 76.46 | 93.87 | 98.04
HiCaD | 17.51 | 60.54 | 84.31 | 93.78

We also analyzed the share of each level of the catalogue in the whole
catalogue. Figure 2 shows the value and proportional relationship of the
average number of catalogue items as well as average word length at different
levels. It can be seen that the second-level headings have the most weight,
being 52.68% of the average number and 54.73% of the average word length.

(a) item number

(b) word length

Figure 2. The value and proportional relationship of the average number of
catalogue items as well as average word length at different levels.

Table 3 shows the weight of the headings at each level in the catalogue from
the perspective of word coverage. ROUGE (Lin, 2004) is a traditional automatic
evaluation method counting the number of overlapping units, such as n-gram and
word sequences, between the generated summary and reference summaries. We
calculate ROUGE 888The ROUGE scores in this paper are all computed by
ROUGE-1.5.5 script with the option ”-c 95 -r 1000 -n 2 -a -m” scores between
different levels of headings ($L_{1},L_{2},L_{3}$) and the general catalogue
”Total”. Similar to the above, the secondary headings have the highest Rouge-1
score of 68.7 for the entire catalogue. Moreover, the Rouge-1 score of 18.0
for the first and second-level headings indicates some repetition between the
first and second levels.

Table 3. ROUGE scores between different levels of headings. ”Total” represents
the whole catalogue.

| $L_{1}$ | $L_{2}$ | $L_{3}$
---|---|---|---
| R-1/R-2/R-L | R-1/R-2/R-L | R-1/R-2/R-L
$L_{1}$ | / | 18.0/5.2/17.6 | 8.9/2.6/8.7
$L_{2}$ | 18.0/5.2/17.6 | / | 5.1/1.0/5.0
$L_{3}$ | 8.9/2.6/8.7 | 5.1/1.0/5.0 | /
Total | 41.2/34.3/41.2 | 68.7/63.8/68.7 | 22.2/20.5/22.2

##  4\. Metrics

To evaluate the quality of generated catalogues, we propose two evaluation
methods, Catalogue Quality Estimate (CQE) and Catalogue Edit Distance
Similarity (CEDS). The former takes a textual perspective, while the latter
integrates text and structure. We use these evaluation methods to assess the
quality of the generated catalogues and the gap between the generated
catalogues and ground truth, respectively.

###  4.1. Catalogue Quality Estimate

Catalogue Quality Estimate (CQE) evaluation metric measures whether the
generated answer is similar to the catalogue through template words
statistics.

Table 4. All template words used. introduction overview methodology method(s)
---
preliminaries preliminary background definition(s) sources
purpose related work data dataset(s) outlook eassy future
summary benefit(s) advantage(s) disadvantage(s) challenge(s)
application(s) notation(s) result(s) discussion analysis
observation(s) conclusion(s) reference

There are some fixed templates in the catalogue, such as ”introduction”,
”method”, ”conclusion”, and so on. We collated a list of templates and
calculated the percentage of templates in the catalogue items as CQE. Table 4
lists all template words. The catalogue templates rate in the test set is
$8.3\%$.

###  4.2. Catalogue Edit Distance Similarity

Catalogue Edit Distance Similarity (CEDS) measures the semantic similarity and
structural similarity between the generated and ground truth. Traditional
automatic evaluation methods in summarization tasks (such as ROUGE (Lin, 2004)
and BERTScore (Zhang* et al., 2020)) can only measure similarity from
semantics. However, catalogues are texts with a hierarchical structure, so the
level of headings in catalogues should also be checked.

We borrow the way edit distance that is used in the ordered labelled tree
similarity metric. An ordered labelled tree is one in which the order from
left to right among siblings is significant (Zhang and Shasha, 1989). The tree
edit distance (TED) between two ordered labelled trees $T_{a},T_{b}$ is
defined as the minimum number of node edit operations that transform $T_{a}$
into $T_{b}$ (Paaßen, 2018). There are three edit operations on the ordered
labelled tree, each with a distance of one:

  * •

deletion: delete a node and connect its children to its parent, maintaining
the order.

  * •

insertion: insert a node between an existing node and a sub-sequence of
consecutive children of this node.

  * •

modification: rename the label of a node.

The tree edit distance similarity (TEDS) is defined as follows:

$TEDS(T_{a},T_{b})=100*\left(1-\frac{TED(T_{a},T_{b})}{max(|T_{a}|,|T_{b}|)}\right)$

where $|T|$ is the number of nodes in the ordered labelled tree $T$.

Like TED, the definition of catalogue edit distance (CED) is the minimum
distance of item edit operations (deletion, insertion, modification) that
transform a catalogue $C_{a}$ into another catalogue $C_{b}$. Each entry in
the catalogue is a node. The difference is that we calculate the distance
between two items according to the similarity of nodes when modification:

$Distance(x,y)=min(1,\alpha*(1-Similarity(x,y))$

We leverage BERTScore to obtain the similarity between two items of the
catalogue based on the pre-trained contextual embeddings from SciBERT (Beltagy
et al., 2019):

$Similarity(x,y)=BERTScore(x,y)$

SciBert is more suitable for scientific literature than BERT because it was
pretrained on a large multi-domain corpus of scientific publications. After
experiments, the parameter is $\alpha=1.2$. Therefore, CEDS is defined as:

$CEDS(C_{a},C_{b})=100*\left(1-\frac{CED(C_{a},C_{b})}{max(|C_{a}|,|C_{b}|)}\right)$

We use the library ”Python Edit Distances”999https://gitlab.ub.uni-
bielefeld.de/bpaassen/python-edit-distances/-/tree/master proposed by Paaßen
et al. (2015) for the implementation of algorithms. An example of node
alignment and the conversion process between two catalogues is given in
Subsection 6.4.

Figure 3. Architectures of the end-to-end approach.

Figure 4. Architecture of the step-by-step approach.

##  5\. Models

We now introduce our initial explorations in hierarchical catalogue generation
for literature review, including end-to-end and step-by-step approaches.

###  5.1. End-to-End Approaches

One of the main challenges in generating the catalogue is handling long text.
The most intuitive and simplest generation method is the end-to-end model, so
we experimented with two base language models that can input long text. Each
method has two input forms, the difference being whether the title is input to
the encoder or decoder.

As shown in Figure 3, we consider an encoder-decoder model as a benchmark end-
to-end model. The model takes as input the title of a survey and the
information of its reference papers; it then generates the survey catalogue.
In Figure 3, the title and references are concatenated together and fed to the
encoder. In Figure 3, the title is passed as input to the decoder. Existing
neural models show an exponential explosion in computational overhead as the
input length increases. Therefore, the range of models that can be used is
limited due to the input length.

Izacard and Grave (2021) propose Fusion-in-Decoder (FiD), a model for open
domain question answering that we use for catalogue generation. The model
processes the combination of the survey title and information of a reference
paper independently from other papers by the encoder, as shown in Figure 3.
The decoder pays attention to the concatenation of all representations from
the encoder. Compared to Figure 3, Figure 3 only encodes papers separately,
and the title is the input to the decoder end. Unlike the previous encoder-
decoder model, the FiD model processes papers separately in the encoder. The
input can be extended to a large number of contexts since it only performs
self-attention over one context each time. This allows the computation time of
the model to grow linearly with the length of the input. Also, the joint
processing of reference papers in the decoder can better facilitate the
interaction of multiple papers.

###  5.2. Step-by-Step Approach

The other main challenge in generating the catalogue is modelling
relationships between catalogue items at different levels. Motivated by Tan et
al. (2021), we explore a conceptually simple approach to incremental
generation. Intuitively, progressive generation divides the complicated
problem of generating a complete catalogue into more manageable steps, namely
the generation of hierarchical levels of the catalogue. In contrast to
generating everything in one step, the progressive generation allows the model
first to perform high-level abstract planning and then shift attention to
increasingly concrete details. Figure 4 illustrates the generation process.

Instead of generating the whole catalogue $C$ directly, we propose to generate
step-by-step: $source\rightarrow L_{1}\rightarrow L_{1,2}\rightarrow
L_{1,2,3}(target)$, where $L_{i}$ is the $i-th$ level headings of the
catalogue. For example, in the first step, we input survey title and
information about its reference documents and generate the first-level
headings $L_{1}$ of the catalogue for that survey. Then, $L_{1}$ is added to
the input, and the next step generates the first two levels headings $L_{1,2}$
of the catalogue. Finally, it generates the whole catalogue with a method
similar to the previous step. The generation process corresponds to a
decomposition of the conditional probability as:

$\begin{split}P(target|source)=P(L_{1}&|source)\\\
&P(L_{1,2}|source,L_{1})P(L_{1,2,3}|source,L_{1,2})\end{split}$

Table 5. Automatic evaluation results on HiCaD. Bold indicates the best value
in each setting. $\ddagger$ represents the second best result and $\dagger$
represents the third best.

Method | ROUGE-1/ROUGE-2/ROUGE-L $\uparrow$ | CEDS $\uparrow$ | CQE$\downarrow$
---|---|---|---
$L_{1}$ | $L_{2}$ | $L_{3}$ | Total
End-to-End-Query-Encoder |  |  |  |
LED | base | 28.6/9.5/27.8 | 15.1/3.6/14.9 | 0.7/0.1/0.7 | 23.4/6.5/23.1 | 26.4 | 20.7
large | 30.3/10.1/29.5$\ddagger$ | 18.7/4.5/18.4$\ddagger$ | 1.6/0.3/1.6 | 27.4/7.6/27.0 | 29.0 | 13.7$\dagger$
FiD(BART) | base | 28.8/9.6/28.1 | 15.1/3.8/14.9 | 1.5/0.3/1.5 | 23.5/6.8/23.3 | 26.0 | 19.7
large | 29.0/10.0/28.4 | 18.8/4.5/18.5 | 2.2/0.4/2.2$\ddagger$ | 26.8/7.7/26.5$\ddagger$ | 27.9$\ddagger$ | 13.6$\ddagger$
End-to-End-Query-Decoder |  |  |  |
LED | base | 24.7/7.4/24.1 | 11.9/2.7/11.7 | 0.5/0.1/0.5 | 18.5/4.5/18.3 | 24.7 | 29.0
large | 27.0/8.2/26.2 | 16.1/3.7/15.9 | 1.2/0.3/1.2 | 23.8/5.9/23.4 | 27.5$\dagger$ | 18.5
FiD(BART) | base | 25.1/7.2/24.5 | 14.8/3.3/14.6 | 1.4/0.3/1.4 | 21.2/5.0/21.0 | 26.4 | 22.4
large | 25.6/7.6/25.0 | 17.6/4.0/17.4 | 1.9/0.3/1.8 | 23.8/5.9/23.5 | 27.4 | 17.8
Step-by-Step-Query-Encoder |  |  |  |
LED | base | 28.0/9.8/27.4 | 9.7/2.5/9.5 | 2.2/0.3/2.2$\dagger$ | 19.8/5.3/19.6 | 22.6 | 21.6
large | 29.8/10.7/29.0$\dagger$ | 18.3/4.4/18.0$\dagger$ | 1.2/0.9/1.0 | 25.2/7.0/24.9 | 26.6 | 15.4
FiD(BART) | base | 29.3/10.7/28.6 | 12.5/3.0/12.3 | 1.8/0.3/1.8 | 21.2/5.9/21.0 | 24.9 | 20.2
large | 30.7/10.8/30.1 | 16.1/3.8/15.8 | 4.1/0.6/4.0 | 26.2/6.8/25.8$\dagger$ | 25.0 | 13.2

##  6\. Experiments

We study the performance of multiple models on the HiCaD dataset. Detailed
analysis of the generation quality is also provided, including correlation
validation of the proposed evaluation metrics with human evaluation and ROUGE
in addition to the abstractiveness study.

###  6.1. Baselines

Because of the length of inputs, we choose an encoder-decoder transformer
model that can handle the long text and its backbone model to implement FiD
besides various extractive models in our experiments.

LexRank LexRank (Erkan and Radev, 2004) is an unsupervised extractive
summarization approach based on graph-based centrality scoring of sentences.

TextRank TextRank (Mihalcea and Tarau, 2004) is a graph-based ranking
algorithm for keyword extraction and document summarization, improved from
Google’s PageRank (Page et al., 1999). TextRank uses co-occurrence information
(semantics) between words within a document to extract keywords.

BART Lewis et al. (2019) build a pre-trained sequence-to-sequence Transformer
model to reconstruct the original input text from the corrupted text with a
denoising auto-encoder. It is worth noting that its limit input length is 1024
tokens.

Longformer-Encoder-Decoder LED (Beltagy et al., 2020) is built on BART but
adopts the sparse global attention mechanisms in the encoder part, which
circumvents the input size limitations of the BART model and allows to
experiment with input sizes reaching 16384 tokens.

###  6.2. Implementation Details

We use dropout with the probability $0.1$ and a learning rate of $4e-5$. The
optimizer is Adam with $\beta 1=0.9$ and $\beta 2=0.999$. We also adopt the
learning rate warmup and decay. During the decoding process, we use beam
search with a beam size of 4 and tri-gram blocking to reduce repetitions. We
adopt the implementations of LED from HuggingFace’s Transformers (Wolf et al.,
2020) and FiD from SEGENC (Vig et al., 2022). We maximize the input length to
16,384 tokens. All the models are trained on one NVIDIA A100-PCIE-80GB.

###  6.3. Results and Discussion

We experimented with twelve settings in total, with four implementations (LED-
base, LED-large, FiD (BART-base), FiD (BART-large)) for each of the three
architectures (end-to-end-query-encoder, end-to-end-query-decoder, step-by-
step-query-encoder). The titles of literature reviews are only fed to the
encoder in the experiments of step-by-step.

####  6.3.1. Automatic Evaluation

As shown in Figure 5, we calculated the ROUGE scores, catalogue edit distance
similarity (CEDS) and Catalogue Quality Estimate (CQE) between the generated
results and the ground truth separately. We especially remove all level mark
symbols (e.g. $<L_{1}>$) for evaluation. In order to compare the ability of
different methods to generate different levels of headings in detail, we
calculated ROUGE scores for each level of headings with the corresponding
level of reference answers.

Since we take the title of the literature review as input, we also calculate
the ROUGE scores of the title to the ground truth, which is the lowest result
we expect to accept. Because both the title and the abstracts generated by
extractive approaches do not have catalogue characteristics, we only calculate
the ROUGE scores between generated results with the full answers. TextRank
gives the worst results (4.5/1.4/4.4) and bears little resemblance to the
standard answers. LexRank achieves similar results 17.2/3.0/16.3 as the title
of survey 15.2/5.1/14.5. It can be concluded that the results generated by
these two extractive methods do not apply to hierarchical catalogue
generation.

####  6.3.2. Human Evaluation

To demonstrate the validity of our proposed evaluation metrics, we conducted
consistency tests on CEDS and corresponding human evaluation. We hired two
master’s degree candidates and one PhD candidate who passed an intermediate
English test (Level 6 on the University English Test) and had an NLP
background. We sampled the same 50 generated results for each implementation
and scrambled the final 600 pairs. We asked the testers to evaluate the
quality of the catalogues from the similarity with the ground truth
(SIM_human). The scores ranged from one to five integers, with five
representing the best results. Because data outliers significantly impact
correlation analysis, we need to detect the presence of outliers first. As
shown in Table 5, no data points in the two data sets (CEDS_sam and SIM_human)
are located in the upper or lower range of the error line of the box plot, so
there is no abnormal data.

Figure 5. Testing data for the presence of outliers.

Commonly used consistency evaluation methods are divided into parametric tests
(e.g., Pearson’s correlation coefficient) and non-parametric tests (e.g.,
Spearman’s rank correlation coefficient). The normal distribution of data is
one of the prerequisites for Pearson correlation analysis. If the data do not
satisfy the normal distribution, only non-parametric analysis methods can be
used. Next step, we tested the data for normal distribution. After the S-W
(Shapiro-Wilk) test, the p-values for the two sets are 0.520 and 0.250, all
greater than 0.05. The two data groups could be considered to meet the normal
distribution so that Pearson correlation analysis can continue.

Pearson $r$-value indicates the correlation coefficient between the variables.
The closer to 1, the more correlated and the closer to 0, the less correlated.
The $p$-value is the test value, and $p<0.05$ indicates a correlation.
Especially, $p<0.01$ means a strong positive correlation exists between the
two variables. Person correlation analysis showed that the p-values between
CEDS_sam and SIM_human are 0.027, more diminutive than 0.05. The $r$-value is
0.634, which represents a strong positive correlation. Therefore, CEDS is a
valid evaluation indicator.

####  6.3.3. Consistency Check

We conducted a two-by-two consistency test between all the indicators measured
(Table 6) where only ROUGE-L in ROUGE was computed. According to the pre-
tests, we have processed an outlier by replacing ROUGE-L 4.0 with average
ROUGE-L 1.67 in the step-by-step-query-encoder $L_{3}$. It makes each set of
data have no outliers and conform to a normal distribution (Figure 5).
Therefore, we used Pearson’s correlation coefficient to check the consistency.

Table 6. Pearson’s consistency test results. **. Significant correlation at
the 0.01 level (two-tailed). *. Significant correlation at the 0.05 level
(two-tailed)

|  | L1RL | L2RL | L3RL | TotalRL | CQE | CEDS
---|---|---|---|---|---|---|---
L1RL | r | 1 | 0.313 | 0.323 | .668* | -.747** | 0.087
p |  | 0.322 | 0.306 | 0.018 | 0.005 | 0.788
L2RL | r | 0.313 | 1 | 0.042 | .889** | -.757** | .895**
p | 0.322 |  | 0.898 | 0.000 | 0.004 | 0.000
L3RL | r | 0.323 | 0.042 | 1 | 0.265 | -0.492 | -0.053
p | 0.306 | 0.898 |  | 0.405 | 0.104 | 0.870
TotalRL | r | .668* | .889** | 0.265 | 1 | -.940** | .721**
p | 0.018 | 0.000 | 0.405 |  | 0.000 | 0.008
CQE | r | -.747** | -.757** | -0.492 | -.940** | 1 | -0.530
p | 0.005 | 0.004 | 0.104 | 0.000 |  | 0.076
CEDS | r | 0.087 | .895** | -0.053 | .721** | -0.530 | 1
p | 0.788 | 0.000 | 0.870 | 0.008 | 0.076 |

Table 7.  An example of aligned nodes and distances between them when
calculating CED.

Generated Result | Ground Truth | distance
---|---|---
$<l_{1}>$ introduction [0] |  $<l_{1}>$ introduction [0] | 0.0
- |  $<l_{1}>$ neural machine translation [1] | 1.0
$<l_{1}>$ background [1] |  $<l_{1}>$ domain adaptation for smt [2] | 0.50
$<l_{2}>$ neural machine translation [2] |  $<l_{2}>$ data centric [3] | 0.48
$<l_{2}>$ domain adaptation [3] |  $<l_{2}>$ model centric [4] | 0.35
$<l_{1}>$ survey of domain adaptation for nmt [4] |  $<l_{1}>$ domain adaptation for nmt [5] | 0.12
$<l_{2}>$ parallel data adaptation [5] |  $<l_{2}>$ data centric [6] | 0.34
$<l_{3}>$ fine-tuning [6] |  $<l_{3}>$ using monolingual corpora [7] | 0.44
$<l_{3}>$ instance weighting [7] |  $<l_{3}>$ synthetic parallel corpora generation [8] | 0.35
$<l_{3}>$ multi-task learning [8] |  $<l_{3}>$ using out-of-domain parallel corpora [9] | 0.38
- |  $<l_{2}>$ model centric [10] | 1.0
- |  $<l_{3}>$ training objective centric [11] | 1.0
$<l_{2}>$ monolingual data adaptation [9] |  $<l_{3}>$ architecture centric [12] | 0.44
$<l_{2}>$ synthetic parallel data adaptation [10] |  $<l_{3}>$ decoding centric [13] | 0.41
- |  $<l_{1}>$ domain adaptation in real-world scenarios [14] | 1.0
$<l_{1}>$ conclusion and future directions [11] |  $<l_{1}>$ future directions [15] | 0.18
- |  $<l_{2}>$ domain adaptation for state-of-the-art nmt architectures [16] | 1.0
- |  $<l_{2}>$ domain specific dictionary incorporation [17] | 1.0
- |  $<l_{2}>$ multilingual and multi-domain adaptation [18] | 1.0
- |  $<l_{2}>$ adversarial domain adaptation and domain generation [19] | 1.0
- |  $<l_{1}>$ conclusion [20] | 1.0

First, we observe that the ROUGE-L of the secondary catalogue (L2RL) has a
more substantial similarity to the ROUGE-L for the whole catalogue (TotalRL)
than the first-level (L1RL). This is consistent with our conclusion in Section
3.3 that secondary catalogues account for greater weight.

Second, a noticeable trend is that Catalogue Quality Estimate (CQE) negatively
correlates with all ROUGE scores. The degree of correlation is
$L3<L1<L2<Total$. In addition, the value of CQE in the test set is only 8.3%,
which is a tiny share of the overall. It means that the more the generated
catalogue contains template words, the less detail correspondingly, and the
more it differs from the standard answer.

Finally, we study the difference between TotalRL and CEDS (Catalogue Edit
Distance Similarity). TotalRL and CEDS have a strong positive correlation with
a correlation coefficient of 0.721, proving that CEDS can measure the
similarity of the generated results to ground truth. Also, TotalRL has a
robust negative correlation with CQE, while CEDS does not correlate with CQE.
These two findings suggest that while both can be used to measure similarity,
ROUGE is more concerned with word matching, and CEDS is more concerned with
the structure of catalogues.

####  6.3.4. Discussion

Based on the above analysis, we derived the polarity of the proposed
evaluation indicators. The larger the CEDS and the smaller the CQE, the better
the quality of the generated catalogue.

For ROUGE scores, the implementation of LED-large on the end-to-end method
achieves the best results in the generation of the second level of headings
and the whole catalogue, and the BART-large on the step-by-step approach
achieves the best results in the first and third level of headings.
Considering CQE and CEDS together, the best model is LED-large on the end-to-
end method, and the second best model is FiD(BART-large) on the end-to-end
approach among all twelve implementations.

By comparing all evaluation scores, the end-to-end approach achieves higher
similarity with the ground truth than step-by-step on the whole catalogue, and
inputting titles at the encoder side is better than the decoder side.
Generally, the model with a larger number of parameters is better than a
smaller one.

###  6.4. Case Study

Figure 6. Examples of end-to-end-query-encoder and step-by-step-query-encoder.

Table 7 and Figure 6 present an example of the best catalogues generated by
end-to-end models and step-by-step models about the title ”a survey of domain
adaptation for neural machine translation”. The result in Table 7 is generated
by end-to-end-query-decoder, and Figure 6 is end-to-end-query-encoder, step-
by-step-query-encoder from left to right respectively.

Table 7 gives an example of alignment between catalogue items when calculating
the Catalogue Edit Distance (CED). If an item does not match any other entity,
then the required action for this item is insertion or deletion, so the cost
is 1. If a node matches a node, then the required action is modification. The
operation cost of modification is calculated according to the similarity. It
is worth noting that Node[2] in the generated result does not match Node[1] in
the ground truth, even though they have exactly the same content. It is
because the former is a secondary heading, and the latter is a primary
heading. If the alignment is forced, the cost required will be greater than
the current result. Thus this example shows that not only does the metric CEDS
measure catalogues from a semantic perspective, but also takes into account
the hierarchical structure of the catalogue.

The parts marked in red in Figure 6 are the generation problems we observe.
The first problem is duplicating the content in a single catalogue item, e.g.
”monolingual monolingual data”. The second one is the hierarchy error between
sibling nodes. For example, ”evaluation metrics” conceptually contain
”automatic evaluation”. Finally, there is a recurrence of the heading
”applications of nmt domain adaptation”. In summary, there are still
duplication and hierarchical errors in the current results.

###  6.5. Foundation Models

Figure 7. Three catalogue generation results of ChatGPT model around ”a survey
of domain adaptation for neural machine translation”.

Thanks to the large amount of memory of the massive corpus in the pre-training
stage, the super-large scale pre-training model has shown excellent
performance on many downstream tasks. In this section, we experiment with the
effect of two foundation models on the hierarchical catalogue generation task.

####  6.5.1. Galactica

GAL (Taylor et al., 2022) is a large language model for science that achieve
state-of-the-art results over many scientific tasks. The corpus it used
includes over 48 million papers, textbooks and lecture notes, millions of
compounds and proteins, scientific websites, encyclopedias and so on. It can
write a literature review with catalogues by itself, given a query. We
experimented zero-shot test by the base model with 1.3B parameters model on
our test set. There is a lot of repetition in the results, and the ROUGE score
is $3.2/0.7/3.1$, reserving catalogues only. It indicates that the ability of
GAL to generate surveys with catalogues is not particularly stable.

####  6.5.2. ChatGPT

ChatGPT101010https://openai.com/blog/chatgpt/ is fine-tuned from GPT-3.5 and
was optimized for dialogue by using Reinforcement Learning with Human Feedback
(RLHF). ChatGPT is not just a gossip bot but can write code, translate
literature, write novels, evaluate assignments and perform a host of other
everyday text output-based tasks. The magic spell we use to activate ChatGPT
is ’I want to write a literature review with the title ”a survey of domain
adaptation for neural machine translation”, please write an outline to me.’
Unlike the previous experimental method, which could only generate one result,
the results generated by ChatGPT have great diversity. Figure 7 shows the
results of three attempts.

Compared to the ground truth, the overall length of the catalogues generated
by ChatGPT is longer because of the recurrence of some keywords in the title,
such as ”Domain Adaptation” and ”Neural Machine Translation”, which obviously
reduces the simplicity of the catalogue. Besides the template words, ChatGPT
can also generate some details. For example, the right one contains the
heading ”BLEU Score”, ”Perplexity”, and ”Accuracy/Word Error Rate”, which are
not in the title. Compared to the example on the left in Figure 6, the results
generated by ChatGPT are farther from the title and do not have tertiary
headings with more detailed content.

##  7\. Conclusion

In this work, we introduce a new task hierarchical catalogue generation for
literature review (HiCatGLR) and develop a dataset HiCaD out of arXiv and
Semantic Scholar Dataset. We empirically evaluate two methods with twelve
implementations for catalogue generation and found that the end-to-end model
with the title input to the encoder based on LED-large achieves the best
result. CQE and CEDS are designed to measure the quality of generated results
from semantics and hierarchical structures. Our analysis illustrates how CEDS
resolves some of the limitations of current metrics. We have also demonstrated
that our model has the ability to generate promising catalogues for literature
reviews with sparse attention or multi-encoders. In the future, we plan to
explore a better understanding of input content and the solution to headings
level error, repetition and semantic conflict between headings.

## References

  * (1)
  * Abu-Jbara et al. (2013) Amjad Abu-Jbara, Jefferson Ezra, and Dragomir Radev. 2013\.  Purpose and polarity of citation: Towards nlp-based bibliometrics. In _Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: Human language technologies_. 596–606.
  * An (2021) STM An. 2021.  STM Global Brief 2021–Economics & Market Size.  (2021).
  * Bedi et al. (2022) Manjot Bedi, Tanisha Pandey, Sumit Bhatia, and Tanmoy Chakraborty. 2022.  Why Did You Not Compare with That? Identifying Papers for Use as Baselines. In _Advances in Information Retrieval: 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10–14, 2022, Proceedings, Part I_. Springer, 51–64.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A Pretrained Language Model for Scientific Text. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 3615–3620.  https://doi.org/10.18653/v1/D19-1371
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020\.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_ (2020).
  * Booth et al. (2021) Andrew Booth, Anthea Sutton, Mark Clowes, and Marrissa Martyn-St James. 2021.  Systematic approaches to a successful literature review.  (2021).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Erkan and Radev (2004) Günes Erkan and Dragomir R Radev. 2004.  Lexrank: Graph-based lexical centrality as salience in text summarization.  _Journal of artificial intelligence research_ 22 (2004), 457–479.
  * Färber and Jatowt (2020) Michael Färber and Adam Jatowt. 2020.  Citation recommendation: approaches and datasets.  _International Journal on Digital Libraries_ 21, 4 (2020), 375–405.
  * Fire and Guestrin (2019) Michael Fire and Carlos Guestrin. 2019.  Over-optimization of academic publishing metrics: observing Goodhart’s Law in action.  _GigaScience_ 8, 6 (2019), giz053.
  * Fu et al. (2022) Tsu-Jui Fu, William Yang Wang, Daniel McDuff, and Yale Song. 2022\.  Doc2ppt: Automatic presentation slides generation from scientific documents. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 36. 634–642.
  * Ge et al. (2021) Yubin Ge, Ly Dinh, Xiaofeng Liu, Jinsong Su, Ziyao Lu, Ante Wang, and Jana Diesner. 2021.  BACO: A background knowledge-and content-based framework for citing sentence generation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 1466–1478.
  * Hoang and Kan (2010) Cong Duy Vu Hoang and Min-Yen Kan. 2010.  Towards automated related work summarization. In _Coling 2010: Posters_. 427–435.
  * Hu and Wan (2014) Yue Hu and Xiaojun Wan. 2014.  Automatic generation of related work sections in scientific papers: an optimization approach. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 1624–1633.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_. Association for Computational Linguistics, Online, 874–880.  https://doi.org/10.18653/v1/2021.eacl-main.74
  * Jha et al. (2015) Rahul Jha, Catherine Finegan-Dollak, Ben King, Reed Coke, and Dragomir Radev. 2015.  Content models for survey generation: a factoid-based evaluation. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 441–450.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019\.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_ (2019).
  * Li et al. (2022) Pengcheng Li, Wei Lu, and Qikai Cheng. 2022.  Generating a related work section for scientific papers: an optimized approach with adopting problem and method information.  _Scientometrics_ 127, 8 (2022), 4397–4417.
  * Lin (2004) Chin-Yew Lin. 2004\.  ROUGE: A Package for Automatic Evaluation of Summaries. In _Text Summarization Branches Out_. Association for Computational Linguistics, Barcelona, Spain, 74–81.  https://aclanthology.org/W04-1013
  * LIU et al. (2022) Shuaiqi LIU, Jiannong Cao, Ruosong Yang, and Zhiyuan Wen. 2022\.  Generating a Structured Summary of Numerous Academic Papers: Dataset and Method. In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22_ , Lud De Raedt (Ed.). International Joint Conferences on Artificial Intelligence Organization, 4259–4265.  https://doi.org/10.24963/ijcai.2022/591 Main Track.
  * Lo et al. (2020) Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020.  S2ORC: The Semantic Scholar Open Research Corpus. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 4969–4983.  https://doi.org/10.18653/v1/2020.acl-main.447
  * Lu et al. (2020) Yao Lu, Yue Dong, and Laurent Charlin. 2020.  Multi-XScience: A large-scale dataset for extreme multi-document summarization of scientific articles.  _arXiv preprint arXiv:2010.14235_ (2020).
  * Luu et al. (2020) Kelvin Luu, Xinyi Wu, Rik Koncel-Kedziorski, Kyle Lo, Isabel Cachola, and Noah A Smith. 2020\.  Explaining relationships between scientific documents.  _arXiv preprint arXiv:2002.00317_ (2020).
  * Mihalcea and Tarau (2004) Rada Mihalcea and Paul Tarau. 2004.  Textrank: Bringing order into text. In _Proceedings of the 2004 conference on empirical methods in natural language processing_. 404–411.
  * Mohammad et al. (2009) Saif Mohammad, Bonnie Dorr, Melissa Egan, Ahmed Hassan, Pradeep Muthukrishnan, Vahed Qazvinian, Dragomir Radev, and David Zajic. 2009\.  Using citations to generate surveys of scientific paradigms. In _Proceedings of human language technologies: The 2009 annual conference of the North American chapter of the association for computational linguistics_. 584–592.
  * Paaßen (2018) Benjamin Paaßen. 2018\.  Revisiting the tree edit distance and its backtracing: A tutorial.  _arXiv preprint arXiv:1805.06869_ (2018).
  * Paaßen et al. (2015) Benjamin Paaßen, Bassam Mokbel, and Barbara Hammer. 2015\.  A Toolbox for Adaptive Sequence Dissimilarity Measures for Intelligent Tutoring Systems. In _Proceedings of the 8th International Conference on Educational Data Mining (EDM 2015)_ (2015-06), Olga Christina Santos, Jesus Gonzalez Boticario, Cristobal Romero, Mykola Pechenizkiy, Agathe Merceron, Piotr Mitros, Jose Maria Luna, Christian Mihaescu, Pablo Moreno, Arnon Hershkovitz, Sebastian Ventura, and Michel Desmarais (Eds.). International Educational Datamining Society, 632–632.  http://www.educationaldatamining.org/EDM2015/uploads/papers/paper_257.pdf
  * Page et al. (1999) Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999.  _The PageRank citation ranking: Bringing order to the web._ Technical Report. Stanford InfoLab.
  * Shpynov and Nikolai (2021) Oleg Shpynov and Kapralov Nikolai. 2021.  PubTrends: A scientific literature explorer. In _Proceedings of the 12th ACM Conference on bioinformatics, computational biology, and health informatics_. 1–1.
  * Shuaiqi et al. (2022) LIU Shuaiqi, Jiannong Cao, Ruosong Yang, and Zhiyuan Wen. 2022\.  Generating a Structured Summary of Numerous Academic Papers: Dataset and Method.  (2022).
  * Sun and Zhuge (2019) Xiaoping Sun and Hai Zhuge. 2019.  Automatic Generation of Survey Paper Based on Template Tree. In _2019 15th International Conference on Semantics, Knowledge and Grids (SKG)_. IEEE, 89–96.
  * Tan et al. (2021) Bowen Tan, Zichao Yang, Maruan Al-Shedivat, Eric Xing, and Zhiting Hu. 2021.  Progressive Generation of Long Text with Pretrained Language Models. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Online, 4313–4324.  https://doi.org/10.18653/v1/2021.naacl-main.341
  * Taylor et al. (2022) Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022.  Galactica: A large language model for science.  _arXiv preprint arXiv:2211.09085_ (2022).
  * Tkaczyk et al. (2015) Dominika Tkaczyk, Paweł Szostek, Mateusz Fedoryszak, Piotr Jan Dendek, and Łukasz Bolikowski. 2015\.  CERMINE: automatic extraction of structured metadata from scientific literature.  _International Journal on Document Analysis and Recognition (IJDAR)_ 18 (2015), 317–335.
  * Vig et al. (2022) Jesse Vig, Alexander Fabbri, Wojciech Kryscinski, Chien-Sheng Wu, and Wenhao Liu. 2022.  Exploring Neural Models for Query-Focused Summarization. In _Findings of the Association for Computational Linguistics: NAACL 2022_. Association for Computational Linguistics, Seattle, United States, 1455–1468.  https://doi.org/10.18653/v1/2022.findings-naacl.109
  * Viswanathan et al. (2021) Vijay Viswanathan, Graham Neubig, and Pengfei Liu. 2021\.  Citationie: Leveraging the citation graph for scientific information extraction.  _arXiv preprint arXiv:2106.01560_ (2021).
  * Wang and Lo (2021) Lucy Lu Wang and Kyle Lo. 2021.  Text mining approaches for dealing with the rapidly expanding literature on COVID-19.  _Briefings in Bioinformatics_ 22, 2 (2021), 781–799.
  * Wang et al. (2022) Pancheng Wang, Shasha Li, Kunyuan Pang, Liangliang He, Dong Li, Jintao Tang, and Ting Wang. 2022.  Multi-Document Scientific Summarization from a Knowledge Graph-Centric View.  _arXiv preprint arXiv:2209.04319_ (2022).
  * Webster and Watson (2002) Jane Webster and Richard T Watson. 2002.  Analyzing the past to prepare for the future: Writing a literature review.  _MIS quarterly_ (2002), xiii–xxiii.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020\.  Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations_. 38–45.
  * Wu et al. (2021) Jia-Yan Wu, Alexander Te-Wei Shieh, Shih-Ju Hsu, and Yun-Nung Chen. 2021.  Towards generating citation sentences for multiple references with intent control.  _arXiv preprint arXiv:2112.01332_ (2021).
  * Xing et al. (2020) Xinyu Xing, Xiaosheng Fan, and Xiaojun Wan. 2020.  Automatic generation of citation texts in scholarly papers: A pilot study. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 6181–6190.
  * Xu et al. (2021) Huiyan Xu, Zhongqing Wang, Yifei Zhang, Xiaolan Weng, Zhijian Wang, and Guodong Zhou. 2021\.  Document structure model for survey generation using neural network.  _Frontiers of Computer Science_ 15 (2021), 1–10.
  * Yasunaga and Lafferty (2019) Michihiro Yasunaga and John D Lafferty. 2019.  Topiceq: A joint topic and mathematical equation model for scientific texts. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 33. 7394–7401.
  * Zhang and Shasha (1989) Kaizhong Zhang and Dennis Shasha. 1989.  Simple fast algorithms for the editing distance between trees and related problems.  _SIAM journal on computing_ 18, 6 (1989), 1245–1262.
  * Zhang* et al. (2020) Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  BERTScore: Evaluating Text Generation with BERT. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=SkeHuCVFDr


# CleanRL: High-quality Single-file Implementations of Deep Reinforcement
Learning Algorithms

Shengyi Huang1<EMAIL_ADDRESS>
Rousslan Fernand Julien Dossa2<EMAIL_ADDRESS>
Chang Ye3<EMAIL_ADDRESS>
Jeff Braga1<EMAIL_ADDRESS>
1College of Computing and Informatics, Drexel University, USA
2Graduate School of System Informatics, Kobe University, Japan
3Tandon School of Engineering, New York University, USA

###### Abstract

CleanRL is an open-source library that provides high-quality single-file
implementations of Deep Reinforcement Learning algorithms. It provides a
simpler yet scalable developing experience by having a straightforward
codebase and integrating production tools to help interact and scale
experiments. In CleanRL, we put all details of an algorithm into a single
file, making these performance-relevant details easier to recognize.
Additionally, an experiment tracking feature is available to help log metrics,
hyperparameters, videos of an agent’s gameplay, dependencies, and more to the
cloud. Despite succinct implementations, we have also designed tools to help
scale, at one point orchestrating experiments on more than 2000 machines
simultaneously via Docker and cloud providers. Finally, we have ensured the
quality of the implementations by benchmarking against a variety of
environments.

##  1 Introduction

In recent years, Deep Reinforcement Learning (DRL) algorithms have achieved
great success in training autonomous agents for tasks ranging from playing
video games directly from pixels to robotic control (Mnih et al., 2013;
Lillicrap et al., 2015; Schulman et al., 2017). At the same time, open-source
DRL libraries also flourish in the community, such as Stable Baselines 3 (SB3)
(Raffin et al., 2019), RLLib (Liang et al., 2018), MushroomRL (D’Eramo et al.,
2020), PFRL (Fujita et al., 2021), and others. Many of them have adopted good
modular designs and fostered vibrant development communities. Nevertheless,
understanding all implementation details of an algorithm remains difficult
because these details are spread to different modules. However, it has become
increasingly important to understand these details because implementations
matter (Engstrom et al., 2019).

In this paper, we introduce CleanRL, a DRL library based on single-file
implementations to help researchers understand all algorithms’ details,
prototype new features, analyze experiments, and scale with ease. CleanRL is a
_non-modular_ library. Each algorithm in CleanRL is self-contained in a single
file. We have trimmed the lines of code (LOC) to a minimal degree while
maintaining readability and decent performance w.r.t reputable reference
implementations. For instance, our Proximal Policy Optimization (PPO)
(Schulman et al., 2017) implementation matches SB3’s PPO performance in the
Atari game Breakout using only 337 LOC in a single
file111https://github.com/vwxyzjn/cleanrl/blob/2486b90102bd87fa43cda481fe711e0eda0c162b/cleanrl/ppo_atari.py,
making it much easier to understand the algorithm in one go. In contrast, the
researchers of other libraries need to understand the modularity design (on
average 7-15 files) which can be comprised of thousands of LOC.

Figure 1: A screenshot of the Open RL Benchmark where the users can (1) zoom
into the learning curve, (2) click to trace back to the original experiment
for more info (e.g. hyperparameters), and (3) checkout videos at different
training stages.

While CleanRL is stand-alone, for convenience, it supports optional
integration with production-quality tool providers. Firstly, integrating
Weights and Biases (W&B) helps track the logs and metrics to the cloud. Over
the years, we have tracked thousands of experiments across 7+ algorithms and
40+ games in our Open RL Benchmark (https://benchmark.cleanrl.dev) to ensure
and showcase the quality of our implementations (see Figure 1). Secondly,
integrating AWS Batch helps provision any AWS instances as computational nodes
for large-scale experiment orchestration. Adopting these tools has helped
CleanRL focus just on algorithms without worrying about analysis or scaling
utilities. Also, our architecture design allows the users to flexibly use
other tracking and orchestration vendors if desired, so there is no dependency
on proprietary services.

##  2 Single-file Implementations

Even small implementation details can hugely impact the performance of deep RL
algorithms (Engstrom et al., 2019), however the nature of the modular
libraries encapsulates and hides theses details. CleanRL helps the researchers
to recognize these details by adopting single-file implementations. That is,
we put the algorithm details and the environment-specific details into a
single file. For example, we have a

  1. 1.

ppo.py (321 LOC) for the classic control environments, such as CartPole-v1.

  2. 2.

ppo_atari.py (337 LOC) for the Atari environments (Bellemare et al., 2013).

  3. 3.

ppo_continuous_action.py (331 LOC) for the robotics environments (e.g. MuJoCo,
PyBullet) with continuous action spaces (Schulman et al., 2017).

Despite having duplicate code among these files, the single-file
implementations have the following benefits.

  1. 1.

Transparent learning experience It becomes easier to recognize all aspects of
the code in one place. By looking at ppo.py, it is straightforward to
recognize the 11 general implementation details of PPO, especially when
watching the corresponding tutorial222See https://youtu.be/MEt6rrxH8W4. Then,
comparing it with ppo_atari.py shows about 30 LOC difference to implement 9
Atari-specific details333See https://youtu.be/05RMTj-2K_Y, and another
comparison with ppo_continuous_action.py shows about 25 LOC difference to
implement 6 details for continuous action spaces444See
https://youtu.be/_rDUEOsjUds.

  2. 2.

Faster debug experience Most variables in the files live in the _global python
name scope_. This means the researchers can do Ctrl+C to stop the program
execution and check most variables and their shapes in the interactive shell
(see Appendix A). This is arguably more convenient than using the Python’s
debugger, which only shows the variables in a specific name scope like in a
function.

  3. 3.

Faster prototype experience Because of the faster debug experience, it becomes
easier to develop new features. For example, invalid action masking (Huang and
Ontañón, 2020) is a common technique used in games with large parameterized
action spaces. With CleanRL, it takes about 40 LOC to
implement555https://www.diffchecker.com/g9TYkKCS. In comparison, it takes
substantially more LOC (e.g. 600+ LOC not counting test cases) because of
overhead such as re-factoring the functional arguments, making the classes
more general.

For this motivation, we have also implemented dqn.py, dqn_atari.py (Mnih et
al., 2013), c51.py, c51_atari.py (Bellemare et al., 2017), apex_atari.py
(Horgan et al., 2018), ddpg_continuous_action.py (Lillicrap et al., 2015),
td3_continuous_action.py (Fujimoto et al., 2018), and sac_continuous_action.py
(Haarnoja et al., 2018).

##  3 Experiment Tracking and Analysis

CleanRL supports integration with W&B, a modern experiment tracking and
analysis solution. When running an experiment, W&B automatically tracks the 1)
source code, 2) dependency (i.e. requirements.txt), 3) hyperparameters, 4) the
exact terminal command used to run the experiment, 5) training metrics, 6)
videos of the agents playing the game, 7) system metrics, and 8) logs (stdout,
stderr). See Appendix B for 9 selected features that help query and clean the
experiment data.

Although W&B is a proprietary service that we elect to use, our implementation
uses Tensorboard to record metrics, which is open-source. So the users could
easily change about 10 LOC to use other experiment tracking vendors such as
ClearML (https://clear.ml) that supports Tensorboard integration.

Figure 2: The performance of CleanRL’s algorithms in Atari and continuous
action domains.

##  4 Cloud Integration

Despite succinct implementations, we are able to use CleanRL for large-scale
studies by leveraging public cloud infrastructure such as Amazon Web Services
(AWS). We package the code into a Docker container to orchestrate the
experiments on any cloud provider. When needing to run experiments on demand,
the users of CleanRL can use Terraform to reproduce a preset infrastructure in
a few commands, then utilize our utility scripts to create as many experiments
as needed. In 2020 alone, the authors have run over 50,000+ hours of
experiments using this workflow. See https://docs.cleanrl.dev for more
documentation.

##  5 Open RL Benchmark

We have created a project called Open RL Benchmark
(https://benchmark.cleanrl.dev) to track canonical RL experiments. To date, we
have tracked thousands of experiments across 7+ algorithms in various domains
(e.g. Atari, MuJoCo, PyBullet, Procgen (Cobbe et al., 2020), Griddly (Bamford
et al., 2020), Gym-$\mu$RTS (Huang et al., 2021)). See Figure 2 for selected
experiments. Our benchmark is interactive as shown in Figure 1: researchers
can easily query information such as GPU utilization and videos of an agent’s
gameplay that are normally hard to acquire in other RL benchmarks.

##  6 Conclusion

In this paper, we introduced CleanRL as a deep RL library that provides high-
quality single-file implementations with experiment tracking and cloud
integration. The source code can be found at
https://github.com/vwxyzjn/cleanrl.

## References

  * Bamford et al. (2020) Chris Bamford, Shengyi Huang, and Simon Lucas.  Griddly: A platform for ai research in games, 2020.
  * Bellemare et al. (2013) M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling.  The arcade learning environment: An evaluation platform for general agents.  _Journal of Artificial Intelligence Research_ , 47:253–279, jun 2013.
  * Bellemare et al. (2017) Marc G Bellemare, Will Dabney, and Rémi Munos.  A distributional perspective on reinforcement learning.  In _International Conference on Machine Learning_ , pages 449–458. PMLR, 2017.
  * Cobbe et al. (2020) Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman.  Leveraging procedural generation to benchmark reinforcement learning.  In _International conference on machine learning_ , pages 2048–2056. PMLR, 2020.
  * D’Eramo et al. (2020) Carlo D’Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters.  Mushroomrl: Simplifying reinforcement learning research.  _Journal of Machine Learning Research_ , 2020.
  * Engstrom et al. (2019) Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry.  Implementation matters in deep rl: A case study on ppo and trpo.  In _International Conference on Learning Representations_ , 2019.
  * Fujimoto et al. (2018) Scott Fujimoto, Herke Hoof, and David Meger.  Addressing function approximation error in actor-critic methods.  In _International Conference on Machine Learning_ , pages 1587–1596. PMLR, 2018.
  * Fujita et al. (2021) Yasuhiro Fujita, Prabhat Nagarajan, Toshiki Kataoka, and Takahiro Ishikawa.  Chainerrl: A deep reinforcement learning library.  _Journal of Machine Learning Research_ , 22(77):1–14, 2021.
  * Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _International conference on machine learning_ , pages 1861–1870. PMLR, 2018.
  * Horgan et al. (2018) Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver.  Distributed prioritized experience replay.  _arXiv preprint arXiv:1803.00933_ , 2018.
  * Huang and Ontañón (2020) Shengyi Huang and Santiago Ontañón.  A closer look at invalid action masking in policy gradient algorithms.  _arXiv preprint arXiv:2006.14171_ , 2020.
  * Huang et al. (2021) Shengyi Huang, Santiago Ontañón, Chris Bamford, and Lukasz Grela.  Gym-$\mu$rts: Toward affordable full game real-time strategy games research with deep reinforcement learning.  _arXiv preprint arXiv:2105.13807_ , 2021.
  * Liang et al. (2018) Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica.  Rllib: Abstractions for distributed reinforcement learning.  In _International Conference on Machine Learning_ , pages 3053–3062. PMLR, 2018.
  * Lillicrap et al. (2015) Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous Control with Deep Reinforcement Learning.  _arXiv preprint arXiv:1509.02971_ , 2015.
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing Atari with Deep Reinforcement Learning.  _arXiv preprint arXiv:1312.5602_ , 2013.
  * Raffin et al. (2019) Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dormann.  Stable baselines3.  https://github.com/DLR-RM/stable-baselines3, 2019.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal Policy Optimization Algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.

## Appendix A. Interactive Shell

In CleanRL, we have put most of the variables in the _global python name
scope_. This makes it easier to inspect the variables and their shapes. The
following figure shows a screenshot of the Spyder editor
666https://www.spyder-ide.org/, where the code is on the left and the
interactive shell is on the right. In the interactive shell, we can easily
inspect the variables for debugging purposes without modifying the code.

## Appendix B. W&B Editing Panel

A screenshot of the W&B panel that allows the the users to change smoothing
weight, add panels to show different metrics like losses, visualize the videos
of the agents’ gameplay, filter, group, sort, and search for desired
experiments.