# Retrieval-based Full-length Wikipedia Generation for Emergent Events

Jiebin Zhang ∗†, Eugene J. Yu∗†, Qinyu Chen∗†, Chenhao Xiong† Dawei Zhu†
Han Qian†, Mingbo Song†, Xiaoguang Li‡, Qun Liu‡, Sujian Li†
† National Key Laboratory for Multimedia Information Processing,
School of Computer Science, Peking University
<EMAIL_ADDRESS>
‡ Huawei Technologies
<EMAIL_ADDRESS>∗Equal contribution.

###### Abstract

In today’s fast-paced world, the growing demand to quickly generate
comprehensive and accurate Wikipedia documents for emerging events is both
crucial and challenging. However, previous efforts in Wikipedia generation
have often fallen short of meeting real-world requirements. Some approaches
focus solely on generating segments of a complete Wikipedia document, while
others overlook the importance of faithfulness in generation or fail to
consider the influence of the pre-training corpus. In this paper, we simulate
a real-world scenario where structured full-length Wikipedia documents are
generated for emergent events using input retrieved from web sources. To
ensure that Large Language Models (LLMs) are not trained on corpora related to
recently occurred events, we select events that have taken place recently and
introduce a new benchmark WikiGenBen, which consists of 309 events paired with
their corresponding retrieved web pages for generating evidence. Additionally,
we design a comprehensive set of systematic evaluation metrics and baseline
methods, to evaluate the capability of LLMs in generating factual full-length
Wikipedia documents. The data and code are open-sourced at
WikiGenBench111https://github.com/zhzihao/WikiGenBench.

##  1 Introduction

Wikipedia serves as an important reference for individuals seeking
comprehensive and reliable overviews across a diverse range of topics (Singer
et al., 2017; Lemmerich et al., 2019). Many NLP tasks, especially knowledge-
intensive tasks, also increasingly demand support from Wikipedia to provide
rich and accurate information. For instance, information retrieval (Lehmann et
al., 2015), question-answering systems (Chen et al., 2017), and automatic
summarization (Liu et al., 2018) tasks all rely on extracting information from
Wikipedia to accomplish their objectives. Despite the growing reliance of
these tasks on Wikipedia, the task of automatically generating Wikipedia
content in real-world scenarios has not been fully explored.

Figure 1: An example of the proposed Wikipedia generation system.

Up to now, the primary methods for generating Wikipedia documents rely on a
large community of volunteer editors who play a crucial role in ensuring
accuracy, relevance, credibility and completeness. As the volume of
information on the internet explodes, previous manual methods fail to keep up
with increasingly emergence of various new topics, especially when the
relevant information is scattered around the Internet
222https://en.wikipedia.org/wiki/Wikipedia:Statistics. Generating Wikipedia
also requires a certain level of expertise in the subject. Therefore, an
efficient approach to automatically generating high-quality, full-length and
factual Wikipedia documents (Fig 1) becomes exceptionally challenging.

Many attempts have been made to generate Wikipedia documents. The early works
(Sauper and Barzilay, 2009) are mainly based on information extraction
framework to constitute Wikipedia documents from retrieved information. With
the development of text generation techniques, recent works widely adopt
summarization system and large language models with retrievers (Liu et al.,
2018; Menick et al., 2022; Qian et al., 2023).

However, these works do not align with real scenario in that: 1) they usually
target at generating segments of a complete Wikipedia document like first
paragraphs or short articles rather than a full-length structured article; 2)
not much emphasis has been placed on the faithfulness of Wikipedia generation;
3) they are not focused on emergent topics which means the generation models
may have seen the Wikipedia article during pre-training stage, weakening the
validity of their results. Therefore, these approaches are not suitable for
real-life production scenarios in which systems need to generate an entirely
new Wikipedia entry for an emergent topic based on provided information. In
addition, the context window length of large language models also becomes a
constraint on Wikipedia generation when the retrieved documents are lengthy.
It is also challenging to effectively assess the quality of a structured
Wikipedia article, due to the lack of proper evaluation methods.

Given the aforementioned circumstances, we propose a new benchmark WikiGenBen
based on real-life scenarios. We underscore the accessibility of retrieved
documents which can be easily obtained from the internet. Besides, we require
the system’s output to be as comprehensive as possible. It must follow the
usual structure of Wikipedia including title, introduction, body and
references. To prevent training data leakage for fair comparison, we
deliberately filter out Wikipedia articles that are created more than a
certain amount of time (e.g. a year) ago. Previous works generally regard
Wikipedia as a multi-document summarization task (Liu et al., 2018) and adopt
similar evaluation methods to measure the similarity between generated output
and ground truth. We argue that there may not necessary be a “gold answer".
The generated outputs are informative as long as the information in the ground
truth Wiki-document is preserved or paraphrased. In addition to comparing the
generated results with ground truth, we should also pay more attention to
whether the generated outputs are supported by relevant information. It is
common that the references contradict with the parametric knowledge stored in
LLMs. We also design a coarse-to-fine multi-stage retrieval augmented
generation framework RPRR (see section §4) to address major obstacles in
Wikipedia generation. Besides, we conduct thorough analysis on the impact of
different retrieval methods and LLM selection for this challenging task.

The main contributions of this work can be summarized as follows:

  * •

We introduce a new task, generate full-length Wikipedia for emergent events
with documents retrieved from a large web corpus as citations.

  * •

We construct a new benchmark WikiGenBen with 309 emergent events, and
introduce novel evaluation metrics to measure the performance of existing
methods from three aspects of fluency, informativeness and faithfulness.

  * •

The impact of different components in RAG methods for Wikipedia generation is
comprehensively analyzed. Our findings both highlight substantial potential
for enhancement and provide guidance for future research on this intricate
task.

We hope our work can contribute to the advancement in the domain of generating
Wikipedia content.

##  2 Retrieval-based Wikipedia Generation Benchmark

###  2.1 Task Definition

We formalize full-length Wikipedia generation task as follows:

For a given event $\mathbf{E}$ and its $L$ related documents
$\mathcal{C}_{\mathbf{E}}$ obtained through search engine or human editors’
manual extracting from a large external corpus $\mathcal{C}$ (the internet),
this task aims to generate a full-length Wikipedia article $\mathcal{W}$ with
$M$ sentences:

$\displaystyle\mathcal{D}_{\mathbf{E}}$
$\displaystyle=\\{D_{\mathbf{E}}^{1},...,D_{\mathbf{E}}^{L}\\}=\textit{Retriever}(\mathbf{E},\mathcal{C}_{\mathbf{E}})$
(1) $\displaystyle\mathcal{W}_{\mathbf{E}}$
$\displaystyle=\\{S_{\mathbf{E}}^{1},...,S_{\mathbf{E}}^{M}\\}=\textit{Generator}(\mathbf{E},\mathcal{D}_{\mathbf{E}})$
(2) $\displaystyle\forall i\;\exists$ $\displaystyle j\;,S_{\mathbf{E}}^{i}\in
T_{\mathbf{E}}^{j},\mathcal{W}_{\mathbf{E}}=T_{\mathbf{E}}^{1}\cup
T_{\mathbf{E}}^{2}\cup...T_{\mathbf{E}}^{N}$ (3)

Since we are considering a structured Wikipedia, each sentence belongs to one
of the sections $T=\\{T_{1},...,T_{N}\\}$ with corresponding titles. Each
sentence $S_{i}$ should also have its own references
$R_{i}\subset\mathcal{D}_{\mathbf{E}}$, which may be composed by several
retrieved documents. The generator is usually a seq2seq model (Qian et al.,
2023) or an auto-regressive model (Ouyang et al., 2022), which is responsible
for processing retrievers’ input and generating final outputs.

###  2.2 Dataset Construction

During dataset construction, we meticulously select Wikipedia entries from the
HuggingFace November 2023 checkpoint, ensuring their exclusion from the March
2022 checkpoint. As we mainly experiment on GPT-3.5, which is pre-trained on
corpus before September 2021, opting for entries postdating this date can help
mitigate potential training data leakage. Additionally, we maintain entries
with word counts between 1000 to 3000. This selection criteria not only aligns
with the typical length of Wikipedia documents but also helps filter out low-
resource entries that may not serve as good evaluation cases.

Our processed collections include Wikipedia articles, their sections,
infoboxes, and crawlable reference links provided by human editors. We also
utilize Google’s search API333https://www.valueserp.com/ to extract up to 10
relevant web pages based on article titles. Thus, we obtain related documents
$C_{E}$ from two sources: human editors and search engines. It is important to
note that we wish to compare the outcomes between editor-provided and search-
retrieved related documents, with the aim to explore whether real-world
scenarios which rely solely on search engine results, can support the
generation of Wikipedia documents.

Given that most Wikipedia entries have no more than five crawlable reference
links (Liu et al., 2018), for the sake of fairness, we focus on entries with
10 to 100 links and a link crawling success rate above 40%, ensuring we gather
enough information to be comparable to that from search engines. Finally, we
filter Wikipedia entries focused on only event types, yielding a dataset of
309 entries featuring articles beyond the model’s prior knowledge, which is
particularly important for challenging and evaluating the model’s accuracy in
presenting factual information.

###  2.3 Dataset Statistics

Data Source | Dataset Statistics |
---|---|---
Wikipages | Sections (avg.) | 5.12
Word count (avg.) | 1639
Related documents by human editor | # Related docs (avg.) | 12.23
Word count (avg.) | 44k
Related documents by search engine | # Related docs (avg.) | 6.5
Word count (avg.) | 89k
Wikipages + Related documents | Events | 309
# Related docs | 5788
Word count | 41M

Table 1: Statistics of WikiGenBen dataset. We report the statistics of
Wikipages and related documents.

Table 1 outlines the statistics of the WikiGenBen dataset, which totally
consists of 41 million words across 309 Wikipedia entries and 5,788 related
documents. Wikipedia articles in the dataset have an average of 1,639 words
and 5.12 sections. Related documents curated by humans average 44,109 words,
whereas documents retrieved via search engine average 89,732 words. These
related documents substantially cover the information and knowledge about the
events, making them suitable input for automatic generation of Wikipedia
documents.More details can be found in Appendix A

##  3 Evaluation Metrics

A wide variety of evaluation metrics should be adopted to give a comprehensive
and fair evaluation of Wiki-generation performance. With reference to previous
work Gao et al. (2023a); Qian et al. (2023), we measure one generated wiki
document from the following three dimensions according to the characteristics
of the proposed benchmark: (1) Fluency: whether the generated text is fluent
and coherent; (2) Informativeness: whether the generated wiki document contain
enough information to describe the targeted event; (3) Faithfulness: whether
the generated content is faithful to or supported by the inputted information.
Figure 2 lists all the metrics for each dimension, which will be introduced
next.

###  3.1 Fluency

Evaluating fluency typically involves methods like MAUVE (Pillutla et al.,
2021). However, such methods exhibit instability when assessing long texts
(Gao et al., 2023b). Since effective evaluation methods for long texts are
lacking and rely heavily on manual scoring, as the capabilities of GPT-4
continue to advance, there has been a noticeable surge in its adoption for
long text evaluation tasks (Zheng et al., 2023). Given our need to generate
full-length Wikipedia, we employ GPT-4 to assign scores from 0 to 5 to assess
text fluency, and name it Fluent Score, the detailed prompt is provided in
Appendix B.

Figure 2: Overview of evaluation metrics.

###  3.2 Informativeness

We aim to evaluate whether the generated Wikipedia can cover various aspects
of the information related to specific events. We utilize metrics based on
n-gram overlapping between generated Wikipedia and reference Wikipedia like
BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and ROUGE
(Lin, 2004). A higher value indicates the generated text is more similar to
the reference text. These metrics are computed using the NLG-eval package
(Sharma et al., 2017).

Since there may not necessary be a “gold answer", we also employ GPT-4 to
score the informativeness from several aspects including Outline Score, Focus
Score and Info Score . Since Wikipedia documents are structured into sections,
we adopt a hierarchical approach to evaluate informativeness. Initially, we
evaluate the quality of section titles using Outline Score. Subsequently, we
assess the degree to which the generated content aligns with these titles,
termed Focus Score. This method facilitates a detailed examination of the
entirety of Wikipedia. Additionally, we compute the overall richness of
information with Info Score. All three metrics utilize GPT-4 for scoring on a
scale of 0-5, using the prompt provided in the Appendix B.

Furthermore, we introduce IB Score inspired by the QG-QA method (Wang et al.,
2020; Durmus et al., 2020) to extract questions from the Info Box in the
referenced Wikipedia and answer them solely based on the generated Wikipedia.
This allows us to determine whether the generated Wikipedia adequately covers
key details. The detailed calculation process is summarized in Figure 3.

Figure 3: Calculation process of IB Score (InfoBox Score).

###  3.3 Faithfulness

Building upon the definition presented in Min et al. (2023), our evaluation
revolves around determining whether a given fact is supported by the provided
reference documents $\mathcal{D}_{\mathbf{E}}$. We investigate whether
documents of citations can support the statements by utilizing the T5-11B
model from Honovich et al. (2022) to determine if statements can be
corroborated, which is also used in Gao et al. (2023b).

The proportion of words in the entire text that can be substantiated, the
Citation Rate, serves as a measure of the article’s authenticity.
Additionally, we employ methods from Gao et al. (2023b) to evaluate the
quality of model-generated citations, including Citation Recall which
determines if the output is entirely supported by cited passages, and Citation
Precision, which identifies any irrelevant citations.

Due to the need for processing a large amount of text citations, the method
introduced in Gao et al. (2023b), which involves individually excluding each
citation to observe if the remaining ones can still support the sentences,
incurs a high time complexity. Therefore, we differentiate from it that
determining whether citation $R$ can support sentence $S$ separately. If there
exists a citation that can support the sentence determined by the NLI model,
the recall for this sentence is 1. For precision, we calculate the number of
supporting citations in the sentence divided by the total number of citations.
Citation Recall and Citation Precision are averaged over all sentences.
Citation Rate primarily considers word-level aspects, calculating the total
number of words in sentences with recall equal to 1 divided by the total
number of words in the entire document, to address potential unevenness in
sentence length during citation generation.

##  4 Baseline Methods

The retrieval-based generation of full-length Wikipedia articles is a common
scenario ideally suited for the RAG method (Lewis et al., 2020). Initially, we
investigate the efficacy of the "Retrieve-then-Read" framework (Ma et al.,
2023). Subsequently, we employ a generator incorporating multi-retrieval,
referred to as "Retrieve-Plan-Retrieve-Read". For both frameworks, we utilize
DPR (Karpukhin et al., 2020) as the retriever and examine the influence of
various language model architectures and frameworks.

RR (Retrieve-then-Read) The "Retrieve-then-Read" is a standard retrieval-
augmented approach featuring a retriever that extracts $L$ related documents
$\mathcal{D}_{\mathbf{E}}$ to the given event $\mathbf{E}$. It employs a
frozen LLM as the generator to read the documents $\mathcal{D}_{\mathbf{E}}$
and directly produce the Wikipedia article $\mathcal{W}$.

RPRR (Retrieve-Plan-Retrieve-Read) Due to the limited context window of the
existing LLMs, the "Retrieve-then-Read" strategy can only handle a small
amount of documents. As we define a structured Wikipedia generation task, we
use a "Retrieve-Plan-Retrieve-Read" framework to overcome this problem. We
first utilize the frozen LLM to generate the titles of sections
$T_{\mathbf{E}}$ according to the retrieved documents, and then for each
section $T_{\mathbf{E}}^{i}$, we employ the "Retrieve-then-Read" method to
generate the content of $T_{\mathbf{E}}^{i}$. We combine the content of each
section together to get the final Wikipedia article $\mathcal{W}$. We test the
performance of different LLMs compared to its Retrieve-then-Read framework.

Models | Fluency | Informativeness | Faithfulness | Length
---|---|---|---|---
| Fluent
---
Score
B-4 | MET | R-L |  | IB
---
Score
| Info
---
Score
| Focus
---
Score
| Outline
---
Score
| Cit.
---
Rate
| Cit.
---
Recall
| Cit.
---
Prec.
Retrieve-then-Read (RR)
GPT-3.5-turbo | 4.31 | 1.06 | 10.73 | 17.81 | 12.91 | 3.49 | 3.21 | 2.86 | 42.09 | 38.78 | 36.7 | 579.1
GPT-3.5-turbo-16k | 4.29 | 0.9 | 10.29 | 17.42 | 10.72 | 3.39 | 3.15 | 2.84 | 38.38 | 33.98 | 32.68 | 541.3
Llama2-13b-chat | 3.97 | 0.82 | 9.74 | 15.89 | 9.8 | 3.03 | 2.51 | 2.39 | 7.91 | 8.91 | 8.91 | 552.9
Vicuna-13b-v1.5 | 4.18 | 0.86 | 9.8 | 17.33 | 12.35 | 3.39 | 2.7 | 2.27 | 16.88 | 11.03 | 10.7 | 491.8
Vicuna-7b-v1.5 | 4.06 | 0.96 | 10.18 | 17.34 | 11.11 | 3.27 | 2.49 | 1.61 | 6.4 | 4.41 | 4.38 | 535.2
Retrieval-Plan-Retrieval-Read (RPRR)
GPT-3.5-turbo | 4.02 | 4.18 | 22.29 | 22.26 | 13.01 | 3.78 | 3.65 | 2.76 | 50.96 | 53.47 | 52.43 | 1991.2
GPT-3.5-turbo-16k | 4.02 | 4.18 | 22.24 | 22.27 | 11.9 | 3.76 | 3.67 | 2.76 | 50.6 | 53.57 | 52.53 | 1988.9
Llama2-13b-chat | 3.7 | 3.58 | 24.18 | 20.94 | 11.9 | 3.29 | 3.26 | 2.7 | 12.77 | 14.3 | 14.3 | 3789.5
Vicuna-13b-v1.5 | 3.65 | 3.82 | 24.5 | 23.29 | 12.73 | 3.37 | 2.98 | 3.01 | 18.96 | 24.93 | 23.37 | 4182.9
Vicuna-7b-v1.5 | 2.89 | 3.25 | 23.43 | 21.87 | 11.03 | 2.61 | 2.46 | 2.71 | 14.89 | 22.4 | 22.08 | 5146.4

Table 2: Wikipedia generation results of different combination of LLMs and
generation methods. Cit. is short for citation, B-4 and R-L are short for
BLEU-4 and ROUGE-L, respectively. The GPT-4 scores are italicized and ranged
between 0 to 5 while the rest of metrics are ranged between 0-100.

##  5 Experiments

###  5.1 Experimental Settings

Our experiments are conducted using two variants of ChatGPT: GPT-3.5-turbo and
GPT-3.5-turbo-16k, as well as open-source LLMs, including instruction-tuned
versions of LLama2: LLama2-13b-chat (Touvron et al., 2023), Vicuna-7b-v1.5,
and Vicuna-13b-v1.5 (Chiang et al., 2023). FastChat is employed to enhance
inference efficiency in open-source LLMs. We refrain from testing the
performance of GPT-4 to avoid self-enhancement bias (Zheng et al., 2023), as
GPT-4 serves as an important judge in our evaluation.

Our retrieval methods include both traditional sparse retrievers like TF-IDF,
BM25 and advanced dense retrievers, specifically DPR (Karpukhin et al., 2020)
and GTR (Ni et al., 2022). We implement DPR-base and GTR-base as the dense
retrievers to generate embeddings for queries and documents, and then identify
the most relevant document chunks by calculating dot product similarities.

By default, we employ DPR as our retrieval method and adopt the RR strategy
unless otherwise specified. DPR is configured to select the top 5 most
relevant documents during the retrieval process. Note that we segment the
related documents into 256-word chunks (Borgeaud et al., 2022) before feeding
into retriever module, thus the retrieved documents $\mathcal{D}_{\mathbf{E}}$
are actually document chunks. This setting ensures efficient and effective
retrieval of relevant information for Wiki-generation task.

For the results showcased in Table 3, we utilize GPT-3.5 as the generator. For
the experiments demonstrated in Table 4, we switch to using GPT-3.5-16k. This
shift enables us to include a greater number of chunks in our analysis. The
same prompt (more details can be found in Appendix B) is employed for all
language models in all experiments, ensuring consistency in our approach
across the board.

###  5.2 Main Results

The results of various models and frameworks are shown in Table 2. Across all
LLMs, fluency scores are consistently high. GPT-3.5 stands out as the top
performer in this regard, achieving an impressive Fluent Score of 4.31. This
demonstrates that modern LLMs are exceptionally adept at generating coherent
and readable text, resembling the natural flow and grammatical correctness of
human writing.

However, achieving high fluency does not automatically guarantee that the
content is rich in information and faithful to factual accuracy. A notable
challenge observed across all models is their struggle to maintain the
reliability of the content produced. The best-performing models reach citation
metrics just above 50% and an IB Score around 10%, highlighting the complexity
involved in generating accurate and reliable content.

There is also a considerable disparity in the ability of LLMs to accurately
cite sources. The citation capabilities show considerable variation, with
GPT-3.5 achieving a citation rate of 42.09%, citation recall of 38.78%, and
citation precision of 36.70% in RR setting. This contrasts sharply with
smaller models, which do not meet a 20% threshold in any citation metric, even
with the application of RPRR setting.

An intriguing finding relates to the inconsistency between n-gram metrics and
more holistic evaluation criteria, such as those applied to GPT-4 metrics.
This issue is further explored and discussed in Section 5.5.

###  5.3 RR vs RPRR

In the RR settings, a Wikipedia article typically spans around 500 words in
length, whereas employing the RPRR approach can yield articles with
approximately 2000 words. This contrast underscores the efficacy of
structuring content generation according to the inherent organization of
Wikipedia, particularly its sections. It is important to note that the RPRR
approach involves conducting retrieval and reading operations within each
section separately during the generation process. As a result, the available
reference information for each section is richer. Consequently, while RPRR
generates significantly longer text compared to the RR method, the overall
citation coverage and the quality of citations are notably enhanced.

The Outline Score is quite low for small models in RR settings, implies
generating a formatted outline directly is a rather challenging task.
Therefore, generating titles first in RPRR setting can significantly improve
the situation of outlines generated by small models. Conversely, for powerful
models, directly generating outlines may perform better. This may because the
previously generated content helps the model better determine subsequent
content.For each type of LLM, the Focus Score is consistently higher in the
RPRR setting compared to the RR setting. This indicates that generating while
focusing on one section at a time allows the model to concentrate precisely on
the content of the topic.

The introduction of RPRR marks a significant improvement, particularly for
smaller models. This approach facilitates the extension beyond the constraints
of a model’s context window length, enabling the production of more focused
and relevant content within each section. The gains in informativeness and
faithfulness significantly enhance the quality of the generated articles. Our
experimentation underscores the essential role of task segmentation and
specialization in the content generation workflow. By initiating the process
with outline planning, smaller models demonstrate the ability to structure
content with a coherence level on par with GPT variants. This discovery points
new direction for research, particularly in isolating specific tasks such as
citation for further optimization.

# Docs | Informativeness | Faithfulness
---|---|---
B-4 | MET | R-L |  | IB
---
Score
| Cit.
---
Rate
| Cit.
---
Recall
| Cit.
---
Pre.
0 | 0.87 | 10.51 | 16.22 | 6.96 | - | - | -
5 | 0.90 | 10.29 | 17.42 | 10.72 | 38.38 | 33.98 | 32.68
10 | 0.98 | 10.54 | 17.80 | 10.73 | 37.70 | 34.75 | 32.80
15 | 1.04 | 10.84 | 18.09 | 11.67 | 35.47 | 32.41 | 30.22
20 | 1.03 | 10.99 | 18.44 | 11.78 | 34.80 | 32.85 | 30.85

Table 3: Impact of the Number of Retrieved Documents,more details can be seen
in table 6.

###  5.4 Impact of Retrieval Settings

Number of Retrieved Documents: Table 3 examines how the number of retrieved
documents affects content generation. Direct generation without retrieval
resulted in good n-gram scores but was susceptible to factual inaccuracies, as
shown by the IB score which implies that LLMs may experience hallucinations
during generation. Adding just five document chunks boosted the IB score by
3.76%, Retrieving more chunks increased the length and richness of the
generated text at the cost of citation performance. The quality of citations
peaked at 10 retrieved documents and gradually declined thereafter, indicating
that the model may struggle to effectively handle an excessive number of
reference documents. Therefore, choosing the right number of retrieved
documents is crucial for balancing the content’s informativeness and
faithfulness.

Retrieval Method | Informativeness | Faithfulness
---|---|---
B-4 | MET | R-L |  | IB
---
Score
| Cit.
---
Rate
| Cit.
---
Recall
| Cit.
---
Pre.
Random | 0.78 | 9.99 | 16.66 | 8.56 | 30.31 | 27.26 | 26.16
TF-IDF | 1.01 | 10.67 | 18.10 | 13.47 | 49.31 | 50.32 | 48.34
BM25 | 0.88 | 10.17 | 17.38 | 13.01 | 46.50 | 46.74 | 44.10
DPR | 1.06 | 10.73 | 17.81 | 12.91 | 42.09 | 38.78 | 36.70
GTR | 1.02 | 10.46 | 17.89 | 13.13 | 44.80 | 41.97 | 40.21
golden | 7.75 | 19.12 | 41.04 | 18.86 | 69.32 | 58.27 | 54.85

Table 4: Performance of different retrievers,more details can be seen in table
7.

Sparse Retriever vs Dense Retriever: Table 4 displays the performance of
traditional sparse retrievers compared to dense retrievers. We utilized
original Wikipedia pages as the documents for retrieval to establish golden
retrieval results and employed a random selection method as the baseline,
setting clear benchmarks for the best and worst possible outcomes of our
experiments. As shown in Table 4 and Figure 4, term-matching sparse retrievers
especially TF-IDF outperform dense retrievers. This result aligns with
findings by Sciavolino et al. (2022) that dense retrievers often struggle to
identify rare entities not included during training, a significant issue
considering Wikipedia’s frequent inclusion of such entities. In the RPRR
setting, sparse retrievers encounter difficulties with queries spanning
different article sections due to their limited semantic understanding,
essential for accurate section-specific retrieval. Improving dense retrievers
to recognize rare entities could markedly enhance performance, aligning better
with our task.

Figure 4: GPT-4 evaluation of different retrievers.

###  5.5 Is GPT-4 a Better Judge?

In most cases, we observe that n-gram metrics exhibit a trend similar to GPT-4
scores. However, an anomaly is noticed with the Vicuna-7b model in the RPRR
setting: while comparing to any other method in the RR setting, it achieves a
much higher score in n-gram-based metrics, including a 23.29% ROUGE-L score,
but a notably lower Info Score at 2.61. Faced with this discrepancy, we
manually inspect the results of the Vicuna-7b-1.5 model in the RPRR setting
and find that the content is primarily a restatement of the retrieved
documents without effectively focusing on the title.

This lack of focus is also evident from the relatively poor Focus Score at
2.46. Despite the extensive coverage of various aspects in the retrieved
contents, it results in longer generated text, but its informativeness about
the events is actually quite poor due to the presence of numerous irrelevant
details. We find it challenging to differentiate this aspect based solely on
n-gram scoring. In this regard, the GPT-4 score may be relatively more
reliable.

| Related
---
Document
Source
B-4 | MET | R-L |  | IB
---
Score
| Info
---
Score
| Focus
---
Score
| Outline
---
Score
Search Engine | 0.91 | 10.23 | 17.54 |  12.67 | 3.51 |  3.16 |  2.84
Human Editor |  0.93 |  10.36 |  17.7 | 11.13 | 3.43 | 3.1 | 2.77
Mixed | 1.07 | 10.79 | 17.89 | 12.91 |  3.49 | 3.21 | 2.86

Table 5: Influence of different related document source. The best performance
is highlighted in bold, the second best performance is underlined. Search
engine alone achieves comparable results compared to human editors,more
details can be seen in table 8.

###  5.6 Effectiveness of Search-Retrieved Related Documents

In Table 5, we present the impact of utilizing related documents from
different sources. Our primary focus is on how the source of these documents
affects informativeness. While it is commonly believed that references
provided by human editors are of higher quality (Liu et al., 2018; Qian et
al., 2023), our findings suggest that for emergent events, search engine
results can be equally effective, if not more so.

The analysis shows that a mix of search engine and human editor sources offers
the best performance, achieving the highest scores across several metrics.
Sole reliance on search engine retrieved documents not only matches but
occasionally surpasses the performance of documents provided by human editors.
Specifically, search engine achieves an Info Score and Focus Score of 3.51 and
3.16 respectively, compared to human editor’s 3.43 and 3.10. This
effectiveness is likely due to the inherent nature of emergent events in the
digital age, where search engines provide access to a wide array of up-to-date
and extensive news sources, thereby ensuring a breadth and depth of
information that rivals or exceeds what human editors can compile.

##  6 Related Work

###  6.1 Automated Wikipedia Generation

Automatically generating Wikipedia documents has been widely studied for more
than a decade. However, most of these works are constrained to information
extraction rather than text generation. Sauper and Barzilay (2009) propose a
structure-aware method to produce Wikipedia documents given relevant articles.
They first cluster all headings in the document to construct the templates
then retrieve result pages from the internet. This approach requires a set of
in-domain documents as prerequisites to constitute the template, thereby
rendering it unsuitable for addressing emerging topics. WikiWrite (Banerjee
and Mitra, 2016) captures the relation between referenced entities and the
target entities to classify retrieved information from the internet. The
introduction of summarization subsystem provides WikiWrite with the
paraphrasing ability to produce more coherent sentences. Liu et al. (2018)
address Wikipedia generation using multi-document summarization by designing a
decoder-only transformers model. Similar to previous works, it follows a two-
stage process of extraction then summarization. The most recent work WebBrain
(Qian et al., 2023) leverages a more complex system to ground on web corpus
and produce short factual articles to answer queries.

###  6.2 Retrieval-Augmented Text Generation

Enhancing language models (LMs) with retrieval during inference has become a
common practice for generative tasks. By first retrieving relevant document
excerpts from external knowledge base, retrieval-augmented generation models
exhibit the capacity to ground on references and facilitate the generation of
informative and faithful text (Li et al., 2022; Gao et al., 2023c; Guu et al.,
2020). In the era of LLMs, RAG has been proven to be an effective and
universal paradigm on a series of NLP tasks (Weston et al., 2018; Jiang et
al., 2023). Some studies have found that retrieval can be employed to fetch
in-context examples (Brown et al., 2020), thereby enhancing certain
capabilities of LLMs (Huang et al., 2023; Ram et al., 2023; Chen et al.,
2023).

Lewis et al. (2020) first introduce RAG models which combine pre-trained
seq2seq model with non-parametric dense vector index of Wikipedia and set new
state-of-the-art on open domain QA tasks. To alleviate LLMs’ tendency to
produce hallucination, some researchers (Nakano et al., 2021) propose to
integrate language model with result pages from search engine to compose the
final outcome. Depending on the complexity of retrieval strategy, the system
may conduct multiple retrieval process during generation and combine with
techniques like Chain of Thought (Trivedi et al., 2022; Feng et al., 2023).

##  7 Conclusion

We introduce a novel task aimed at generating full-length Wikipedia articles
based on emergent events through retrieval techniques, which closely aligns
with real-world scenarios. The proposed task leads to the creation of a new
benchmark WikiGenBen, which consists of a dataset for emergent events, paired
with related documents and comprehensive evaluation metrics to measure the
generation of fluent, informative, and faithful Wikipedia documents. Our
experiments shed light on the intricate balance between various metrics,
highlighting the task’s complexity. Moreover, we emphasis on the necessity of
refining task segmentation methods to further enhance the performance. Through
our work, we aim to contribute to the streamlining of knowledge dissemination
in this data-explosive digital era, ensuring the availability of timely,
reliable, and detailed information.

## Limitations

Our research encounters limitations, notably in our section-by-section
generation approach, which may lead to redundancy and necessitate a rewriting
strategy to ensure article cohesion. Further limitations include the challenge
of direct citation by LLMs, a bottleneck that might necessitate the
exploration of post-citation methods such as employing NLI for improvement.
Additionally, the information from related documents does not fully cover the
content of original Wikipedia articles, making the n-gram metric comparison
between the generated text and the original articles a weak reference rather
than a definitive standard.

## Ethics Statement

This work adheres to the ACL Ethics Policy. We assert that, to the best of our
knowledge, our work does not present any ethical issues. We have conducted a
thorough review of potential ethical implications in our research and found
none.

## References

  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Banerjee and Mitra (2016) Siddhartha Banerjee and Prasenjit Mitra. 2016.  Wikiwrite: Generating wikipedia articles automatically.  In _IJCAI_ , pages 2740–2746.
  * Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022.  Improving language models by retrieving from trillions of tokens.  In _International conference on machine learning_ , pages 2206–2240. PMLR.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading wikipedia to answer open-domain questions.  _arXiv preprint arXiv:1704.00051_.
  * Chen et al. (2023) Qinyu Chen, Wenhao Wu, and Sujian Li. 2023.  Exploring in-context learning for knowledge grounded dialog generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 10071–10081.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020.  FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5055–5070, Online. Association for Computational Linguistics.
  * Feng et al. (2023) Zhangyin Feng, Xiaocheng Feng, Dezhi Zhao, Maojin Yang, and Bing Qin. 2023.  Retrieval-generation synergy augmented large language models.  _arXiv preprint arXiv:2310.05149_.
  * Gao et al. (2023a) Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023a.  Enabling large language models to generate text with citations.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 6465–6488, Singapore. Association for Computational Linguistics.
  * Gao et al. (2023b) Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023b.  Enabling large language models to generate text with citations.  _arXiv preprint arXiv:2305.14627_.
  * Gao et al. (2023c) Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023c.  Retrieval-augmented generation for large language models: A survey.  _arXiv preprint arXiv:2312.10997_.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020.  Retrieval augmented language model pre-training.  In _International conference on machine learning_ , pages 3929–3938. PMLR.
  * Honovich et al. (2022) Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022.  TRUE: Re-evaluating factual consistency evaluation.  In _Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering_ , pages 161–175, Dublin, Ireland. Association for Computational Linguistics.
  * Huang et al. (2023) Jie Huang, Wei Ping, Peng Xu, Mohammad Shoeybi, Kevin Chen-Chuan Chang, and Bryan Catanzaro. 2023.  Raven: In-context learning with retrieval augmented encoder-decoder language models.  _arXiv preprint arXiv:2308.07922_.
  * Jiang et al. (2023) Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023.  Active retrieval augmented generation.  _arXiv preprint arXiv:2305.06983_.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Lehmann et al. (2015) Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. 2015.  Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia.  _Semantic web_ , 6(2):167–195.
  * Lemmerich et al. (2019) Florian Lemmerich, Diego Sáez-Trumper, Robert West, and Leila Zia. 2019.  Why the world reads wikipedia: Beyond english speakers.  In _Proceedings of the twelfth ACM international conference on web search and data mining_ , pages 618–626.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474.
  * Li et al. (2022) Huayang Li, Yixuan Su, Deng Cai, Yan Wang, and Lemao Liu. 2022.  A survey on retrieval-augmented text generation.  _arXiv preprint arXiv:2202.01110_.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2018) Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018.  Generating wikipedia by summarizing long sequences.
  * Ma et al. (2023) Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023.  Query rewriting for retrieval-augmented large language models.
  * Menick et al. (2022) Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. 2022.  Teaching language models to support answers with verified quotes.  _arXiv preprint arXiv:2203.11147_.
  * Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023.  FActScore: Fine-grained atomic evaluation of factual precision in long form text generation.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 12076–12100, Singapore. Association for Computational Linguistics.
  * Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021.  Webgpt: Browser-assisted question-answering with human feedback.  _arXiv preprint arXiv:2112.09332_.
  * Ni et al. (2022) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. 2022.  Large dual encoders are generalizable retrievers.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9844–9855, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Pillutla et al. (2021) Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021.  Mauve: Measuring the gap between neural text and human text using divergence frontiers.  _Advances in Neural Information Processing Systems_ , 34:4816–4828.
  * Qian et al. (2023) Hongjing Qian, Yutao Zhu, Zhicheng Dou, Haoqi Gu, Xinyu Zhang, Zheng Liu, Ruofei Lai, Zhao Cao, Jian-Yun Nie, and Ji-Rong Wen. 2023.  Webbrain: Learning to generate factually correct articles for queries by grounding on large web corpus.  _arXiv preprint arXiv:2304.04358_.
  * Ram et al. (2023) Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023.  In-context retrieval-augmented language models.  _arXiv preprint arXiv:2302.00083_.
  * Sauper and Barzilay (2009) Christina Sauper and Regina Barzilay. 2009.  Automatically generating wikipedia articles: A structure-aware approach.  In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_ , pages 208–216.
  * Sciavolino et al. (2022) Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2022.  Simple entity-centric questions challenge dense retrievers.
  * Sharma et al. (2017) Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017.  Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation.
  * Singer et al. (2017) Philipp Singer, Florian Lemmerich, Robert West, Leila Zia, Ellery Wulczyn, Markus Strohmaier, and Jure Leskovec. 2017.  Why we read wikipedia.  Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.
  * Trivedi et al. (2022) Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022.  Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions.  _arXiv preprint arXiv:2212.10509_.
  * Wang et al. (2020) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020.  Asking and answering questions to evaluate the factual consistency of summaries.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5008–5020, Online. Association for Computational Linguistics.
  * Weston et al. (2018) Jason Weston, Emily Dinan, and Alexander H Miller. 2018.  Retrieve and refine: Improved sequence generation models for dialogue.  _arXiv preprint arXiv:1808.04776_.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023.  Judging llm-as-a-judge with mt-bench and chatbot arena.

##  Appendix A Example Data Entry for WikiGenBen Datasest

This section presents a detailed example of a data entry corresponding to the
"2023 USFL season", as discussed in Section 2, illustrating the dataset’s
structure and the richness of information it offers.

###  A.1 Wikipages

Our dataset includes detailed entries from Wikipedia pages. The acquisition of
this information is facilitated through the Python library MediaWiki, which
allows for systematic harvesting of data from Wikipedia.

• ID: 71284256 • Keyword: 2023 USFL season • URL:
https://en.wikipedia.org/wiki/2023%20USFL%20season • Summary: The 2023 USFL
season was the second season of the United States Football League. The regular
season started on April 15 and ended on June 18. The postseason began on June
24 and ended with the 2023 USFL Championship Game on July 1. The league
expanded the locations their teams play to four total stadiums, adding Ford
Field in Detroit, Michigan, and Simmons Bank Liberty Stadium in Memphis,
Tennessee. • Sections: Offseason, Locations, Teams, Players, … • Content for
each section: 1. During the 2022 season, … 2. The league stated its … 3. On
November 15, 2022, in conjunction… 4. For the 2023 season, each USFL team… 5.
… • Infobox:

Key Value League United States Football League Sport American football
Duration Regular season: April 15 – June 18 Playoffs: June 24 – July 1 … …

###  A.2 Related Documents

Beyond the core Wikipedia page entries, our dataset is enriched with related
documents. These documents are categorized into two types: ’Human’ and
’Search’. ’Human’-sourced documents are derived from the Wikipedia Reference
Section, offering a layer of human-curated and highly credible information. On
the other hand, ’Search’-sourced documents are obtained through Google
searches, broadening the dataset’s horizon with a diverse range of
perspectives and additional context.

• Doc ID: 1 • Title: Johnson, Roy S. (2022-11-14). ÜSFL reveals season 2
details for Birmingham.̈ al. Retrieved 2022-12-13. • URL:
https://www.al.com/news/2022/11/usfl-reveals-season-2-details-for-
birmingham.html • Content: usfl reveals season 2 details for… • Source: Human
/ Search

##  Appendix B Prompt Design

Our method involves creating a base template for the prompt, which is then
supplemented with relevant documents until reaching a maximum input length.
Specifically, for 4k models, the maximum input length is strictly capped at
2048 tokens.

###  B.1 Retrieve-then-Read

The RR approach involves a straightforward, one-stage process for directly
generating an article.

Article Generation Prompt Input: I have a topic "{keyword}" that contains the
following documents: Document 1: {doc1} Document 2: {doc2} … Based on the
above information, you are assigned to write a Wikipedia article on the topic.
Organize the content of your article by sections. Before writing each section,
always starts with "==SECTION NAME==". You must cite the most relevant
document for every sentence you write, in the format of "This is an example
sentence.[k]", where k denotes Document k.

###  B.2 Retrieve-Plan-Retrieve-Read

In contrast, the RPRR method necessitates a more structured approach to
article generation. Initially, it requires the planning of an article outline.
Once the outline is established, each section name generated during the
planning phase serves as a guide for the subsequent retrieval and writing
phases.

Outline Generation Prompt Input: I have a topic "{keyword}" that contains the
following documents: Document 1: {doc1} Document 2: {doc2} … Based on the
above information, you are assigned to write an outline for a Wikipedia
article about this topic. Your outline should only include the names of the
sections, without any further details. Do not use document name as your
outline. The format of your outline should be as follows: 1\. Introduction 2\.
<Section Name 1> … n. <Section Name n>

Section Generation Prompt Input: I have a topic "{keyword}" and a section
"{section}" that contains the following documents: Document 1: {doc1} Document
2: {doc2} … Based on the above information, you are assigned to write a
Wikipedia article on the topic. You must cite the most relevant document for
every sentence you write, in the format of "This is an example sentence.[k]",
where k denotes Document k.

###  B.3 Evaluation

Evaluation Prompt for Fluent Score and Informativeness Score Input: Evaluate a
encyclopedia text of a keyword in three
metrics,fluency,informativeness,faithfulness. Give a score from 0-5 to each
metric. Fluency: Assess the text for grammatical correctness, coherence of
ideas, and overall readability. Look for smooth transitions between sentences
and paragraphs, as well as clear organization of information. Informativeness:
Evaluate the depth and breadth of information provided about the keyword.
Check if the text covers various aspects of the topic, including its
definition, background, significance, related concepts, and any relevant
examples or applications. Faithfulness: Verify the accuracy of the information
presented in the text by cross-referencing with credible sources or
established knowledge,assess whether the information aligns with accepted
facts and evidence. Only give three scores in this form:
Fluency:score1,Informativeness:score2 Faithfulness:socre3.No need to
Explaination.

Case the GPT-4 has a training data up to April 2023,but the events in our
benchmark has nearly 1/3 happens after April 2023,we don’t report the
faithfulness score which can not evaluate accurately.

Evaluation Prompt for Title Score Input: Given a keyword,and a outlines about
the wikipedia of the keyword, assign a score ranging from 0 to 5 to evaluate
the quality of the outlines.Only give the score without explaination.

Evaluation Prompt for Focus Score Input: Given a keyword, a subtitle, and
corresponding content, assess whether the generated content sufficiently
elaborates on the subtitle. Assign a score from 0 to 5 based on the following
criteria: \- **Score 5 (Excellent):** The content comprehensively covers the
subtitle, providing in-depth information and insights. \- **Score 4 (Good):**
The content is thorough, addressing key points of the subtitle effectively. \-
**Score 3 (Moderate):** The content provides a decent overview of the subtitle
but lacks some essential details. \- **Score 2 (Fair):** The content misses
crucial aspects related to the subtitle and needs improvement. \- **Score 1
(Poor):** The content focuses on irrelevant or unimportant aspects, failing to
address the subtitle adequately. \- **Score 0 (Unacceptable):** The content is
entirely incorrect or meaningless concerning the subtitle. Only reply the
score without explaniation.

Evaluation Prompt for IB Score Input for answering the questions: Given a
question and a passage,answer the question only according to the information
in the passage,if the passage don’t give any information about the
question,answer no information.Only give the answer without explanation. Input
for evaluating the answers: Given a question,a golden answer and a candidate
answer,if the candidate answer has the same meaning as the golden answer,reply
1,else 0.Only reply 0 or 1 without explaination"

##  Appendix C Generation Format

For generation, we require the model to append references in the format [1][2]
after each claim, where [1][2] signifies that the claim is supported by chunk1
and chunk2. Additionally, we generate titles before each sections, using ’==’
as delimiters, for example, ==Introduction==. We decompose the article
according to this format and evaluate the quality of references and the
content quality of titles.

An example of model output 2022 EFL League One play-off final:
==Introduction== The 2022 EFL League One play-off final is a highly
anticipated football match that will determine the final promotion spot to the
EFL Championship for the 2022-2023 season. The play-off final is the
culmination of the EFL League One play-offs, which involve the teams that
finish in the third to sixth positions in the league table. The winner of the
final will join the top two teams in the league in gaining promotion to the
higher division. This article will provide an overview of the 2022 EFL League
One play-off final, including the date, venue, and teams involved. ==Date and
Venue== The 2022 EFL League One play-off final is scheduled to take place on
Saturday, May 21, 2022.[2] The final will be held at Wembley Stadium in
London, England, which has been the traditional venue for the EFL play-off
finals.[2] Wembley Stadium is known for its iconic status in English football
and has a seating capacity of over 90,000 spectators.[2] ==Teams== The teams
that will compete in the 2022 EFL League One play-off final will be determined
through the League One play-off semi-finals. The semi-finals will involve the
teams that finish in the third to sixth positions in the League One table.[2]
The semi-final matches are scheduled to take place on Thursday, May 5, 2022,
and Friday, May 6, 2022.[2] The winners of the semi-finals will advance to the
final and compete for the promotion spot to the EFL Championship. ==Ticket
Information== Tickets for the 2022 EFL League One play-off final are expected
to go on sale in May.[3] The English Football League (EFL) will provide more
information on ticket sales closer to the date of the final.[3] In previous
years, the play-off finals have attracted a significant number of spectators,
with Wembley Stadium hosting capacity-limited crowds of just under 12,000.[3]
However, this year, with the return to full capacity crowds, the atmosphere is
expected to be even more electric. ==Broadcasting== The 2022 EFL League One
play-off final, along with the semi-finals, will be televised by Sky Sports,
allowing fans to watch the matches from the comfort of their homes.[3]
Additionally, highlights of the finals will be available on Quest, providing
fans with a recap of the key moments from the matches.[3] ==Conclusion== The
2022 EFL League One play-off final is set to be an exciting and highly
anticipated event in English football. Taking place at Wembley Stadium, the
final will determine the final promotion spot to the EFL Championship for the
2022-2023 season. With tickets expected to go on sale in May and the matches
being broadcasted on Sky Sports, fans will have the opportunity to witness the
drama and excitement of the play-off final.

##  Appendix D Impact of Retrieval Settings

Document Nums | Fluency | Informativeness | Faithfulness | Length
---|---|---|---|---
| Fluent
---
Score
B-1 | B-4 | MET | R-L |  | IB
---
Score
| Info
---
Score
| Focus
---
Score
| Outline
---
Score
| Cit.
---
Rate
| Cit.
---
Recall
| Cit.
---
Prec.
0 | 4.62 | 6.77 | 0.87 | 10.51 | 16.22 | 6.96 | 3.70 | 3.23 | 2.91 | 0 | 0 | 0 | 574.7
5 | 4.29 | 6.10 | 0.90 | 10.29 | 17.42 | 10.72 | 3.39 | 3.15 | 2.84 | 38.38 | 33.98 | 32.68 | 541.3
10 | 4.30 | 6.45 | 0.98 | 10.54 | 17.8 | 10.73 | 3.44 | 3.18 | 2.8 | 37.70 | 34.75 | 32.8 | 559.9
15 | 4.29 | 7.00 | 1.04 | 10.84 | 18.09 | 11.67 | 3.52 | 3.22 | 2.83 | 35.47 | 32.41 | 30.22 | 583.2
20 | 4.30 | 7.09 | 1.03 | 10.99 | 18.44 | 11.78 | 3.55 | 3.22 | 2.82 | 34.80 | 32.85 | 30.85 | 584.9

Table 6: Impact of the Number of Retrieved Documents

Retrieval Method | Fluency | Informativeness | Faithfulness | Length
---|---|---|---|---
| Fluent
---
Score
B-1 | B-4 | MET | R-L |  | IB
---
Score
| Info
---
Score
| Focus
---
Score
| Outline
---
Score
| Cit.
---
Rate
| Cit.
---
Recall
| Cit.
---
Prec.
Random | 4.17 | 5.86 | 0.78 | 9.99 | 16.66 | 8.56 | 3.23 | 3.04 | 2.80 | 30.31 | 27.26 | 26.16 | 534.0
TF-IDF | 4.37 | 6.62 | 1.01 | 10.67 | 18.10 | 13.47 | 3.59 | 3.30 | 2.89 | 49.31 | 50.32 | 48.34 | 576.9
BM25 | 4.39 | 5.87 | 0.88 | 10.17 | 17.38 | 13.01 | 3.61 | 3.25 | 2.83 | 46.50 | 46.74 | 44.10 | 546.8
DPR | 4.31 | 6.97 | 1.06 | 10.73 | 17.81 | 12.91 | 3.49 | 3.21 | 2.86 | 42.09 | 38.78 | 36.70 | 577.2
GTR | 4.33 | 6.31 | 1.02 | 10.46 | 17.89 | 13.13 | 3.54 | 3.20 | 2.86 | 44.80 | 41.97 | 40.21 | 560.0
golden | 4.46 | 13.1 | 7.75 | 19.12 | 41.04 | 18.86 | 4.04 | 3.28 | 2.63 | 69.32 | 58.27 | 54.85 | 632.1

Table 7: Performance of different retrievers.

Document Source | Fluency | Informativeness | Faithfulness | Length
---|---|---|---|---
| Fluent
---
Score
B-1 | B-4 | MET | R-L |  | IB
---
Score
| Info
---
Score
| Focus
---
Score
| Outline
---
Score
| Cit.
---
Rate
| Cit.
---
Recall
| Cit.
---
Prec.
Search Engine | 4.31 | 6 | 0.91 | 10.23 | 17.54 | 12.67 | 3.51 | 3.16 | 2.84 | 40.37 | 35.96 | 34.31 | 542.8
Human Editor | 4.31 | 6.05 | 0.93 | 10.36 | 17.7 | 11.13 | 3.43 | 3.1 | 2.77 | 38.70 | 35.71 | 33.76 | 540.8
Mixed | 4.31 | 7.04 | 1.07 | 10.79 | 17.89 | 12.91 | 3.49 | 3.21 | 2.86 | 42.09 | 38.78 | 36.70 | 579.1

Table 8: Influence of different related document source.


distribution of DM, the magnetization signal spreads with width
$\Delta\omega_{\mathrm{DM}}$ determined by that of DM field. We find that the
response of the system derived from quantum magnon picture is consistent to
that of classical theory given in table 3 and 3. On the other hand, the power
of the system can be derived from the relation $P=\hbar\omega_{\zeta}\langle
n_{\zeta}\rangle{2}/{T_{2\zeta}}$ where $n_{\zeta}=\zeta^{\dagger}\zeta$ is
the number operator.

## References

  * [1] Particle Data Group collaboration, _Review of Particle Physics_ , _PTEP_ 2022 (2022) 083C01.
  * [2] R.D. Peccei and H.R. Quinn, _CP Conservation in the Presence of Instantons_ , _Phys. Rev. Lett._ 38 (1977) 1440.
  * [3] R.D. Peccei and H.R. Quinn, _Constraints Imposed by CP Conservation in the Presence of Instantons_ , _Phys. Rev. D_ 16 (1977) 1791.
  * [4] S. Weinberg, _A New Light Boson?_ , _Phys. Rev. Lett._ 40 (1978) 223.
  * [5] F. Wilczek, _Problem of Strong $P$ and $T$ Invariance in the Presence of Instantons_, _Phys. Rev. Lett._ 40 (1978) 279.
  * [6] P. Svrcek and E. Witten, _Axions In String Theory_ , _JHEP_ 06 (2006) 051 [hep-th/0605206].
  * [7] A. Arvanitaki, S. Dimopoulos, S. Dubovsky, N. Kaloper and J. March-Russell, _String Axiverse_ , _Phys. Rev. D_ 81 (2010) 123530 [0905.4720].
  * [8] M. Cicoli, M. Goodsell and A. Ringwald, _The type IIB string axiverse and its low-energy phenomenology_ , _JHEP_ 10 (2012) 146 [1206.0819].
  * [9] J. Jaeckel and A. Ringwald, _The Low-Energy Frontier of Particle Physics_ , _Ann. Rev. Nucl. Part. Sci._ 60 (2010) 405 [1002.0329].
  * [10] J. Jaeckel, _A force beyond the Standard Model - Status of the quest for hidden photons_ , _Frascati Phys. Ser._ 56 (2012) 172 [1303.1821].
  * [11] P. Arias, D. Cadamuro, M. Goodsell, J. Jaeckel, J. Redondo and A. Ringwald, _WISPy Cold Dark Matter_ , _JCAP_ 06 (2012) 013 [1201.5902].
  * [12] M. Fabbrichesi, E. Gabrielli and G. Lanfranchi, _The Dark Photon_ , 2005.01515.
  * [13] CDMS-II collaboration, _Dark Matter Search Results from the CDMS II Experiment_ , _Science_ 327 (2010) 1619 [0912.3592].
  * [14] CDMS collaboration, _Search for Axions with the CDMS Experiment_ , _Phys. Rev. Lett._ 103 (2009) 141802 [0902.4693].
  * [15] SuperCDMS collaboration, _First Dark Matter Constraints from a SuperCDMS Single-Charge Sensitive Detector_ , _Phys. Rev. Lett._ 121 (2018) 051301 [1804.10697].
  * [16] XENON collaboration, _First Results from the XENON10 Dark Matter Experiment at the Gran Sasso National Laboratory_ , _Phys. Rev. Lett._ 100 (2008) 021303 [0706.0039].
  * [17] XENON collaboration, _Projected WIMP sensitivity of the XENONnT dark matter experiment_ , _JCAP_ 11 (2020) 031 [2007.08796].
  * [18] PandaX-II collaboration, _Dark Matter Results From 54-Ton-Day Exposure of PandaX-II Experiment_ , _Phys. Rev. Lett._ 119 (2017) 181302 [1708.06917].
  * [19] PandaX-II collaboration, _Constraining Dark Matter Models with a Light Mediator at the PandaX-II Experiment_ , _Phys. Rev. Lett._ 121 (2018) 021304 [1802.06912].
  * [20] SENSEI collaboration, _SENSEI: Direct-Detection Results on sub-GeV Dark Matter from a New Skipper-CCD_ , _Phys. Rev. Lett._ 125 (2020) 171802 [2004.11378].
  * [21] R. Barbieri, C. Braggio, G. Carugno, C.S. Gallo, A. Lombardi, A. Ortolan et al., _Searching for galactic axions through magnetized media: the QUAX proposal_ , _Phys. Dark Univ._ 15 (2017) 135 [1606.02201].
  * [22] D. Budker, P.W. Graham, M. Ledbetter, S. Rajendran and A. Sushkov, _Proposal for a Cosmic Axion Spin Precession Experiment (CASPEr)_ , _Phys. Rev. X_ 4 (2014) 021030 [1306.6089].
  * [23] D.F. Jackson Kimball et al., _Overview of the Cosmic Axion Spin Precession Experiment (CASPEr)_ , _Springer Proc. Phys._ 245 (2020) 105 [1711.08999].
  * [24] D. Aybas, J. Adam, E. Blumenthal, A.V. Gramolin, D. Johnson, A. Kleyheeg et al., _Search for axionlike dark matter using solid-state nuclear magnetic resonance_ , _Phys. Rev. Lett._ 126 (2021) 141802.
  * [25] Y. Hochberg, Y. Zhao and K.M. Zurek, _Superconducting Detectors for Superlight Dark Matter_ , _Phys. Rev. Lett._ 116 (2016) 011301 [1504.07237].
  * [26] Y. Hochberg, M. Pyle, Y. Zhao and K.M. Zurek, _Detecting Superlight Dark Matter with Fermi-Degenerate Materials_ , _JHEP_ 08 (2016) 057 [1512.04533].
  * [27] Y. Hochberg, T. Lin and K.M. Zurek, _Absorption of light dark matter in semiconductors_ , _Phys. Rev. D_ 95 (2017) 023013 [1608.01994].
  * [28] I.M. Bloch, R. Essig, K. Tobioka, T. Volansky and T.-T. Yu, _Searching for Dark Absorption with Direct Detection Experiments_ , _JHEP_ 06 (2017) 087 [1608.02123].
  * [29] Y. Hochberg, Y. Kahn, M. Lisanti, K.M. Zurek, A.G. Grushin, R. Ilan et al., _Detection of sub-MeV Dark Matter with Three-Dimensional Dirac Materials_ , _Phys. Rev. D_ 97 (2018) 015004 [1708.08929].
  * [30] A. Coskuner, A. Mitridate, A. Olivares and K.M. Zurek, _Directional Dark Matter Detection in Anisotropic Dirac Materials_ , _Phys. Rev. D_ 103 (2021) 016006 [1909.09170].
  * [31] T. Trickle, Z. Zhang, K.M. Zurek, K. Inzani and S.M. Griffin, _Multi-Channel Direct Detection of Light Dark Matter: Theoretical Framework_ , _JHEP_ 03 (2020) 036 [1910.08092].
  * [32] S.M. Griffin, K. Inzani, T. Trickle, Z. Zhang and K.M. Zurek, _Multichannel direct detection of light dark matter: Target comparison_ , _Phys. Rev. D_ 101 (2020) 055004 [1910.10716].
  * [33] R.M. Geilhufe, F. Kahlhoefer and M.W. Winkler, _Dirac Materials for Sub-MeV Dark Matter Detection: New Targets and Improved Formalism_ , _Phys. Rev. D_ 101 (2020) 055005 [1910.02091].
  * [34] T. Trickle, Z. Zhang and K.M. Zurek, _Effective field theory of dark matter direct detection with collective excitations_ , _Phys. Rev. D_ 105 (2022) 015001 [2009.13534].
  * [35] Y. Hochberg, Y. Kahn, N. Kurinsky, B.V. Lehmann, T.C. Yu and K.K. Berggren, _Determining Dark-Matter–Electron Scattering Rates from the Dielectric Function_ , _Phys. Rev. Lett._ 127 (2021) 151802 [2101.08263].
  * [36] S. Knapen, J. Kozaczuk and T. Lin, _Dark matter-electron scattering in dielectrics_ , _Phys. Rev. D_ 104 (2021) 015031 [2101.08275].
  * [37] S. Knapen, J. Kozaczuk and T. Lin, _python package for dark matter scattering in dielectric targets_ , _Phys. Rev. D_ 105 (2022) 015014 [2104.12786].
  * [38] A. Mitridate, T. Trickle, Z. Zhang and K.M. Zurek, _Dark matter absorption via electronic excitations_ , _JHEP_ 09 (2021) 123 [2106.12586].
  * [39] H.-Y. Chen, A. Mitridate, T. Trickle, Z. Zhang, M. Bernardi and K.M. Zurek, _Dark matter direct detection in materials with spin-orbit coupling_ , _Phys. Rev. D_ 106 (2022) 015024 [2202.11716].
  * [40] A. Mitridate, T. Trickle, Z. Zhang and K.M. Zurek, _Snowmass White Paper: Light Dark Matter Direct Detection at the Interface With Condensed Matter Physics_ , in _Snowmass 2021_ , 3, 2022 [2203.07492].
  * [41] T. Trickle, _Extended calculation of electronic excitations for direct detection of dark matter_ , _Phys. Rev. D_ 107 (2023) 035035 [2210.14917].
  * [42] S. Knapen, T. Lin, M. Pyle and K.M. Zurek, _Detection of Light Dark Matter With Optical Phonons in Polar Materials_ , _Phys. Lett. B_ 785 (2018) 386 [1712.06598].
  * [43] S. Griffin, S. Knapen, T. Lin and K.M. Zurek, _Directional Detection of Light Dark Matter with Polar Materials_ , _Phys. Rev. D_ 98 (2018) 115034 [1807.10291].
  * [44] P. Cox, T. Melia and S. Rajendran, _Dark matter phonon coupling_ , _Phys. Rev. D_ 100 (2019) 055011 [1905.05575].
  * [45] A. Mitridate, T. Trickle, Z. Zhang and K.M. Zurek, _Detectability of Axion Dark Matter with Phonon Polaritons and Magnons_ , _Phys. Rev. D_ 102 (2020) 095005 [2005.10256].
  * [46] D.J.E. Marsh, J.I. McDonald, A.J. Millar and J. Schütte-Engel, _Axion detection with phonon-polaritons revisited_ , _Phys. Rev. D_ 107 (2023) 035036 [2209.12909].
  * [47] T. Trickle, Z. Zhang and K.M. Zurek, _Detecting Light Dark Matter with Magnons_ , _Phys. Rev. Lett._ 124 (2020) 201801 [1905.13744].
  * [48] S. Chigusa, T. Moroi and K. Nakayama, _Detecting light boson dark matter through conversion into a magnon_ , _Phys. Rev. D_ 101 (2020) 096013 [2001.10666].
  * [49] A. Esposito and S. Pavaskar, _Optimal anti-ferromagnets for light dark matter detection_ , 2210.13516.
  * [50] S. Chigusa, M. Hazumi, E.D. Herbschleb, N. Mizuochi and K. Nakayama, _Light Dark Matter Search with Nitrogen-Vacancy Centers in Diamonds_ , 2302.12756.
  * [51] D.J.E. Marsh, K.-C. Fong, E.W. Lentz, L. Smejkal and M.N. Ali, _Proposal to Detect Dark Matter using Axionic Topological Antiferromagnets_ , _Phys. Rev. Lett._ 123 (2019) 121601 [1807.08810].
  * [52] J. Schütte-Engel, D.J.E. Marsh, A.J. Millar, A. Sekine, F. Chadha-Day, S. Hoof et al., _Axion quasiparticles for axion dark matter detection_ , _JCAP_ 08 (2021) 066 [2102.05366].
  * [53] S. Chigusa, T. Moroi and K. Nakayama, _Axion/hidden-photon dark matter conversion into condensed matter axion_ , _JHEP_ 08 (2021) 074 [2102.06179].
  * [54] A.V. Dixit, S. Chakram, K. He, A. Agrawal, R.K. Naik, D.I. Schuster et al., _Searching for Dark Matter with a Superconducting Qubit_ , _Phys. Rev. Lett._ 126 (2021) 141302 [2008.12231].
  * [55] S. Chen, H. Fukuda, T. Inada, T. Moroi, T. Nitta and T. Sichanugrist, _Detection of hidden photon dark matter using the direct excitation of transmon qubits_ , 2212.03884.
  * [56] P.G. de Gennes, P.A. Pincus, F. Hartmann-Boutron and J.M. Winter, _Nuclear magnetic resonance modes in magnetic material. i. theory_ , _Phys. Rev._ 129 (1963) 1105.
  * [57] A. Borovik-Romanov, Y.M. Bun’kov, B.S. Dumesh, M.I. Kurkin, M.P. Petrov and V. Chekmarev, _The spin echo in systems with a coupled electron-nuclear precession_ , _Soviet Physics Uspekhi_ 27 (1984) 235.
  * [58] Y. Shiomi, J. Lustikova, S. Watanabe, D. Hirobe, S. Takahashi and E. Saitoh, _Spin pumping from nuclear spin waves_ , _Nature Physics_ 15 (2019) 22.
  * [59] T. Kikkawa, D. Reitz, H. Ito, T. Makiuchi, T. Sugimoto, K. Tsunekawa et al., _Observation of nuclear-spin seebeck effect_ , _Nature Communications_ 12 (2021) 4356.
  * [60] S.M. Rezende, _Introduction to nuclear spin waves in ferro- and antiferromagnets_ , _Journal of Applied Physics_ 132 (2022) 091101.
  * [61] H. Suhl, _Effective nuclear spin interactions in ferromagnets_ , _Phys. Rev._ 109 (1958) 606.
  * [62] T. Nakamura, _Indirect Coupling of Nuclear Spins in Antiferromagnet with Particular Reference to MnF2 at Very Low Temperatures_ , _Progress of Theoretical Physics_ 20 (1958) 542.
  * [63] M. Date, _Magnetic resonance in mnco3_ , _Journal of the Physical Society of Japan_ 15 (1960) 2251.
  * [64] D. Shaltiel and H.J. Fink, _Nuclear antiferromagnetic double resonance in mnco3_ , _Journal of Applied Physics_ 35 (1964) 848.
  * [65] K. Lee, A.M. Portis and G.L. Witt, _Magnetic properties of the hexagonal antiferromagnet csmn ${\mathrm{f}}_{3}$_, _Phys. Rev._ 132 (1963) 144.
  * [66] V. Minkiewicz and A. Nakamura, _Direct observation of ${\mathrm{mn}}^{55}$ nmr in antiferromagnetic csmn${\mathrm{f}}_{3}$_, _Phys. Rev._ 143 (1966) 361.
  * [67] L.B. Welsh, _Properties of the ${\mathrm{mn}}^{55}$ nuclear-magnetic-resonance modes in csmn${\mathrm{f}}_{3}$_, _Phys. Rev._ 156 (1967) 370.
  * [68] M. Gorghetto and G. Villadoro, _Topological Susceptibility and QCD Axion Mass: QED and NNLO corrections_ , _JHEP_ 03 (2019) 033 [1812.01008].
  * [69] J.E. Kim, _Weak-interaction singlet and strong $\mathrm{CP}$ invariance_, _Phys. Rev. Lett._ 43 (1979) 103.
  * [70] M. Shifman, A. Vainshtein and V. Zakharov, _Can confinement ensure natural cp invariance of strong interactions?_ , _Nuclear Physics B_ 166 (1980) 493.
  * [71] M. Dine, W. Fischler and M. Srednicki, _A simple solution to the strong cp problem with a harmless axion_ , _Physics Letters B_ 104 (1981) 199.
  * [72] A.R. Zhitnitsky, _On Possible Suppression of the Axion Hadron Interactions. (In Russian)_ , _Sov. J. Nucl. Phys._ 31 (1980) 260.
  * [73] C.-Y. Chen and S. Dawson, _Exploring Two Higgs Doublet Models Through Higgs Production_ , _Phys. Rev. D_ 87 (2013) 055016 [1301.0309].
  * [74] Y. Ema, K. Hamaguchi, T. Moroi and K. Nakayama, _Flaxion: a minimal extension to solve puzzles in the standard model_ , _JHEP_ 01 (2017) 096 [1612.05492].
  * [75] L. Calibbi, F. Goertz, D. Redigolo, R. Ziegler and J. Zupan, _Minimal axion model from flavor_ , _Phys. Rev. D_ 95 (2017) 095009 [1612.08040].
  * [76] P.W. Graham, J. Mardon and S. Rajendran, _Vector Dark Matter from Inflationary Fluctuations_ , _Phys. Rev. D_ 93 (2016) 103520 [1504.02102].
  * [77] Y. Ema, K. Nakayama and Y. Tang, _Production of purely gravitational dark matter: the case of fermion and vector boson_ , _JHEP_ 07 (2019) 060 [1903.10973].
  * [78] P. Agrawal, N. Kitajima, M. Reece, T. Sekiguchi and F. Takahashi, _Relic Abundance of Dark Photon Dark Matter_ , _Phys. Lett. B_ 801 (2020) 135136 [1810.07188].
  * [79] R.T. Co, A. Pierce, Z. Zhang and Y. Zhao, _Dark Photon Dark Matter Produced by Axion Oscillations_ , _Phys. Rev. D_ 99 (2019) 075002 [1810.07196].
  * [80] M. Bastero-Gil, J. Santiago, L. Ubaldi and R. Vega-Morales, _Vector dark matter production at the end of inflation_ , _JCAP_ 04 (2019) 015 [1810.07208].
  * [81] N. Kitajima and F. Takahashi, _Resonant production of dark photons from axion without a large coupling_ , 2303.05492.
  * [82] J.A. Dror, K. Harigaya and V. Narayan, _Parametric Resonance Production of Ultralight Vector Dark Matter_ , _Phys. Rev. D_ 99 (2019) 035036 [1810.07195].
  * [83] K. Nakayama and W. Yin, _Hidden photon and axion dark matter from symmetry breaking_ , _JHEP_ 10 (2021) 026 [2105.14549].
  * [84] A.J. Long and L.-T. Wang, _Dark Photon Dark Matter from a Network of Cosmic Strings_ , _Phys. Rev. D_ 99 (2019) 063529 [1901.03312].
  * [85] N. Kitajima and K. Nakayama, _Dark Photon Dark Matter from Cosmic Strings and Gravitational Wave Background_ , 2212.13573.
  * [86] N. Kitajima and K. Nakayama, _Viable Vector Coherent Oscillation Dark Matter_ , 2303.04287.
  * [87] A. Caputo, A.J. Millar, C.A.J. O’Hare and E. Vitagliano, _Dark photon limits: A handbook_ , _Phys. Rev. D_ 104 (2021) 095029.
  * [88] S. Chaudhuri, P.W. Graham, K. Irwin, J. Mardon, S. Rajendran and Y. Zhao, _Radio for hidden-photon dark matter detection_ , _Phys. Rev. D_ 92 (2015) 075012 [1411.7382].
  * [89] J.B. Lee, W.G. Hong, H.J. Kim, Z. Jagličić, S. Jazbec, M. Wencka et al., _Canted antiferromagnetism on a nanodimensional spherical surface geometry: The case of mnco 3 small hollow nanospheres_, _Phys. Rev. B_ 86 (2012) 224407.
  * [90] W. Liang, L. Li, R. Li, Y. Yin, Z. Li, X. Liu et al., _Crystal structure of impurity-free rhodochrosite (mnco3) and thermal expansion properties_ , _Physics and Chemistry of Minerals_ 47 (2020) 9.
  * [91] K. Yoshida, _Magnetism (translated from Japanese)_ , Iwanami Shoten (2015).
  * [92] L.V. Abdurakhimov, Y.M. Bunkov and D. Konstantinov, _Normal-mode splitting in the coupled system of hybridized nuclear magnons and microwave photons_ , _Phys. Rev. Lett._ 114 (2015) 226402.
  * [93] L.V. Abdurakhimov, M.A. Borich, Y.M. Bunkov, R.R. Gazizulin, D. Konstantinov, M.I. Kurkin et al., _Nonlinear nmr and magnon bec in antiferromagnetic materials with coupled electron and nuclear spin precession_ , _Phys. Rev. B_ 97 (2018) 024425.
  * [94] W.J. Ince, _Coupled antiferromagnetic-nuclear-magnetic resonance in ${\mathrm{rbmnf}}_{3}$_, _Phys. Rev._ 184 (1969) 574.
  * [95] J.B. Merry and D.I. Bolef, _Nuclear acoustic resonance of ${\mathrm{mn}}^{55}$ in antiferromagnetic rbmn${\mathrm{f}}_{3}$_, _Phys. Rev. B_ 4 (1971) 1572.
  * [96] A.M. Portis, G.L. Witt and A.J. Heeger, _Excitation of nuclear magnetic resonance modes in antiferromagnetic kmnf3_ , _Journal of Applied Physics_ 34 (1963) 1052.
  * [97] A.R. King, V. Jaccarino and S.M. Rezende, _Nuclear magnons and nuclear magnetostatic modes in mn ${\mathrm{f}}_{2}$_, _Phys. Rev. Lett._ 37 (1976) 533.
  * [98] A.J. Heeger and T.W. Houston, _Nuclear magnetic resonance in ferrimagnetic mn ${\mathrm{fe}}_{2}$${\mathrm{o}}_{4}$_, _Phys. Rev._ 135 (1964) A661.
  * [99] E. Turov, I. Kurkin and V. Nikolaev, _Nuclear spin motion with allowance for suhl-nakamura interaction_ , _Soviet Physics JETP_ 37 (1973) 147.
  * [100] R.E. Sheriff and D. Williams, _Nuclear gyromagnetic ratios. iii_ , _Phys. Rev._ 82 (1951) 651.
  * [101] N. Stone, _Table of nuclear magnetic dipole and electric quadrupole moments_ , _Atomic Data and Nuclear Data Tables_ 90 (2005) 75.
  * [102] H. Fink and D. Shaltiel, _Nuclear frequency pulling in a dzialoshinskii-moriya-type weak ferromagnet: Mnc ${\mathrm{o}}_{3}$_, _Phys. Rev._ 136 (1964) A218.
  * [103] D. Shaltiel, _Nuclear magnetic resonance of mnc ${\mathrm{o}}_{3}$ in the canted spin state_, _Phys. Rev._ 142 (1966) 300.
  * [104] P.F.A. Klinkenberg, _Tables of nuclear shell structure_ , _Rev. Mod. Phys._ 24 (1952) 63.
  * [105] H. Ye, J. Li, , D. Yang, H. Jin and X. Huang, _Low-lying state investigations of odd-a mn isotopes around n = 28_ , _Communications in Theoretical Physics_ 75 (2023) 025302.
  * [106] Y.M. Bun’kov and B.S. Dumesh, _Dynamic effects in pulsed nmr in easy plane antiferromagnets with large dynamic frequency shifts_ , _Zh. Eksp. Teor. Fiz._ 68 (1975) .
  * [107] D. Aybas et al., _Search for Axionlike Dark Matter Using Solid-State Nuclear Magnetic Resonance_ , _Phys. Rev. Lett._ 126 (2021) 141802 [2101.01241].
  * [108] J.A.B. Mates, G.C. Hilton, K.D. Irwin, L.R. Vale and K.W. Lehnert, _Demonstration of a multiplexer of dissipationless superconducting quantum interference devices_ , _Applied Physics Letters_ 92 (2008) 023514.
  * [109] J.A.B. Mates, _The Microwave Squid Multiplexer_ , Ph.D. thesis, University of Colorado, 2011.
  * [110] S. Chaudhuri, P.W. Graham, K. Irwin, J. Mardon, S. Rajendran and Y. Zhao, _Radio for hidden-photon dark matter detection_ , _Phys. Rev. D_ 92 (2015) 075012.
  * [111] M. Tarasov, V. Belitsky and G. Prokopenko, _Dc squid rf amplifiers_ , _IEEE Transactions on Applied Superconductivity_ 2 (1992) 79.
  * [112] M. Mück, J.B. Kycia and J. Clarke, _Superconducting quantum interference device as a near-quantum-limited amplifier at 0.5 ghz_ , _Applied Physics Letters_ 78 (2001) 967.
  * [113] M. Mück, M.-O. André, J. Clarke, J. Gail and C. Heiden, _Radio-frequency amplifier based on a niobium dc superconducting quantum interference device with microstrip input coupling_ , _Applied Physics Letters_ 72 (1998) 2885.
  * [114] R. Therrien, _The microstrip SQUID amplifier_ , Ph.D. thesis, University of California, Berkeley, Jan., 2002.
  * [115] J.B. Miller, B.H. Suits, A.N. Garroway and M.A. Hepp, _Interplay among recovery time, signal, and noise: series- and parallel-tuned circuits are not always the same_ , _Concepts in Magnetic Resonance_ 12 (2000) 125.
  * [116] D. Hoult and R. Richards, _The signal-to-noise ratio of the nuclear magnetic resonance experiment_ , _Journal of Magnetic Resonance (1969)_ 24 (1976) 71.
  * [117] ADMX collaboration, _Design and performance of the ADMX SQUID-based microwave receiver_ , _Nucl. Instrum. Meth. A_ 656 (2011) 39 [1105.4203].
  * [118] R.H. Dicke, _The measurement of thermal radiation at microwave frequencies_ , _Review of Scientific Instruments_ 17 (1946) 268.
  * [119] P. Carenza, T. Fischer, M. Giannotti, G. Guo, G. Martínez-Pinedo and A. Mirizzi, _Improved axion emissivity from a supernova via nucleon-nucleon bremsstrahlung_ , _JCAP_ 10 (2019) 016 [1906.11844].
  * [120] P. Carenza, B. Fore, M. Giannotti, A. Mirizzi and S. Reddy, _Enhanced Supernova Axion Emission and its Implications_ , _Phys. Rev. Lett._ 126 (2021) 071102 [2010.02943].
  * [121] T. Fischer, P. Carenza, B. Fore, M. Giannotti, A. Mirizzi and S. Reddy, _Observable signatures of enhanced axion emission from protoneutron stars_ , _Phys. Rev. D_ 104 (2021) 103012 [2108.13726].
  * [122] K. Hamaguchi, N. Nagata, K. Yanagi and J. Zheng, _Limit on the Axion Decay Constant from the Cooling Neutron Star in Cassiopeia A_ , _Phys. Rev. D_ 98 (2018) 103015 [1806.07151].
  * [123] F. Capozzi and G. Raffelt, _Axion and neutrino bounds improved with new calibrations of the tip of the red-giant branch using geometric distance determinations_ , _Phys. Rev. D_ 102 (2020) 083007 [2007.03694].
  * [124] M. Giannotti, I.G. Irastorza, J. Redondo, A. Ringwald and K. Saikawa, _Stellar Recipes for Axion Hunters_ , _JCAP_ 10 (2017) 010 [1708.02111].
  * [125] M.M. Miller Bertolami, B.E. Melendez, L.G. Althaus and J. Isern, _Revisiting the axion bounds from the Galactic white dwarf luminosity function_ , _JCAP_ 10 (2014) 069 [1406.7712].
  * [126] LUX collaboration, _First Searches for Axions and Axionlike Particles with the LUX Experiment_ , _Phys. Rev. Lett._ 118 (2017) 261301 [1704.02297].
  * [127] PandaX collaboration, _Limits on Axion Couplings from the First 80 Days of Data of the PandaX-II Experiment_ , _Phys. Rev. Lett._ 119 (2017) 181806 [1707.07921].
  * [128] XENON100 collaboration, _First Axion Results from the XENON100 Experiment_ , _Phys. Rev. D_ 90 (2014) 062009 [1404.1455].
  * [129] S.K. Lamoreaux, K.A. van Bibber, K.W. Lehnert and G. Carosi, _Analysis of single-photon and linear amplifier detectors for microwave cavity dark matter axion searches_ , _Phys. Rev. D_ 88 (2013) 035020.
  * [130] T. Chattopadhyay and K. Siemensmeyer, _Hyperfine induced nuclear polarization in nd2cuo4_ , _Europhysics Letters_ 29 (1995) 579.
  * [131] T. Chatterji, O. Holderer and H. Schneider, _Direct evidence for nuclear spin waves in nd2cuo4 by high-resolution neutron-spin-echo spectroscopy_ , _Journal of Physics: Condensed Matter_ 25 (2013) 476002.
  * [132] D.F.J. Kimball, _Nuclear spin content and constraints on exotic spin-dependent couplings_ , _New Journal of Physics_ 17 (2015) 073008.
  * [133] J. Engel and P. Vogel, _Spin-dependent cross sections of weakly interacting massive particles on nuclei_ , _Phys. Rev. D_ 40 (1989) 3132.
  * [134] B. Hernández, P. Sarriguren, O. Moreno, E. Moya de Guerra, D.N. Kadrev and A.N. Antonov, _Nuclear shape transitions and elastic magnetic electron scattering_ , _Phys. Rev. C_ 103 (2021) 014303.
  * [135] S.G. Nilsson, _Binding states of individual nucleons in strongly deformed nuclei_ , _Dan. Mat. Fys. Medd._ 29 (1955) 1.
  * [136] A. Bohr and B. Mottelson, _Nuclear Structure_ , no. v. 2 in Nuclear Structure, World Scientific (1998).
  * [137] V.V. Flambaum and A.F. Tedesco, _Dependence of nuclear magnetic moments on quark masses and limits on temporal variation of fundamental constants from atomic clock experiments_ , _Phys. Rev. C_ 73 (2006) 055501.
  * [138] S.M. Rezende, _Fundamentals of Magnonics_ , Lecture Notes in Physics, 969, Springer International Publishing, Cham, 1st ed. 2020. ed. (2020).
  * [139] R.M. White, M. Sparks and I. Ortenburger, _Diagonalization of the antiferromagnetic magnon-phonon interaction_ , _Phys. Rev._ 139 (1965) A450.