O#1

#  Multi-News+: Cost-efficient Dataset Cleansing
via LLM-based Data Annotation

Juhwan Choi, Jungmin Yun, Kyohoon Jin  YoungBin Kim
Chung-Ang University, Republic of Korea, Seoul
{gold5230, cocoro357, fhzh123<EMAIL_ADDRESS>

###### Abstract

The quality of the dataset is crucial for ensuring optimal performance and
reliability of downstream task models. However, datasets often contain noisy
data inadvertently included during the construction process. Numerous attempts
have been made to correct this issue through human annotators. However, hiring
and managing human annotators is expensive and time-consuming. As an
alternative, recent studies are exploring the use of large language models
(LLMs) for data annotation.

In this study, we present a case study that extends the application of LLM-
based data annotation to enhance the quality of existing datasets through a
cleansing strategy. Specifically, we leverage approaches such as chain-of-
thought (CoT) and majority voting to imitate human annotation and classify
unrelated documents from the Multi-News dataset, which is widely used for the
multi-document summarization task. Through our proposed cleansing method, we
introduce an enhanced Multi-News+. By employing LLMs for data cleansing, we
demonstrate an efficient and effective approach to improving dataset quality
without relying on expensive human annotation efforts.

Figure 1: Overall framework for cleansing data and composing Multi-News+.

##  1 Introduction

The significance of dataset quality in deep learning applications cannot be
overstated as mislabeled or noisy data can severely degrade performance Song
et al. (2023). Datasets with incorrect labels, noise, or inconsistencies
undermine the consistency and stability of model training. Cleansing these
datasets contributes to enhancing model performance and generalization
capabilities. Hence, ensuring the quality of the dataset by identifying and
eliminating noisy data is imperative. In the realm of natural language
processing, several researchers have attempted to improve the quality of noisy
datasets. For example, ReDocRED Tan et al. (2022) addressed issues such as
false negatives in DocRED Yao et al. (2019), a widely used dataset for
relation extraction. Similarly, annotation inconsistencies were found in the
MultiWOZ dataset Budzianowski et al. (2018) for dialogue state tracking Qian
et al. (2021), leading to efforts to rectify these issues Eric et al. (2020);
Zang et al. (2020); Han et al. (2021); Ye et al. (2022a).

Source 1
---
Starting in 1996, alexa internet has been donating their crawl data to the
internet archive. Flowing in every day, these data are added to the wayback
machine after an embargo period.
Source 2
… For the first time in decades, researchers trying to develop a vaccine for
malaria have discovered a new target they can use to attack this deadly and
common parasite…
Source 3
Focused crawls are collections of frequently-updated webcrawl data from narrow
( as opposed to broad or wide ) web crawls, often focused on a single domain
or subdomain.
Summary
Researchers think they’ve found a promising new potential weapon in the fight
against malaria in a fairly unlikely place: the blood of toddlers. In a paper
published in science today, …
Table 1: Examples of noisy documents in Multi-News dataset. Sources 1 and 3 do
not contribute to the summary. We aim to identify such noisy documents without
a human annotator.

Despite these efforts, relying on human annotators to enhance datasets poses
challenges such as high costs and time constraints. The quality of the
annotation might also be affected by potential variations, such as subjective
bias and the proficiency of the annotator Rashtchian et al. (2010).
Furthermore, cleansing a noisy dataset typically requires a larger budget,
often involving majority voting by multiple annotators or validation by
experts Tan et al. (2022). Given the significance and necessity of enhancing
the quality of existing datasets, these obstacles hinder practical efforts to
cleanse datasets efficiently. Therefore, it is crucial to explore cost-
effective methods that can cleanse the existing dataset, minimizing human
involvement.

In this study, we propose leveraging large language model (LLM)-based
annotation for dataset cleansing. Researchers have explored cost-efficient
alternatives to human annotators by employing LLMs across various tasks Wang
et al. (2021); Ding et al. (2023); He et al. (2023); Bansal and Sharma (2023);
Zhang et al. (2023); Choi et al. (2024). However, the real-world applicability
of LLM-based annotation on existing datasets is still less explored. Building
on these insights, we extend the application of LLM-based annotations to
denoise the existing dataset and improve its quality. Specifically, we conduct
a case study to cleanse the Multi-News Fabbri et al. (2019), a dataset for
multi-document summarization tasks. This dataset consists of news articles
crawled from the internet and is widely used in multi-document summarization
research. However, as shown in Table 1, we identify several issues related to
the noise in the dataset. For instance, the set of documents contained system
messages from platforms such as Twitter, Wayback Machine, or Dow Jones that
are unrelated to the summary and degrade the dataset quality.

To accomplish our purpose, we utilize LLMs to analyze the summary and
associated documents, identifying and excluding any documents that are not
relevant to the summary. Specifically, we employ approaches such as chain-of-
thought (CoT), providing the rationale for decision-making with enhanced
transparency and facilitating human investigation. We further enhance our
cleansing process by incorporating self-consistency considerations, which
mimic the majority voting process used by human annotators Wang et al.
(2023b). Based on our carefully designed framework, we introduce Multi-News+,
an enhanced version of the existing Multi-News dataset, achieved through our
LLM-based cleansing strategy. To the best of our knowledge, this is the first
attempt to exploit LLMs to enhance the quality of real-world datasets. Our
experiments demonstrate the effectiveness of Multi-News+, providing a valuable
resource for future research. We make Multi-News+ and our source code publicly
available for further study.

##  2 Related Work

Dataset quality has been an interest to researchers because of its importance
in ensuring the quality of the model trained with the dataset Budach et al.
(2022). Previous studies found that large amounts of data automatically
crawled from the web may contain noisy documents, and proper filtering
procedures can be an efficient solution against them Xu and Koehn (2017);
Khayrallah and Koehn (2018); Kryściński et al. (2019); Luccioni and Viviano
(2021); Kreutzer et al. (2022). Accordingly, several studies in text
summarization investigated various strategies to filter out noisy data
Matsumaru et al. (2020); Nan et al. (2021); Guo et al. (2022) and released new
datasets with better quality Grusky et al. (2018); Urlana et al. (2022).
However, their strategies are primarily composed of coarse rule-based methods
and less interpretable model output, or costly human investigation has been
applied for constructing new datasets. Furthermore, such strategies have not
been applied to multi-document summarization datasets.

In the meantime, with the advancement of LLMs Zhao et al. (2023), researchers
have explored the usage of LLMs for data annotation, a task that traditionally
relied on human annotators. Initial attempts have revealed the potential
capabilities of models like GPT-3 for data annotation Wang et al. (2021).
These studies indicate that GPT-3 can annotate datasets more efficiently and
cost-effectively than human annotators. This results in enhanced downstream
task performance, with the model trained on the GPT-3 annotated dataset
outperforming the one trained on the human-annotated dataset. Subsequent
studies have further demonstrated the capabilities of GPT-3, showing its
ability to generate labeled data using external knowledge or instructions
about desired labels and domains Ding et al. (2023). Additionally, researchers
have examined the usefulness of newer models like GPT-3.5 and evaluated the
effectiveness of CoT in improving annotation quality He et al. (2023). LLM-
based annotation has also been extended to low-resource languages where hiring
human annotators is challenging Choi et al. (2024).

In this work, we introduce a novel approach to filtering noisy documents from
multi-document summarization dataset by extending cost-efficient LLM-based
annotation beyond traditional data annotation tasks. By leveraging the
capabilities of LLMs, our study facilitates real-world dataset cleansing,
enhancing the quality of existing datasets. This attempt is noteworthy as it
broadens the scope of LLM applications, offering effective solutions for
improving dataset quality and streamlining its cleansing process, minimizing
reliance on human annotations.

##  3 Multi-News+

Figure 2: Histogram comparing the amount of input articles in each dataset.

Model | BART-large-cnn
---|---
Metric | ROUGE-1 | ROUGE-2 | ROUGE-L | BERTScore | BARTScore
Multi-News | 48.64 | 18.86 | 24.11 | 0.6401 | -2.763
Multi-News+ | 49.17 | 19.04 | 24.36 | 0.6418 | -2.698
Model | T5-base
Metric | ROUGE-1 | ROUGE-2 | ROUGE-L | BERTScore | BARTScore
Multi-News | 40.11 | 13.90 | 21.58 | 0.6003 | -2.407
Multi-News+ | 40.45 | 14.17 | 21.84 | 0.6027 | -2.362
|  |  |  |  |

Table 2: Performance comparison of the Multi-News and Multi-News+ datasets on
two models.

The previous Multi-News dataset plays an important role in multi-document
summarization research. It consists of sets of documents and their
corresponding summaries. However, as shown in Table 1 and detailed in Appendix
F and G, the Multi-News dataset contains several noisy and irrelevant articles
that are unrelated to the summary or other documents. This issue arises from
their construction process, which relies on automated crawling from the
Internet Archive.

To solve this issue and cleanse the dataset, we defined our problem as a
classification task determining whether each document is relevant to the
summary. To this end, we designed the prompt for the model as shown in
Appendix I. We integrated CoT to enhance the model’s performance by evaluating
the relevance of each document to the summary. Thus, a rationale for the
decision can be made available, which marks the difference between LLM-based
and human annotations. While traditional human annotation through
crowdsourcing platforms like Amazon Mechanical Turk usually produces
annotation results without underlying reasons due to additional costs, LLM-
based annotators can easily offer explanations through CoT. These rationales
can assist human managers in reviewing results and rectifying erroneous
decisions.

Furthermore, we imitated the conventional dataset cleansing procedure which
typically involves multiple human annotators and their collective judgments,
primarily through majority voting. Similarly to the majority voting process
used by human annotators, we applied this approach to the LLM-based
annotators. In particular, we generated five individual LLM agents to read the
summary and documents and determine if the document is relevant to the
summary. This strategy based on self-consistency can boost the quality of
annotations, by rectifying potential errors made by individual agents Wang et
al. (2023b). Figure 1 presents the summary of the overall process.

Based on the proposed method, we utilized five LLM agents to individually
annotate 56,216 sets of summaries and documents from the Multi-News dataset.
Specifically, we employed the GPT-3.5-turbo-0125 model111GPT-3.5-turbo-0125
charges 0.0005$ for the input of 1,000 tokens, and 0.0015$ for the generation
of 1,000 tokens., the most recent model at the time of this study. With a
prompt designed for a 3-shot CoT, approximately 3,500 tokens were required to
annotate the input summaries and articles, along with around 100 tokens for
generating reasoning processes and annotation results. The cost per annotation
sample amounted to approximately 0.01$ (0.002$ per agent), resulting in a
total cost of approximately 550$ to annotate the entire Multi-News dataset.

After annotation, we found that 27,052 of the 153,091 articles can be
considered noisy documents and do not contribute to the summarization.
Subsequently, we constructed Multi-News+ by removing these noisy documents
from Multi-News while preserving the train/valid/test split. Figure 2 presents
the comparison of the Multi-News and Multi-News+ datasets in terms of the
number of documents per set. More than 15% of the documents in Multi-News are
irrelevant, diminishing the dataset’s quality and degrading the model’s
performance. Furthermore, 379 sets have no relevant source articles, as shown
in Appendix G. In contrast, by deleting noisy documents, Multi-News+
demonstrates enhanced quality.

##  4 Experiment

###  4.1 Experimental Design

To validate the efficacy of data cleansing and the development of Multi-News+
in filtering out noisy documents and improving the performance of downstream
task models, we measured the multi-document summarization performance of
models trained on each dataset, similar to previous study Guo et al. (2022).
Enhanced model performance indicates superior dataset quality Ye et al.
(2022b); Choi et al. (2024). We fine-tuned two different models, BART Lewis et
al. (2020) and T5 Raffel et al. (2020) on Multi-News and Multi-News+.
Performance evaluation metrics included the following metrics: ROUGE Lin
(2004), BERTScore Zhang et al. (2020), and BARTScore Yuan et al. (2021). For a
fair comparison, we used the test set of Multi-News+ for each model and
reported the average performance across three random seeds.

###  4.2 Result

The results in Table 2 demonstrate the superiority of the Multi-News+ dataset
in enhancing the performance of summarization models compared to the original
Multi-News dataset. Across various metrics, models trained on Multi-News+
consistently outperform those trained on Multi-News, indicating better
summarization quality with the refined dataset. This highlights the
effectiveness of dataset cleansing in removing noisy and irrelevant documents,
thereby enhancing the overall performance of summarization models.
Additionally, we performed a human evaluation on the output of 379 sets that
are classified as having no relevant source articles and found that 356 sets
are correctly classified. We provide an example of error analysis in Appendix
H.

##  5 Conclusion

In this study, we suggest deploying cost-efficient LLM-based data annotation
to cleanse real-world datasets by identifying and excluding irrelevant and
noisy data. We conducted a case study using this strategy to cleanse the
Multi-News dataset and proposed the improved Multi-News+ dataset. Our case
study revealed that Multi-News+ provides superior data quality compared to the
original Multi-News dataset. Additionally, we have made Multi-News+ publicly
available, thereby supporting further research in the field of multi-document
summarization.

Our work paves the road to extending our data cleansing strategy to other
datasets, broadening the scope of utilizing LLMs. This extension would enhance
the quality of existing datasets across various domains without the need to
construct new datasets from scratch. As such, our approach not only
contributes to the advancement of multi-document summarization research but
also offers a cost-efficient solution for enhancing dataset quality. We are
committed to extending our LLM-based method to other datasets, further
solidifying its applicability to other tasks.

## Limitations

We acknowledge several limitations regarding our proposed method. First, our
method is primarily limited by the possibility of wrong classification even
with majority voting and CoT. In the future, we may adopt various LLMs as
agents and apply weighted majority voting according to their performance to
alleviate this issue.

Secondly, the nature of the Multi-News dataset might exhibit a real-world case
of automatic collection of documents from the web that are not always relevant
to the summary. In other words, the inclusion of noisy documents might
demonstrate the characteristics of real-world automatic crawling. For
instance, the model trained on the Multi-News dataset may be more suitable for
a real-time system that automatically crawls data from the web and summarizes
them. However, we believe such a possibility can be dealt with through the
reciprocal usage of our Multi-News+ and previous Multi-News dataset. For
instance, one could utilize a previous Multi-News dataset when the trained
model is expected to consistently deal with noisy documents for inference and
there are no pre-defined strategies for filtering out these noisy documents at
inference time. Otherwise, for cases where the model is expected to only
handle clean documents, it will be more beneficial to utilize our proposed
Multi-News+ dataset for training the model.

## Ethics Statement

As we are exploiting LLMs for classifying irrelevant documents rather than
text generation, the ethical concern with our method is smaller than that of
studies that utilize LLMs to generate texts. Nonetheless, recent studies
suggest that the CoT technique may induce ethical bias in LLM Shaikh et al.
(2023). In future work, we plan to investigate this phenomenon’s appearance
using our method.

## References

  * Bansal and Sharma (2023) Parikshit Bansal and Amit Sharma. 2023.  Large language models as annotators: Enhancing generalization of nlp models at minimal cost.  _arXiv preprint arXiv:2306.15766_.
  * Budach et al. (2022) Lukas Budach, Moritz Feuerpfeil, Nina Ihde, Andrea Nathansen, Nele Noack, Hendrik Patzlaff, Felix Naumann, and Hazar Harmouch. 2022.  The effects of data quality on machine learning performance.  _arXiv preprint arXiv:2207.14529_.
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018.  Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5016–5026.
  * Choi et al. (2024) Juhwan Choi, Eunju Lee, Kyohoon Jin, and YoungBin Kim. 2024.  GPTs are multilingual annotators for sequence generation tasks.  In _Findings of the Association for Computational Linguistics: EACL 2024_ , pages 17–40.
  * Ding et al. (2023) Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023.  Is GPT-3 a good data annotator? In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 11173–11195.
  * Eric et al. (2020) Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani-Tur. 2020.  Multiwoz 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 422–428.
  * Fabbri et al. (2019) Alexander Richard Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019.  Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1074–1084.
  * Grusky et al. (2018) Max Grusky, Mor Naaman, and Yoav Artzi. 2018.  Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 708–719.
  * Guo et al. (2022) Yanzhu Guo, Chloé Clavel, Moussa Kamal Eddine, and Michalis Vazirgiannis. 2022.  Questioning the validity of summarization datasets and improving their factual consistency.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5716–5727.
  * Han et al. (2021) Ting Han, Ximing Liu, Ryuichi Takanabu, Yixin Lian, Chongxuan Huang, Dazhen Wan, Wei Peng, and Minlie Huang. 2021.  Multiwoz 2.3: A multi-domain task-oriented dialogue dataset enhanced with annotation corrections and co-reference annotation.  In _Natural Language Processing and Chinese Computing: 10th CCF International Conference_ , pages 206–218. Springer.
  * He et al. (2023) Xingwei He, Zhenghao Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. 2023.  Annollm: Making large language models to be better crowdsourced annotators.  _arXiv preprint arXiv:2303.16854_.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Khayrallah and Koehn (2018) Huda Khayrallah and Philipp Koehn. 2018.  On the impact of various types of noise on neural machine translation.  In _Proceedings of the 2nd Workshop on Neural Machine Translation and Generation_ , pages 74–83.
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations_.
  * Kreutzer et al. (2022) Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. 2022.  Quality at a glance: An audit of web-crawled multilingual datasets.  _Transactions of the Association for Computational Linguistics_ , 10:50–72.
  * Kryściński et al. (2019) Wojciech Kryściński, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Neural text summarization: A critical evaluation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 540–551.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81.
  * Luccioni and Viviano (2021) Alexandra Luccioni and Joseph Viviano. 2021.  What’s in the box? an analysis of undesirable content in the common crawl corpus.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 182–189.
  * Matsumaru et al. (2020) Kazuki Matsumaru, Sho Takase, and Naoaki Okazaki. 2020.  Improving truthfulness of headline generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1335–1346.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gu̇lçehre, and Bing Xiang. 2016.  Abstractive text summarization using sequence-to-sequence rnns and beyond.  In _Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 280–290.
  * Nan et al. (2021) Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen Mckeown, and Bing Xiang. 2021.  Entity-level factual consistency of abstractive text summarization.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2727–2733.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32.
  * Qian et al. (2021) Kun Qian, Ahmad Beirami, Zhouhan Lin, Ankita De, Alborz Geramifard, Zhou Yu, and Chinnadhurai Sankar. 2021.  Annotation inconsistency and entity bias in multiwoz.  In _Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 326–337.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Rashtchian et al. (2010) Cyrus Rashtchian, Peter Young, Micah Hodosh, and Julia Hockenmaier. 2010.  Collecting image annotations using amazon’s mechanical turk.  In _Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon’s Mechanical Turk_ , pages 139–147.
  * Shaikh et al. (2023) Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. 2023.  On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4454–4470.
  * Song et al. (2023) Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. 2023.  Learning from noisy labels with deep neural networks: A survey.  _IEEE Transactions on Neural Networks and Learning Systems_ , 34(11):8135–8153.
  * Tan et al. (2022) Qingyu Tan, Lu Xu, Lidong Bing, Hwee Tou Ng, and Sharifah Mahani Aljunied. 2022.  Revisiting docred-addressing the false negative problem in relation extraction.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8472–8487.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Urlana et al. (2022) Ashok Urlana, Nirmal Surange, Pavan Baswani, Priyanka Ravva, and Manish Shrivastava. 2022.  Tesum: Human-generated abstractive summarization corpus for telugu.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 5712–5722.
  * Wang et al. (2023a) Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2023a.  A survey on large language model based autonomous agents.  _arXiv preprint arXiv:2308.11432_.
  * Wang et al. (2021) Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021.  Want to reduce labeling cost? gpt-3 can help.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4195–4205.
  * Wang et al. (2023b) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023b.  Self-consistency improves chain of thought reasoning in language models.  In _The Eleventh International Conference on Learning Representations_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45.
  * Xu and Koehn (2017) Hainan Xu and Philipp Koehn. 2017.  Zipporah: a fast and scalable data cleaning system for noisy web-crawled parallel corpora.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2945–2950.
  * Yao et al. (2019) Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019.  Docred: A large-scale document-level relation extraction dataset.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 764–777.
  * Ye et al. (2022a) Fanghua Ye, Jarana Manotumruksa, and Emine Yilmaz. 2022a.  Multiwoz 2.4: A multi-domain task-oriented dialogue dataset with essential annotation corrections to improve state tracking evaluation.  In _Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 351–360.
  * Ye et al. (2022b) Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022b.  Zerogen: Efficient zero-shot learning via dataset generation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11653–11669.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021.  Bartscore: Evaluating generated text as text generation.  _Advances in Neural Information Processing Systems_ , 34:27263–27277.
  * Zang et al. (2020) Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020.  Multiwoz 2.2: A dialogue dataset with additional annotation corrections and state tracking baselines.  In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_ , pages 109–117.
  * Zhang et al. (2023) Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. 2023.  LLMaAA: Making large language models as active annotators.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 13088–13103.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_.

##  Appendix A Dataset Statistics

Multi-News+ keeps the train/valid/test split of Multi-News, which is 80%, 10%,
and 10%. Table 3 displays the number of articles per each split.

| Multi-News | Multi-News+ | % of modification
---|---|---|---
| Sets | Articles | Sets | Articles | Sets | Articles
Train | 44,972 | 125,417 | 44,668 | 102,057 | 0.7% | 18.6%
Validation | 5,622 | 15,367 | 5,585 | 12,509 | 0.7% | 18.6%
Test | 5,622 | 15,505 | 5,584 | 12,703 | 0.7% | 18.1%
|  |  |  |  |  |

Table 3: Number of sets and articles per each split.

##  Appendix B Construction Process of Multi-News

Figure 3: A screenshot of a webpage that is relevant to the article in
Appendix G. Multi-News includes the text in the red box instead of the desired
content in the blue box.

In this section, we briefly explain the construction process of the Multi-News
dataset. Multi-News is based on data from newser.com222https://newser.com that
offers human-written summaries of news articles. Each summary is written by
professional human editors and involves several outlinks to the original
articles and relevant websites. Multi-News collected this human-written
summary and documents from its outlinks, which behave as source documents for
summarization. Notably, the authors of Multi-News archived every article
leveraging Wayback Machine333https://web.archive.org, a system that supports
archiving of the circumstances of a given website, to ensure the
reproducibility and support future investigation. Contents of each document
have been accessed and crawled from these Wayback-archived links.

However, this affected problems regarding the quality of the dataset. As shown
in examples of noisy documents in Appendix F, several noisy documents consist
of a message from Wayback Machine. Moreover, the failure to crawl the content
of the webpage caused other problems. We investigated the case shown in
Appendix G and found that it is a result of the crawling of the wrong part of
the website. Figure 3 clearly showcases this phenomenon where the content in
the red box is crawled instead of the content in the blue box, which is
desired. Even though the content in the blue box is different for each
article, the system wrongly crawled the shared red box, which resulted in five
noisy documents that share the same content and do not contribute to the
summary.

From the example above, we revealed the presence of the wrongly crawled
documents, that affect the quality of the dataset. We believe such phenomena
would be alleviated with the advancement of LLM-based autonomous agents Wang
et al. (2023a), as they could visit the website and only crawl the text
relevant to the summary. Even though we leave this as future work, this
research direction should be prompted.

##  Appendix C Implementation Details

We utilized PyTorch Paszke et al. (2019) and Huggingface Transformers Wolf et
al. (2020) to implement and evaluate the model. Specifically, we employed
facebook/bart-large-cnn444Note that this model is already fine-tuned with the
CNN/DM dataset Nallapati et al. (2016), a single-document summarization
dataset. and google-t5/t5-base, with 406M and 220M parameters, respectively,
for BART and T5. Each model was trained using Adam Kingma and Ba (2015) with a
learning rate of 2e-5 over 3 epochs. We used a batch size of 4 and implemented
a gradient accumulation step of 4, resulting in a practical batch size of 16.
For evaluation, we utilized bert-base-uncased and facebook/bart-large-cnn for
BERTScore and BARTScore, respectively. We reported BERTScore-F1 in Table 2.
ROUGE scores were measured using the rouge-
score555https://pypi.org/project/rouge-score/ library, with the F1 score of
each metric. The training was conducted on a single NVIDIA A100 40GB GPU. We
provide the source code and dataset to the
public.666https://anonymous.4open.science/r/MULTINEWS_PLUS/

For the human evaluation, we recruited three volunteers and individually asked
them to determine whether the decision of the model was correct or not given
the summary, original articles, and rationale of the model. We defined the
model made an incorrect decision when at least one human evaluator flagged the
output as an incorrect classification.

Model | Mistral-7B-Instruct-v0.2
---|---
Metric | BERTScore | BARTScore
No Noisy Example | 0.6004 | -2.704
One Noisy Example | 0.5976 | -2.721
Two Noisy Examples | 0.5954 | -2.738
Model | Llama-2-7b-chat-hf
Metric | BERTScore | BARTScore
No Noisy Example | 0.6038 | -2.507
One Noisy Example | 0.6022 | -2.521
Two Noisy Examples | 0.6016 | -2.539
|  |

Table 4: Performance of LLM-based summarization of Multi-News with different
amounts of noisy examples. We only report two model-based metrics as the
human-generated reference summary has a different form compared to the LLM-
generated summary.

##  Appendix D Additional Experiment with Large Language Models

This section introduces our additional experiment that investigates the
influence of noisy examples for LLMs in a few-shot learning scheme. For this
purpose, we used 7B-sized, instruction-tuned Llama2 Touvron et al. (2023) and
Mistral Jiang et al. (2023). Specifically, we used meta-llama/Llama-2-7b-chat-
hf and mistralai/Mistral-7B-Instruct-v0.2 from Transformers Wolf et al.
(2020). In this experiment, we prompted the model to summarize the documents
in the test set of Multi-News with two-shot examples selected from the
training set of Multi-News. Additionally, we differentiated the number of
noisy documents in the examples given as the prompt. Table 4 presents the
experimental result. The result demonstrates that the inclusion of the noise
in the example degrades the quality of the summary generated by the LLM. This
suggests the significance of the exclusion and filtering of the noise for
LLMs, which underscores the necessity of dataset cleansing presented in this
paper.

##  Appendix E Analysis of Multi-News

| Multi-News
---|---
Dataset Size | 56,216
Source Article Size | 156,289
Avg Words in Source | 433.62
Avg Sentences in Source | 23.42
Avg Words in Summary | 228.69
Avg Sentences in Summary | 11.52
Empty Summary | 0
Duplicated Summary | 0
Prefixes Summary | 0
Empty Source | 570
Duplicated Source | 544
Source < 4 Sentences | 45
Source < 40 Words | 7
Summary < 10 Words | 0
Compression < 50% | 31,994
Compression > 80% | 390
Abstractivity < 10 | 496
Abstractivity > 80 | 126
Avg Abstractivity | 41.42
Avg Compression | 46.19%
|

Table 5: The result of analysis of Multi-News dataset with rule-based
filtering methods Urlana et al. (2022). We concatenated every source document
to measure their average word and sentence length.

Following the previous study of TeSum Urlana et al. (2022), we apply filtering
strategies and analyze the characteristics of Multi-News with these
strategies. Table 5 exhibits the result of the analysis. First, we found that
0.7% of total source documents can be considered noisy documents as it is
empty or duplicated from other source documents within the same set. Second,
we found previous rule-based filtering methods are not very effective
standards for the Multi-News dataset. For instance, there were no sets that
had empty summaries, summaries that were duplicated with other summaries, or
summaries that repeated the first few sentences of source documents. The only
exception is Compression < 50%, which identified more than half of the
dataset. However, it should be noted that Multi-News is a multi-document
summarization dataset, which is different from datasets for previous studies.
For instance, average compression is significantly lower than other single-
document summarization datasets reported in the previous study Urlana et al.
(2022), as multiple source documents in Multi-News involve more information
compared to the source document of single-document summarization datasets. In
conclusion, this analysis demonstrates that previous filtering strategies are
less practical for multi-document summarization datasets such as Multi-News
and enlightens the necessity of novel approaches for these datasets.

##  Appendix F Examples of Noisy Documents

This section demonstrates several examples of noisy documents observed in the
Multi-News dataset not related to the summary. Please refer to the released
dataset file for details.

[thickness=1pt]

  * •

Tweet with a location you can add location information to your tweets, such as
your city or precise location, from the web and via third-party applications.
You always have the option to delete your tweet location history. Learn more

  * •

Focused crawls are collections of frequently-updated webcrawl data from narrow
( as opposed to broad or wide ) web crawls, often focused on a single domain
or subdomain.

  * •

Dow jones reprints: this copy is for your personal, non-commercial use only.
To order presentation-ready copies for distribution to your colleagues,
clients or customers, use the order reprints tool at the bottom of any article
or visit www.djreprints.com

  * •

This crawl of online resources of the 115th us congress was performed on
behalf of the united states national archives & records

  * •

The seed for this crawl was a list of every host in the wayback machine this
crawl was run at a level 1 ( urls including their embeds, plus the urls of all
outbound links including their embeds ) the warc files associated with this
crawl are not currently available to the general public.

  * •

These crawls are part of an effort to archive pages as they are created and
archive the pages that they refer to. That way, as the pages that are
referenced are changed or taken from the web, a link to the version that was
live when the page was written will be preserved.then the internet archive
hopes that references to these archived pages will be put in place of a link
that would be otherwise be broken, or

  * •

Please enable cookies on your web browser in order to continue. The new
european data protection law requires us to inform you of the following before
you use our website: we use cookies and other technologies to customize your
experience, perform analytics and deliver personalized advertising on our
sites, apps and newsletters and across the internet based on your interests.
By clicking “i agree” below, you consent to the use by us and our third-party
partners of cookies and data gathered from your use of our platforms. See our
privacy policy and third party partners to learn more about the use of data
and your rights. You also agree to our terms of service.

  * •

Thank you for reading. Please purchase a subscription to continue reading. A
subscription is required to continue reading. Thank you for reading 5 free
articles. You can come back at the end of your 30-day period for another 5
free articles, or you can purchase a subscription and continue to enjoy
valuable local news and information. If you are a current 7-day subscriber you
are granted an all-access pass to the website and digital newspaper replica.
Please click sign up to subscribe, or login if you are already a member. Thank
you for reading 5 free articles. You can come back at the end of your 30-day
period for another 5 free articles, or you can purchase a subscription and
continue to enjoy valuable local news and information. If you are a current
7-day subscriber you are granted an all-access pass to the website and digital
newspaper replica. Please click below to get started.

  * •

Add a location to your tweets when you tweet with a location, twitter stores
that location. You can switch location on/off before each tweet and always
have the option to delete your location history. Learn more

[thickness=1pt]

##  Appendix G Extreme Cases of Noisy Documents

In addition to examples of noisy documents, we discovered the following
extreme case of noisy data in the Multi-News dataset. In this example, five
documents have the same content but offer no information on the summary. Thus,
it cannot generate a reasonable summary based on the given documents. We
witnessed 379 similar cases during the dataset cleansing process, as reported
in Figure 2. While they were excluded from training and testing, we included
them in the dataset file for future investigation.

[thickness=1pt]

Summary
Note to tweeting politicians: watch what you post, because politwoops will
remember it forever. The transparency-minded website is safeguarding
politicians’deleted tweets, enabling the rest of us to giggle or ponder over
them at our leisure, the atlantic reports. The site’s current 6-month stash
includes a few doozey deletions, including john mccain mocking vladimir
putin’s tears and rep. Jeff miller posting a link to a poll that asked, " was
obama born in the united states? " a few deletions are more odd than obvious,
begging us to ask what politicians were thinking. Why, for example, did rep.
Tom graves remove a tweet about going out one night with his wife? or rep.
Kathy hochul delete one about her visit to a cancer institute? perhaps rep.
Stephen fincher’s tweet comparing the bachelor to the hunger games is a more
obvious case, but the online avenues of a politician’s mind can be dimly lit
indeed.

Document 1
An archive of the public statements deleted by u.s. Politicians. Explore the
tweets they would prefer you couldn’t see. If you aren’t an elected official
or running for office and feel your account is being tracked by mistake then
please contact us.

Document 2
An archive of the public statements deleted by u.s. Politicians. Explore the
tweets they would prefer you couldn’t see. If you aren’t an elected official
or running for office and feel your account is being tracked by mistake then
please contact us.

Document 3
An archive of the public statements deleted by u.s. Politicians. Explore the
tweets they would prefer you couldn’t see. If you aren’t an elected official
or running for office and feel your account is being tracked by mistake then
please contact us.

Document 4
An archive of the public statements deleted by u.s. Politicians. Explore the
tweets they would prefer you couldn’t see. If you aren’t an elected official
or running for office and feel your account is being tracked by mistake then
please contact us.

Document 5
An archive of the public statements deleted by u.s. Politicians. Explore the
tweets they would prefer you couldn’t see. If you aren’t an elected official
or running for office and feel your account is being tracked by mistake then
please contact us.

[thickness=1pt]

##  Appendix H Error Analysis

Following the form of the previous study Choi et al. (2024), we provide an
error analysis to provide a more balanced view of the behavior and limitations
of our proposed method. In the first example, we can observe that while
Document 1 can be regarded as irrelevant to the summary except that there is a
mention of fusion tv, Document 2 contains information about Mike Tyson and his
new TV documentary series. However, the model predicted both documents are
irrelevant to the summary, primarily because the model concentrated on the
mention of the “world team tennis exhibition” from Document 2. From this
insight, we hypothesize GPT-3.5 suffers from a mixture of irrelevant and
relevant information in one document.

[thickness=1pt]

Summary
Over his career, former heavyweight champion mike tyson recorded 50 wins and
six losses. But he recently notched another big loss in latin america — this
time as a coach of a bird, reports the ap. Tyson traveled to suriname as part
of the new fusion tv documentary series outpost, and was soundly beaten when
he entered a bird in a songbird contest, a cherished local tradition. Cameras
captured iron mike as he learned about the contest, located a bird to enter —
he dubbed the tiny guy " little mike " — but then suffered a tko when a
competing champion cheeped and peeped more than his bird did in the same
15-minute period. " little mike let us down, man. I was in his corner, though,
" said tyson. " it was just amazing meeting the people, meeting the culture —
i had a great time. " the series, kicking off on sunday with tyson’s episode,
mixes travel adventure, history, and journalism to shine a light on global
stories. The first season focuses on latin america and includes as hosts the
late show with stephen colbert bandleader jon batiste, brain games star jason
silva, and transgender model carmen carrera. Spanish versions air on unimas.
Tyson was lured onto the show by the chance to visit a country he’d never
heard of and his love of birds. The former boxer has loved pigeons and kept
them since he was a kid in brooklyn. ( sunday’s show recorded the moment tyson
lovingly released his bird in suriname. ) " my wife always says the reason i
keep my pigeons is they connect me to my childhood, " tyson said. " once it’s
in your blood, it never leaves. It’s just who you are. "

Document 1
Starting in 1996, alexa internet has been donating their crawl data to the
internet archive. Flowing in every day, these data are added to the wayback
machine after an embargo period. [Abbreviated duplicated text] Outpost shows
you the world like you’ve never seen it. The series lives at the intersection
of investigative journalism and adventure travel, bringing you a local
perspective on faraway places and inviting you to explore. The series
premieres march 26 @ 8 and 11 pm on fusion tv. In the first episode,
transgender model carmen carrera travels to brazil, a place where rates of
violence against lgbt people are some of the highest in the world, to find out
what’s happening, what life is like for young transgendered people in brazil,
and what the future might hold. Gabriel leigh takes us to el alto, bolivia,
where some of the craziest architecture on earth is taking shape as part of a
surge in indigenous purchasing power.

Document 2
[Abbreviated duplicated text] file - in this monday, oct. 10, 2016, file
photo, mike tyson attends a world team tennis exhibition to benefit the elton
john aids foundation in las vegas. Tyson traveled to suriname as part of the
new fusion tv documentary series "outpost " and was soundly beaten when he
entered a bird in a songbird… ( associated press ) [Abbreviated duplicated
text] new york ( ap ) — over his career, former heavyweight champion mike
tyson recorded 50 wins and six losses. But he recently notched another big
loss in latin america — this time as a coach of a bird. Tyson traveled to
suriname as part of the new fusion tv documentary series " outpost " and was
soundly beaten when he

[thickness=1pt]

This second example also showcases the characteristics of GPT-3.5 model we
used. In this example, it is obvious that Document 2 is less relevant to the
summary, which is mainly about the relationship between Gwyneth Paltrow and
Chris Martin. However, while it is not the main content of the document as
well as Document 2, Document 1 contains a sentence that mentions the
relationship between the two (“her amicable split from husband chris martin of
coldplay”). Nonetheless, the model predicted Document 1 is also irrelevant to
the summary, implying the model is stringent to the partial contribution of
the document to the summary. However, it is important to note that we
categorized these instances as errors based on rigorous human evaluation, and
such errors constituted fewer than 10% of the total classifications, where a
single flag by multiple human evaluators was sufficient to deem it an error.
We are planning to manually revise these errors in the released version of
Multi-News+.

[thickness=1pt]

Summary
Gwyneth paltrow continues to paint the sunniest of pictures of her post-
conscious-uncoupling life with chris martin, but the description she gives
glamour in a new interview may be the most interesting one so far. " we’re
still very much a family, even though we don’t have a romantic relationship.
He’s like my brother, " she says, explaining that the two of them and their
two kids still spend quite a bit of time together, even staying in one
another’s houses and spending holidays together ( not to mention collaborating
on songs together ). " the ideal is to stay married. But if you can’t stay
married, wouldn’t the ideal be that you could still be a family and you could
put aside your own stuff long enough to explore — what is this new family and
who am i in it? " paltrow muses. " and chris is a great ex-husband ’ cause
he’s a very, very willing partner in how to do that. " she adds that, though
she’s " very independent, " she does see the value in having a husband, and
though she’s not quite divorced yet, she could perhaps see herself getting
married again someday. ( click to see what she has to say about her other
famous exes. )

Document 1
Gwyneth paltrow is in a state of deep focus. The new goop office is under
construction — "it’s like a dust bowl, " she says with a laugh — so today
she’s helming her company from the kitchen island of her los angeles home.
Fitting, considering it was at her kitchen table ( then in london ) that
paltrow, 43, started goop as a newsletter to friends nearly eight years ago.
Since then, she has built goop into a global brand: it has produced sought-
after collaborations with valentino and stella mccartney; opened pop-up shops;
and brought terms like conscious uncoupling and vaginal steaming to the masses
( the first a description of her amicable split from husband chris martin of
coldplay; the second, a way to cleanse one’s uterus — don’t try it at home ).
Her presence has also unwittingly exposed a dirty little secret: as fans, we
provide actresses with wealth and fame, only to scoff when they actually lead
that rich and famous lifestyle publicly. We want these stars to be "just like
us. " but paltrow’s life simply isn’t. She won’t pretend that she shops at the
dollar store for beauty products or feeds her kids, apple, 11, and moses, 9, a
steady diet of fast food;

Document 2
Gwyneth paltrow was definitely in the mood to share during her appearance on
howard stern’s siriusxm radio show on wednesday…. Especially when it came to
her a-list exes. In the hour-long chat, stern of course wanted to know all
about paltrow’s ex-fiance brad pitt, who the shakespeare in love star was
engaged to when she was 24 years old. The beautiful blondes eventually called
it quits in 1997 after three years together. Getty images " i didn’t think
about it at the time, but i ’ m sure it did help with my career, " the now
42-year-old actress admits about the start of all the paparazzi attention when
the two got together on the set of seven. " i definitely fell in love with
him. He was so gorgeous, and sweet – i mean, he was brad pitt, you know? "
video: a history of gwyneth’s former flames her parents, the late bruce
paltrow and actress blythe danner, also 100 percent approved. " my father was
so devastated when we broke up, " she dishes. " my father loved him like he
was his son. " in hindsight, she blames the demise of their relationship on
her youth. " i was such a kid, i was 22 when we met, " she explains.

[thickness=1pt]

##  Appendix I Prompt

This section describes the prompt used for dataset cleansing. We truncated
several parts to save space and only displayed a 1-shot CoT example. Please
refer to the released source code for the exact prompt.

[thickness=1pt]
System
You are a helpful assistant. Your job is to identify the document that is not
relevant to the summary. The relevance is determined by the mention of the key
points in the summary. For example, the alert message from the twitter, the
location information, and the unrelated information are considered irrelevant.
Please response with the resaoning process and the result as document number
or ‘None’. Do not end the response with a period. If multiple documents are
irrelevant to the summary, please resonse in the form of ‘Document 1|Document
3’

User
Given the summary and documents, please identify the document that is not
relevant to the summary such as advertisements. If every document is relevant
to the summary, output ‘None’. Input document will be given in next dialogue.

User
[Summary]
# The content of summary
[Document 1]
# The content of document 1
[Document 2]
# The content of document 2
[Document 3]
# The content of document 3
[Document 4]
# The content of document 4

Assistant
The summary discusses Tucker Carlson’s controversial tweet regarding Sarah
Palin and the subsequent backlash he faced from conservative women. Document 1
is related to Tucker Carlson’s controversial tweet. Document 2 is unrelated as
it contains information about adding location information to tweets and does
not contribute to the discussion about Tucker Carlson’s tweet and the reaction
it received. Document 3 is a tweet message that calls Tucker Carlson a
misogynist, related to the summary. Document 4 is related to the summary as it
deals with Tucker Carlson’s movement. Therefore, the irrelevant document is:
Document 2

[thickness=1pt]


# Spin wave dispersion of ultra-low damping hematite
($\alpha\text{-Fe}_{2}\text{O}_{3}$) at GHz frequencies

Mohammad Hamdi<EMAIL_ADDRESS>
<EMAIL_ADDRESS>École Polytechnique Fédérale de Lausanne (EPFL),
Institute of Materials, Laboratory of Nanoscale Magnetic Materials and
Magnonics, CH-1015 Lausanne, Switzerland  Ferdinand Posva  École Polytechnique
Fédérale de Lausanne (EPFL), Institute of Materials, Laboratory of Nanoscale
Magnetic Materials and Magnonics, CH-1015 Lausanne, Switzerland  Dirk Grundler
<EMAIL_ADDRESS>École Polytechnique Fédérale de Lausanne (EPFL),
Institute of Materials, Laboratory of Nanoscale Magnetic Materials and
Magnonics, CH-1015 Lausanne, Switzerland  École Polytechnique Fédérale de
Lausanne (EPFL), Institute of Electrical and Micro Engineering, CH-1015
Lausanne, Switzerland

###### Abstract

Low magnetic damping and high group velocity of spin waves (SWs) or magnons
are two crucial parameters for functional magnonic devices. Magnonics research
on signal processing and wave-based computation at GHz frequencies focussed on
the artificial ferrimagnetic garnet Y3Fe5O12 (YIG) so far. We report on spin-
wave spectroscopy studies performed on the natural mineral hematite
($\alpha\text{-Fe}_{2}\text{O}_{3}$) which is a canted antiferromagnet. By
means of broadband GHz spectroscopy and inelastic light scattering, we
determine a damping coefficient of $1.1\times 10^{-5}$ and magnon group
velocities of a few 10 km/s, respectively, at room temperature. Covering a
large regime of wave vectors up to $k\approx 24~{}{\rm rad}/\mu$m, we find the
exchange stiffness length to be relatively short and only about 1 Å. In a
small magnetic field of 30 mT, the decay length of SWs is estimated to be 1.1
cm similar to the best YIG. Still, inelastic light scattering provides
surprisingly broad and partly asymmetric resonance peaks. Their characteristic
shape is induced by the large group velocities, low damping and distribution
of incident angles inside the laser beam. Our results promote hematite as an
alternative and sustainable basis for magnonic devices with fast speeds and
low losses based on a stable natural mineral.

Introduction.— Spin waves (magnons) are collective spin excitations in
magnetically ordered materials. They exhibit promising functionalities for
information transmission and processing at GHz frequencies [1, 2, 3]. To
realize energy efficient magnonic circuits [1, 2, 4, 5, 6] isotropic spin wave
(SW) dispersion relations, high group velocities, and low magnetic damping are
essential. Until today, the artificial garnet Y3Fe5O12 (YIG) [7] played a key
role for the exploration of magnonics functionalities [8]. Already in 1961, M.
Sparks et al. coined the term that YIG was to ferromagnetic resonance research
what the fruit fly was to genetics research [9]. This was particularly true
for high-quality YIG grown by liquid phase epitaxy on the wafer scale [8, 10].
However, in a ferrimagnetic material like YIG, magnon bands in the regime of
small wave vectors, $k$, and low GHz frequencies are inherently anisotropic
due to the dipolar interaction between spins. To overcome this, a lot of
effort has been put into the development of microwave-to-magnon transducers
which allow for the excitation of exchange dominated SWs with isotropic
properties at high frequencies [11, 12, 13]. The transducers suffer however
from typically a narrow bandwidth or require an applied magnetic field in
contrast to conventional transmission lines and coplanar waveguides (CPWs).
In antiferromagnetic (AFM) materials, exchange interaction dominates the
dispersion relation already at small wave vectors $k$. The dipolar
interactions are virtually absent due to net zero magnetization. Still, SWs
can propagate with high group velocities. Values similar to thick YIG [8, 14]
and as high as 30 km/s have been reported [15, 16, 17, 18]. However, the
challenge with most AFMs is their net zero magnetization and sub-THz
frequencies which make on-chip integration hard due to lack of efficient CPWs
and THz sources (THz gap) [19, 20]. Recently, the natural mineral and canted
antiferromagnet hematite ($\alpha\text{-Fe}_{2}\text{O}_{3}$) [21] gained
particular attention for magnonics [22, 23] after the observation of long
distance spin transport [24, 25] and enhanced spin pumping [26]. It is known
that, due to extremely low anisotropy in the basal plane [21, 27, 28], in the
canted phase [Fig. 1(a) and (b)] one branch of the magnon modes resides at
around 10 GHz at small $k$.

Figure 1: Hexagonal unit cell of the crystal structure of hematite from (a)
the side and (b) top view. The cyan spheres and red arrows indicate the Fe
atoms and the spins associated with them, respectively (oxygen atoms are not
shown). We depict the canted antiferromagnetic state above the Morin
temperature for which the sublattice spins lie in the $c$-plane along the
$a$-axis with a small canting. They give rise to sublattice magnetization
vectors $\mathbf{M}_{1}$ and $\mathbf{M}_{2}$. (c) Sketch of the low frequency
quasi-ferromagnetic mode where a small magnetization (green arrow),
$\mathbf{m}$, precesses elliptically around the applied field (black arrow),
$\mathbf{H}$. (d) Schematics of the flip-chip VNA measurement. The sample
(gray disk) is placed on a CPW. The static magnetic field, $\mathbf{H}$, is
applied in the $a$-plane and with an angle, $\theta_{H}$, to the normal of the
$c$-axis of the crystal. The rf magnetic field (orange double-headed arrow),
$\mathbf{h}_{\text{rf}}$, of the CPW is parallel to the $c$-axis. (e) Sketch
of the BLS configuration. The magnetic field, $\mathbf{H}$, is applied
perpendicular to $c$-axis in the $a$-plane. The laser light (green) forms a
Gaussian beam and is focused on the surface of the sample (gray). The cone
angle of the objective lens is $\theta$. The incident laser light with an
incidence angle, $\varphi$ is scattered by magnons. We measure in back-
scattering geometry (dashed green arrow).

Depending on the purity of hematite crystals, a damping coefficient as low as
$7.8\times 10^{-6}$ was reported for the magnetic resonance [29]. The
hematite’s finite net magnetization and strikingly small damping of below
$10^{-5}$ make it hence suitable for magnonic applications. They allow for
inductive coupling to CPWs and long-distance SW transport, respectively.
However, there is no experiment reporting a measured SW dispersion for $k$
values accessible by CPWs with integrated microwave-to-magnon transducers [11,
12, 13]. The dispersion measured over a large wave vector regime is of
fundamental importance as it allows one to quantify the exchange stiffness
length $l_{e}$ with large precision. $l_{e}$ is the key parameter to estimate
the maximum possible spin-wave velocity of hematite in the GHz frequency
regime.
Here, we study the magnon band structure of bulk hematite at different wave
vectors $k$ by means of broadband microwave spectroscopy and $k$-resolved
inelastic Brillouin light scattering (BLS) (Fig. 1). Using a CPW in flip-chip
configuration we extract a magnetic damping parameter of $1.12\times 10^{-5}$
which is similar to the best YIG reported in Ref. 30. Still, the measured
spectra with BLS show broad linewidths. Our modelling substantiates that the
linewidth is explained by the SW dispersion relation, the large SW velocity
and the Gaussian profile of the laser used for inelastic light scattering. The
data substantiate a high group velocity of 10 km/s for $k=2.5$ rad/$\mu$m
which are excited easily by a micron-sized CPW [23, 31]. In an applied field
of 90 mT, the velocity increases to 16 km/s near $k=5$ rad/$\mu$m and levels
off to 23.3 km/s for $k\geq 25$ rad/$\mu$m. The latter value has routinely
been realized by transducers. Our findings substantiate hematite as a very
promising candidate for a sustainable future of magnonics as its growth avoids
the lead-based synthesis route used for high-quality YIG [7, 30].

Properties of hematite.— We first briefly review the relevant magnetic
properties of $\alpha\text{-Fe}_{2}\text{O}_{3}$. Hematite is the stable end
product of oxidation of magnetite [32] and known for its great abundance as
well as stability in an aqueous environment [33]. It is an insulating
antiferromagnet (AFM) with a corundum crystal structure. The arrangement of
the magnetic atoms of Fe in the crystal is shown in Fig. 1(a) [34]. At room
temperature and above the Morin transition temperature of $T_{\text{M}}=262$ K
the Fe+3 magnetic moments lie in the $c$-plane due to an easy plane
anisotropy, $H_{\text{A}}$, and stack antiferromagnetically along the $c$-axis
[Fig. 1(a)] [28, 21]. Within the $c$-plane, there is a weak 6-fold anisotropy
around the $c$-axis, $H_{\text{a}}$, which favors the magnetic moments to
align with the $a$-axes [Fig. 1(b)]. The magnetic moments of the two AFM
sublattices are slightly canted away from the $a$-axis by the Dzyaloshinskii-
Moriya (DM) interaction (Fig. 1(b)), resulting in a weak magnetic moment,
$\mathbf{m}$, perpendicular to both $a$\- and $c$\- axes at equilibrium [28,
21]. The magnetization amounts to about 2 kA/m [32]. The magnetization
dynamics of hematite in this weak ferromagnetic state offers two modes namely
the quasi-ferromagnetic mode (qFM or low frequency mode) and quasi-
antiferromagnetic mode (qAFM or high frequency mode) [27, 28, 21].
Here, we explore the qFM mode schematically depicted in Fig. 1 (c). The canted
AFM sublattice magnetization vectors precess elliptically around their
equilibrium direction. This results in an elliptical precession of the weak
magnetic moment, $\mathbf{m}$ (green arrow in Fig. 1 (c)), around the applied
field, $\mathbf{H}$ [27, 28, 21]. The frequency of the qFM mode was derived by
Pincus [27, 21] according to

$\displaystyle f_{r}=$ (1)
$\displaystyle\frac{\left|\gamma\right|\mu_{0}}{2\pi}\sqrt{H\sin\xi(H\sin\xi+H_{\text{D}})+2H_{\text{E}}(H_{\text{a}}+H_{\text{ME}})},$

where, $\gamma$, $\mu_{0}$, $H_{\text{E}}$, $H_{\text{D}}$ and $H_{\text{ME}}$
is the electron gyromagnetic ratio, vacuum permeability, exchange, DM and
spontaneous magnetoelastic effective field, respectively.
$\xi=\pi/2-\theta_{H}$ is the polar angle between $\mathbf{H}$ and the
$c$-axis ($z$-direction). We define $\theta_{H}$ in Fig. 1(d). Fink [35]
derived the dynamic susceptibility $\chi_{zz}(f,f_{r})$ for the qFM mode. The
real and imaginary parts read

$\displaystyle\text{Re}\left[\chi_{zz}(f,f_{r})\right]$ $\displaystyle=$
$\displaystyle\frac{\left(f_{r}^{2}-f^{2}\right)f_{r}^{2}}{\left(f_{r}^{2}-f^{2}\right)^{2}+\Delta
f^{2}f^{2}}~{}{\rm and}$ (2)
$\displaystyle\text{Im}\left[\chi_{zz}(f,f_{r})\right]$ $\displaystyle=$
$\displaystyle\frac{ff_{r}^{2}\Delta
f}{\left(f_{r}^{2}-f^{2}\right)^{2}+\Delta f^{2}f^{2}},$ (3)

respectively. Here, $f$ is the frequency of the radiofrequency (rf) magnetic
field $\mathbf{h}_{\text{rf}}$ (Fig. 1 (d)). The frequency linewidth, $\Delta
f$, is related to the magnetic damping parameter, $\alpha$, by

$\displaystyle\Delta f\approx 2\alpha H_{\text{E}}(\gamma/2\pi),$ (4)

for $H\ll H_{\text{E}}$. Unlike ferromagnets and uniaxial antiferromagnets,
the resonance line width of a canted antiferromagnet does not depend on
$f_{r}$ and is governed by the field-independent exchange frequency. Following
Turov [15, 16], the SW dispersion of the qFM mode for hematite near $k=0$ is
given by

$\displaystyle f_{m}(k)=$ (5)
$\displaystyle\frac{\left|\gamma\right|\mu_{0}}{2\pi}\sqrt{H(H+H_{\text{D}})+2H_{\text{E}}(H_{\text{a}}+H_{\text{ME}}+Ak^{2})},$

where $A=H_{\text{E}}l_{\text{e}}^{2}$ is the dispersion coefficient and
$l_{\text{e}}$ is the effective magnetic lattice parameter or exchange
stiffness length. For Eq. 5, we considered the experimental geometry for
$k$-resolved BLS measurements with an angle $\xi=\pi/2$ as described below.

Experimental techniques.— The broadband microwave spectroscopy was conducted
at room temperature above $T_{\text{M}}$ (Fig. 1(d)). The disk-shaped
$a$-plane $\alpha\text{-Fe}_{2}\text{O}_{3}$ crystal had a diameter of 2.3 mm
and thickness of 0.5 mm. Measurements were done in flip-chip configuration for
which the sample was placed on a CPW with signal (ground) line width of 165
$\mu$m (295 $\mu$m). The $a$-axis of the crystal was perpendicular to the disk
plane. The $c$-axis was in the plane and perpendicular to the CPW axis.
Injecting a radio-frequency (rf) current, $I_{\text{rf}}^{\text{in}}$, into
the CPW by port 1 of a vector network analyzer (VNA) induced the dynamic
magnetic field $\mathbf{h}_{\text{rf}}$ (orange double-head arrow). The rf
current was collected on the other end of the CPW by port 2 of the VNA
($I_{\text{rf}}^{\text{out}}$). The static magnetic field, $\mathbf{H}$, was
applied in the $a$-plane of the crystal in all VNA measurements ($yz$-plane in
Fig. 1 (d)). For the angle dependent measurements an external field of 90 mT
was applied and the angle, $\theta_{\text{H}}$, was varied in steps of
$\Delta\theta_{\text{H}}=2^{\circ}$. In case of field sweep measurements, the
applied field angle was perpendicular to the $c$-axis and in the $a$-plane of
the crystal ($\theta_{H}=0~{}$deg). The field amplitude was varied in steps of
$\Delta H=0.5$ mT.
Wave vector-resolved BLS measurements were done for $\theta_{H}=0~{}$deg on a
piece of the same crystal in back-scattering geometry (Fig. 1 (e)) using a
green laser with wavelength, $\lambda=532~{}$nm and wave vector
$k_{0}=2\pi/\lambda=11.81$ rad/$\mu$m. The external field was applied in the
$a$-plane and perpendicular to $c$-axis. The sample for BLS was irregularly
shaped. We ensured that it was tilted with respect to the incident laser beam
along the $y$-axis in such a way that the laser beam remained in the
$xz$-plane formed by the $a$\- and $c$-axis. Since the penetration depth of
the green laser in hematite is on the order of 75 nm [36], linear momentum
conservation holds only for the in-plane component of the transferred wave
vectors. Therefore, we define the transferred momentum from the light to
magnons along the $c$-axis as
$k_{z}=2(k_{0}\sin\varphi+k_{x}^{{}^{\prime}}\cos\varphi)$, where $\varphi$ is
the angle between the incident beam and the normal to the plane of the sample
($a$-plane) [37].

Figure 2: (a) Color-coded field dependent $\text{Im}(U)$ parameter measured on
the sample as indicated in Fig. 1(d) with $\theta_{H}=0$ deg. Solid red
circles and wine triangles are the frequencies extracted by fitting Eq. 7 on
the data. Blue solid line is obtained by fitting Eq. 1 with extracted $f_{1}$
values. (b) Imaginary and (c) real part of $U_{\text{exp}}$ (red symbols) and
$U_{\text{fit}}$ (black curves) for 70 mT, respectively. (d) Imaginary and (e)
real parts of $\chi_{1}$ (red curve), $\chi_{2}$ (blue curve) and their sum
(gray curve) for 70 mT, respectively.

We assume a Gaussian beam profile giving rise to a Fourier transform of the
beam intensity $I$ as

$\displaystyle I(k_{x}^{{}^{\prime}})=e^{k_{x}^{{}^{\prime}2}w_{0}^{2}/2}$ (6)

with $w_{0}=\frac{2}{k_{0}\text{NA}}$. $NA$ is the numerical aperture of the
lens [38, 39]. The momentum $k_{x}^{{}^{\prime}}$ has a projection on
$z$-direction due to the focusing of the beam.

Broadband microwave spectroscopy data.— Field-dependent VNA spectra are shown
in Fig. 2 (a). We depict the imaginary part of the quantity
$U(f,H)=i\ln\left[S_{21}(f,H)/S_{21}(f,H=0)\right]$ in a color-coded plot,
where $S_{21}$ is the complex scattering parameter measured by the VNA at a
given field, $H$. We identify two branches which we label $f_{1}$ and $f_{2}$
for positive fields. For a detailed analysis, we consider that the parameter
$U$ contains the susceptibility $\chi$ of the sample [40, 41] and the
electromagnetic response of the rf circuit used in the flip-chip method. To
account for the different contributions, we follow Refs. [40, 41] and fit the
measured $U$ with

$\displaystyle U_{\text{fit}}=$ (7) $\displaystyle
C[1+\chi_{0}+\chi(f,f_{1},\Delta f_{1})e^{i\phi_{1}}+\chi(f,f_{2},\Delta
f_{2})e^{i\phi_{2}}]$

as shown in Fig. 2(b) and (c) (black lines). $C$ is a real-numbered scaling
parameter, $\chi_{0}$ is a complex-numbered offset parameter, and $\phi_{i}$
are phase shift adjustments ($i=1,2$). Using Eq. 7, we extract the resonance
frequencies and linewidths $\Delta f_{i}$ for the two modes labelled by
$f_{1}$ and $f_{2}$. In Fig. 2 (b) and (c), we display the measured imaginary
and real parts of $U$ with red symbols. In Fig. 2 (d) and (e), we show the
extracted real and imaginary parts together with the total susceptibilities
(gray lines) from which the two resonant modes are identified. Their
frequencies extracted for different $H$ are depicted in Fig. 2 (a) with solid
red circles and magenta triangles. We focus on fields larger than 50 mT to
ensure a saturated state.
The blue line in Fig. 2(a) results from fitting Eq. 1 to branch $f_{1}$ with
$\xi=\pi/2$. From the fit, we obtain $\mu_{0}H_{\text{E}}=1003.61$ T,
$\mu_{0}H_{\text{D}}=2.34$ T and
$\mu_{0}(H_{\text{a}}+H_{\text{ME}})=88.64~{}\mu$T. Using Eq. 4 and the
experimental value of $\Delta f_{1}=0.63$ GHz for branch $f_{1}$, we extract a
damping parameter of $\alpha=1.12\times 10^{-5}$. This value is similar to the
best value reported for YIG in Ref. [30]. The branch $f_{2}$ will be discussed
later.
Angle-dependent VNA spectra are shown in Fig. 3.

Figure 3: Color-coded neighbor subtracted angle dependent VNA-FMR spectra
measured on the sample as indicated in Fig. 1(d) with $\mu_{0}H=90$ mT. Orange
curve depicts Eq. 1 by extracted material parameters.

To enhance the signal to noise ratio, we depict
$\Delta\left|S_{12}\right|=\left|S_{12}(H+\Delta H)-S_{12}(H)\right|$. For
such data, a zero-crossing (highlighted by the red curve) represents the
resonance frequency. The spectra show a 2-fold symmetry as expected for the
$c$-axis being in the plane of the applied magnetic fields. Introducing the
extracted parameters discussed above, Eq. 1 models well the resonance
frequency of branch $f_{1}$ as a function of angle $\theta_{H}$ (red line).
The angular dependency is hence consistent with the effective anisotropy field
extracted from the field-dependent data.

BLS data.— The BLS spectra for different values of incident angle, $\varphi$,
are shown in Fig. 4 (a). Black symbols in Fig. 4 (b) depict the frequency
position of the maximum BLS peak as a function of wave vector calculated from
the incidence angles shown in the legend of Fig. 4 (a). The red curve in Fig.
4 (b) is obtained by considering first the material parameters obtained from
VNA measurements and then fitting Eq. 5 to the black symbols in the same
graph. We obtain a dispersion coefficient of $\mu_{0}A=9.153\times 10^{-6}$
T.$\mu$m2 which leads to an exchange stiffness length of $l_{\text{e}}=0.955$
Å. Using Eq. 5 and the obtained parameters we calculate the SW group velocity,
$v_{\rm g}$, for $\mu_{0}H=90~{}$mT [blue line in Fig. 4 (c)]. The velocity
$v_{\rm g}$ increases significantly with $k$ and levels off at 23.3 km/s. Such
a high group velocity is the direct result of the strong exchange interaction
and at the same time vanishingly small net magnetization. Microstructured CPWs
used in magnonics offer spin-wave wave vectors around $k=2.5$ rad/$\mu$m [23,
31]. For such a value $k$, the qFM spin wave in hematite exhibits a group
velocity of $v_{\rm g}=10$ km/s similar to SWs in thick YIG and about a factor
of 10 larger compared to ultra-thin YIG [42]. The decay length of the SWs is
given by $l_{\rm d}=v_{\rm g}\tau$, where $\tau=(1/2\pi\alpha f_{m}(k))$ is
the relaxation time of the SW. For 30 mT applied field and $k=2.5$ rad/$\mu$m
we calculate $v_{\rm g}=13.4$ km/s and $\tau=840.6$ ns which leads to $l_{\rm
d}=11.3$ mm, again similar to thick YIG. This is due to low damping and high
group velocity of the qFM spin waves in hematite.

Figure 4: (a) BLS spectra for different incident angles, $\varphi$, measured
as indicated in Fig. 1(e) with $\mu_{0}H=90$ mT. (b) Extracted peak maxima
frequency (black squares) as a function of corresponding transferred wave
vector, $k$ and fitted dispersion relation (red curve) by Eq. 5. (c) Group
velocity (d) partial density of states and obtained from fitted magnon
dispersion at $\mu_{0}H=90$ mT. Inset in (d) depicts pDOS for $\varphi=0$
degree.

Despite the small damping of the hematite sample measured by the VNA, the
resonance peaks of the qFM mode in the BLS spectra show a large width of 10 to
20 GHz. Furthermore, the BLS peak taken at nearly normal incidence of the
laser beam (e.g. at $\varphi=0$ or $5$ deg) shows a strong asymmetric shape.
As $\varphi$ increases, the intensity of the prominent peak reduces and it
becomes more and more symmetric. To understand these observations, we
calculated the partial density of states (pDOS) by considering the magnon
dispersion relation and the Gaussian beam profile according to

$\displaystyle pDOS(f,\varphi)=$ (8)
$\displaystyle-\frac{1}{\pi}\int_{-\infty}^{+\infty}d{k_{x}^{{}^{\prime}}}\,\cos\varphi\,\text{Im}\left[\frac{I(k_{x}^{{}^{\prime}})}{f-f_{m}(k)+i\Delta
f}\right].$

Assuming only fully back reflected light we have
$k=2(k_{0}\sin\varphi+k_{x}^{{}^{\prime}}\cos\varphi)$. $\Delta f$ is the
resonance broadening due to the damping given by Eq. 4. The numerical aperture
of the objective lens was $NA=0.18$. The calculated pDOS for different
incident angles is plotted in Fig. 4 (d). Comparing Figs. 4(a) and (c), the
experimental BLS peaks are broad because they contain a certain frequency
regime of the band structure which is determined by the Gaussian wave vector
distribution of $k=2k_{0}\sin\varphi+\Delta k$. The central wave vector is
$k_{c}=2k_{0}\sin\varphi$ and the distribution function for $\Delta
k=2k_{x}^{{}^{\prime}}\cos\varphi$ is given by $I(k_{x}^{{}^{\prime}})$ (Eq.
6). We attribute the broad BLS peaks hence to the high group velocity of
magnons which leads to a peak width proportional to $v_{g}\times\Delta k$. The
asymmetry of BLS peaks for small $\varphi$ can be understood in terms of the
Van Hove singularity of the pDOS near $k=0$. As $\varphi$ increases, the part
of the band structure which is relevant for the collected light shifts away
from $k=0$. Consequently, the peak gets more symmetric with $\varphi$. The
peak intensity reduces with $\varphi$ due to the factor $\cos\varphi$ in Eq.
8. We note that Eq. 8 does not include all possible scattering processes.
Still it provides a good qualitative understanding about the contributions
giving rise to the characteristics shape and signal strength of BLS peaks as a
function of $\varphi$.

Discussion.— The branch of mode $f_{2}$ in Fig. 2(a) is higher by about
$\delta f=1$ GHz than $f_{1}$ at the same field $H$. A second branch was
reported also in Ref. 43. Here the authors studied the qAFM mode at zero field
and assumed a distribution of magnetic domains. We applied a large enough
magnetic field to avoid domains. A domain formation can not explain our second
branch. Another possibility for $f_{2}$ is a standing wave along the thickness
of the sample or a SW excited with a discrete wave vector coming from the CPW.
However, for these cases a quantitative estimate based on the magnon
dispersion obtained with BLS (Eq. 5) led to a frequency separation of much
smaller than 1 GHz.
The remaining explanation for $f_{2}$ is a nonuniform magnetoelastic field in
the sample induced when fixing the sample on the CPW. We attribute the
observed frequency splitting to different strains in different parts of the
sample that is either in contact with the CPW conductors or floating on the
CPW gaps. A difference of $\delta H_{\text{ME}}=24~{}\mu$T would account for
$\delta f=f_{2}-f_{1}=1$ GHz. The effect of magnetoelastic interaction by
unidirectional compression, $\mathbf{p}$, in the basal plane of the crystal,
on the qFM mode is given by replacing $H_{\text{ME}}$ with
$H_{\text{ME}}^{\prime}=H_{\text{ME}}-Rp\cos 2\psi$ [44, 45]. Here, $R=287$
$\mu$T/bar [45] is a coefficient determined by the elastic and magnetoelastic
parameters of the crystal and $\psi=\pi/2$ is the angle between $\mathbf{p}$
and $\mathbf{H}$. Using $\delta H_{\text{ME}}=Rp$ we obtain $p=0.084$ bar.
This value corresponds to a force of 14.6 mN on the parts of the crystal that
are in contact with the CPW conductors. The evaluated force is one order of
magnitude smaller than the weight of our sample and indicates the known
sensitivity of hematite towards magnetoelastic effects.
From our experiments, we do not determine $H_{\text{a}}$ and $H_{\text{ME}}$
separately as they enter Eq. 1 in the same way. To separate the small
$H_{\text{a}}$ from $H_{\text{ME}}$ an angle dependent VNA measurement with
the magnetic field applied in the $c$-plane of the crystal would be required.
The existing sample was not suitable for that.
Our BLS data were acquired on a seven times larger wave vector regime compared
to the data used in Ref. 23 where, the authors extracted an exchange stiffness
length $l_{\text{e}}$ of $1.2~{}$Å. From our extended dataset, we evaluate the
value $l_{\text{e}}=0.96$ Å. The precise determination of $l_{\text{e}}$ is of
key importance as it determines the SW group velocity. Contrary to the
ferrimagnetic YIG, dipolar effects are not expected to play an important role
in SW dispersion of hematite. Hence, hematite thin films can provide similarly
large spin-wave velocities as reported here for the bulk crystal. As a
consequence, they potentially outperform YIG thin films concerning speed and
decay lengths at wave vectors which are realized by the state-of-the-art
transducers. Furthermore, hematite is based on earth abundant elements and as
an end product of oxidation of magnetite a stable natural mineral suggesting
sustainable synthesis routes.

Conclusion.— We have measured the magnon dispersion relation of hematite for
wave vectors $k$ which are relevant for timely experiments in magnonics. The
damping coefficient of the studied natural crystal was $1.1\times 10^{-5}$ at
room temperature. This value is only 40 % larger than the best value reported
for pure hematite and is already as good as the best YIG. The estimated spin-
wave decay length for $k=2.5~{}$rad/$\mu$m is larger than 1 cm in a small
magnetic field. In optimized thin films, current microwave-to-magnon
transducers are expected to achieve larger group velocities in hematite than
in YIG. The reported properties suggest that hematite can become the fruit fly
of sustainable modern magnonics.

Note added.— While completing the manuscript about our experiments on hematite
[22], we became aware of Ref. 23. The authors measured group velocities and
the spin-wave dispersion via integrated CPWs for 1 rad$/\mu$m$~{}\leq k\leq
3.5~{}$rad$/\mu$m. Our BLS measurements cover a seven times larger regime of
$k$ values enabling an improved evaluation of the parameter $l_{e}$. This
parameter is decisive to estimate the saturation velocity of GHz SWs in
hematite.
The scientific colour maps developed by Crameri et. al. [46] is used in this
study to prevent visual distortion of the data and exclusion of readers with
colour-vision deficiencies [47].

Acknowledgement.— The authors thank SNSF for financial support via grant
177550. We acknowledge discussions with A. Mucchietto, M. Mruczkiewicz, S.
Nikitov and A. Sadovnikov. The different pieces of the hematite crystal were
provided by J.-P. Ansermet and M. Bialek. We thank them for the support and
discussions.

## References

  * Kruglyak  _et al._ [2010] V. V. Kruglyak, S. O. Demokritov, and D. Grundler, J. Phys. D. Appl. Phys. 43, 264001 (2010).
  * Barman  _et al._ [2021] A. Barman, G. Gubbiotti, S. Ladak, A. O. Adeyeye, M. Krawczyk, J. Grafe, C. Adelmann, S. Cotofana, A. Naeemi, V. I. Vasyuchka, B. Hillebrands, S. A. Nikitov, H. Yu, D. Grundler, A. V. Sadovnikov, A. A. Grachev, S. E. Sheshukova, J. Y. Duquesne, M. Marangolo, G. Csaba, W. Porod, V. E. Demidov, S. Urazhdin, S. O. Demokritov, E. Albisetti, D. Petti, R. Bertacco, H. Schultheiss, V. V. Kruglyak, V. D. Poimanov, S. Sahoo, J. Sinha, H. Yang, M. Münzenberg, T. Moriyama, S. Mizukami, P. Landeros, R. A. Gallardo, G. Carlotti, J. V. Kim, R. L. Stamps, R. E. Camley, B. Rana, Y. Otani, W. Yu, T. Yu, G. E. Bauer, C. Back, G. S. Uhrig, O. V. Dobrovolskiy, B. Budinska, H. Qin, S. Van Dijken, A. V. Chumak, A. Khitun, D. E. Nikonov, I. A. Young, B. W. Zingsem, and M. Winklhofer, J. Phys. Condens. Matter 33, 413001 (2021).
  * Baumgaertl and Grundler [2022] K. Baumgaertl and D. Grundler 10.48550/arxiv.2208.10923 (2022).
  * Tkachenko  _et al._ [2012] V. S. Tkachenko, A. N. Kuchko, M. Dvornik, and V. V. Kruglyak, Appl. Phys. Lett. 101, 152402 (2012).
  * Vogt  _et al._ [2012] K. Vogt, H. Schultheiss, S. Jain, J. E. Pearson, A. Hoffmann, S. D. Bader, and B. Hillebrands, Appl. Phys. Lett. 101, 042410 (2012).
  * Mieszczak  _et al._ [2020] S. Mieszczak, O. Busel, P. Gruszecki, A. N. Kuchko, J. W. Kłos, and M. Krawczyk, Phys. Rev. Appl. 13, 054038 (2020), arXiv:2001.11356 .
  * Doughty and White [1960] D. C. Doughty and E. A. D. White, Acta Crystallogr. 13, 761 (1960).
  * Serga  _et al._ [2010] A. A. Serga, A. V. Chumak, and B. Hillebrands, J. Phys. D. Appl. Phys. 43, 264002 (2010).
  * Sparks  _et al._ [1961] M. Sparks, R. Loudon, and C. Kittel, Phys. Rev. 122, 791 (1961).
  * Maendl  _et al._ [2017] S. Maendl, I. Stasinopoulos, and D. Grundler, Appl. Phys. Lett. 111, 012403 (2017).
  * Yu  _et al._ [2016] H. Yu, O. D’Allivy Kelly, V. Cros, R. Bernard, P. Bortolotti, A. Anane, F. Brandl, F. Heimbach, and D. Grundler, Nat. Commun. 7, 11255 (2016).
  * Baumgaertl  _et al._ [2020] K. Baumgaertl, J. Gräfe, P. Che, A. Mucchietto, J. Förster, N. Träger, M. Bechtel, M. Weigand, G. Schütz, and D. Grundler, Nano Lett. 20, 7281 (2020).
  * Che  _et al._ [2020] P. Che, K. Baumgaertl, A. Kúkol’ová, C. Dubs, and D. Grundler, Nat. Commun. 11, 1445 (2020).
  * Chumak  _et al._ [2017] A. V. Chumak, A. A. Serga, and B. Hillebrands, J. Phys. D. Appl. Phys. 50, 244001 (2017).
  * Turov [1959] E. Turov, J. Exp. Theor. Phys. 9, 890 (1959).
  * Keffer [1966] F. Keffer,  _Spin Waves_, edited by H. P. J. Wijn (Springer, 1966) pp. 1–273.
  * Samuelsen and Shirane [1970] E. J. Samuelsen and G. Shirane, Phys. status solidi 42, 241 (1970).
  * Hortensius  _et al._ [2021] J. R. Hortensius, D. Afanasiev, M. Matthiesen, R. Leenders, R. Citro, A. V. Kimel, R. V. Mikhaylovskiy, B. A. Ivanov, and A. D. Caviglia, Nat. Phys. 17, 1001 (2021), arXiv:2105.05886 .
  * Sirtori [2002] C. Sirtori, Nature 417, 132 (2002).
  * Osborne [2008] I. S. Osborne, Science 320, 1262 (2008).
  * Morrish [1995] A. H. Morrish,  _Canted Antiferromagnetism: Hematite_ (World Scientific, 1995).
  * Hamdi  _et al._ [2022] M. Hamdi, F. Posva, and D. Grundler, in  _67th Annu. Conf. Magn. Magn. Mater. (MMM 2022)_ (2022) pp. GOE–12.
  * Wang  _et al._ [2022] H. Wang, R. Yuan, Y. Zhou, Y. Zhang, J. Chen, S. Liu, H. Jia, D. Yu, J.-P. Ansermet, C. Song, and H. Yu,  (2022), arXiv:2211.10989 .
  * Lebrun  _et al._ [2018] R. Lebrun, A. Ross, S. A. Bender, A. Qaiumzadeh, L. Baldrati, J. Cramer, A. Brataas, R. A. Duine, and M. Kläui, Nature 561, 222 (2018).
  * Lebrun  _et al._ [2020] R. Lebrun, A. Ross, O. Gomonay, V. Baltz, U. Ebels, A. L. Barra, A. Qaiumzadeh, A. Brataas, J. Sinova, and M. Kläui, Nat. Commun. 11, 6332 (2020).
  * Wang  _et al._ [2021] H. Wang, Y. Xiao, M. Guo, E. Lee-Wong, G. Q. Yan, R. Cheng, and C. R. Du, Phys. Rev. Lett. 127, 117202 (2021).
  * Pincus [1960] P. Pincus, Phys. Rev. Lett. 5, 13 (1960).
  * Artman  _et al._ [1965] J. O. Artman, J. C. Murphy, and S. Foner, Phys. Rev. 138, A912 (1965).
  * Searle and Wang [1968] C. W. Searle and S. T. Wang, J. Appl. Phys. 39, 1025 (1968).
  * Shone [1985] M. Shone, Circuits Systems Signal Process 4, 89 (1985).
  * Watanabe  _et al._ [2021] S. Watanabe, V. S. Bhat, K. Baumgaertl, M. Hamdi, and D. Grundler, Sci. Adv. 7, eabg3771 (2021).
  * Moskowitz  _et al._ [2015] B. M. Moskowitz, M. Jackson, and V. Chandler, in  _Treatise Geophys. Resour. Near-Surface Earth_, Vol. 11, edited by G. Schubert (Elsevier, 2015) 2nd ed., Chap. 11.05, pp. 139–174.
  * Iandolo  _et al._ [2015] B. Iandolo, B. Wickman, I. Zorić, and A. Hellman, J. Mater. Chem. A 3, 16896 (2015).
  * Hill  _et al._ [2008] A. H. Hill, F. Jiao, P. G. Bruce, A. Harrison, W. Kockelmann, and C. Ritter, Chem. Mater. 20, 4891 (2008).
  * Fink [1964] H. J. Fink, Phys. Rev. 133, A1322 (1964).
  * Querry [1984] M. R. Querry,  _Optical Constants_, Tech. Rep. (Missouri University, 1984).
  * Sebastian  _et al._ [2015] T. Sebastian, K. Schultheiss, B. Obry, B. Hillebrands, and H. Schultheiss, Front. Phys. 3, 35 (2015).
  * Novotny and Hecht [2012] L. Novotny and B. Hecht,  _Principles of Nano-Optics_ (Cambridge University Press, 2012) p. 584.
  * Levy  _et al._ [2019] U. Levy, N. Davidson, and Y. Silberberg, Adv. Opt. Photonics 11, 828 (2019).
  * Kalarickal  _et al._ [2006] S. S. Kalarickal, P. Krivosik, M. Wu, C. E. Patton, M. L. Schneider, P. Kabos, T. J. Silva, and J. P. Nibarger, J. Appl. Phys. 99, 093909 (2006).
  * Bilzer  _et al._ [2007] C. Bilzer, T. Devolder, P. Crozat, C. Chappert, S. Cardoso, and P. P. Freitas, J. Appl. Phys. 101, 074505 (2007).
  * Yu  _et al._ [2014] H. Yu, O. d. Kelly, V. Cros, R. Bernard, P. Bortolotti, A. Anane, F. Brandl, R. Huber, I. Stasinopoulos, and D. Grundler, Sci. Rep. 4, 6848 (2014).
  * Bialek  _et al._ [2022] M. Bialek, J. Zhang, H. Yu, and J. P. Ansermet, Appl. Phys. Lett. 121, 032401 (2022).
  * Levitin  _et al._ [1969] R. Z. Levitin, A. S. Pakhomov, and V. A. Shchurov, J. Exp. Theor. Phys. 29, 669 (1969).
  * Maksimenkov and Ozhogin [1974] P. P. Maksimenkov and V. I. Ozhogin, J. Exp. Theor. Phys. 38, 324 (1974).
  * [46] F. Crameri, Zenodo 10.5281/ZENODO.5501399.
  * Crameri  _et al._ [2020] F. Crameri, G. E. Shephard, and P. J. Heron, Nat. Commun. 11, 5444 (2020).