2023

[1]Salomon Kabongo

1]L3S Research Center, Leibniz University of Hannover, Hannover, Lower-saxony,
Germany

2]TIB, Leibniz Information Centre for Science and Technology, Hannover, Lower-
saxony, Germany

# ORKG-Leaderboards: A Systematic Workflow for Mining Leaderboards as a
Knowledge Graph

<EMAIL_ADDRESS>Jennifer D’Souza<EMAIL_ADDRESS>Sören Auer
<EMAIL_ADDRESS>[  [

###### Abstract

The purpose of this work is to describe the orkg-Leaderboard software designed
to extract leaderboards defined as Task-Dataset-Metric tuples automatically
from large collections of empirical research papers in Artificial Intelligence
(AI). The software can support both the main workflows of scholarly
publishing, viz. as LaTeX files or as PDF files. Furthermore, the system is
integrated with the Open Research Knowledge Graph (ORKG) platform, which
fosters the machine-actionable publishing of scholarly findings. Thus the
system output, when integrated within the ORKG’s supported Semantic Web
infrastructure of representing machine-actionable ’resources’ on the Web,
enables: 1) broadly, the integration of empirical results of researchers
across the world, thus enabling transparency in empirical research with the
potential to also being complete contingent on the underlying data source(s)
of publications; and 2) specifically, enables researchers to track the
progress in AI with an overview of the state-of-the-art (SOTA) across the most
common AI tasks and their corresponding datasets via dynamic ORKG frontend
views leveraging tables and visualization charts over the machine-actionable
data. Our best model achieves performances above 90% F1 on the leaderboard
extraction task, thus proving orkg-Leaderboards a practically viable tool for
real-world usage. Going forward, in a sense, orkg-Leaderboards transforms the
leaderboard extraction task to an automated digitalization task, which has
been, for a long time in the community, a crowdsourced endeavor.

###### keywords:

Table mining, Information extraction, Scholarly text mining, Neural machine
learning, Semantic networks, Knowledge graphs.

##  1 Introduction

Shared tasks—a long-standing practice in the Natural Language Processing (NLP)
community—are competitions to which researchers or teams of researchers submit
systems that address a specific Task, evaluated based on a predefined Metric
sharedtask2 . Seen as “drivers of progress” for empirical research, they
attract diverse participating groups from both academia and industry, as well
as are harnessed as test-beds for new emerging shared tasks on under-
researched and under-resourced topics sharedtask1 . Examples of long-standing
Shared Tasks include the Conference and Labs of the Evaluation Forum
(CLEF)111http://www.clef-initiative.eu/ organized at the Conference on Natural
Language Learning (CoNLL)222https://www.signll.org/conll, the International
Workshop on Semantic Evaluation (SEMEVAL)333https://semeval.github.io/, or the
biomedical domain-specific BioNLP Shared Task Series bionlp  and the Critical
Assessment of Information Extraction in Biology (BioCreative)
444https://biocreative.bioinformatics.udel.edu/tasks/. Being inherently
competitive, Shared Tasks offer as a main outcome Leaderboards that publish
participating system rankings.

Inspired by Shared Tasks, the Leaderboards construct of progress trackers is
simultaneously taken up for the recording of results in the field of empirical
Artificial Intelligence (AI) at large. Here the information is made available
via the traditional scholarly publishing flow as PDFs and preprints, unlike in
Shared Tasks where the community is relegated to a list of researchers wherein
tracking the dataset creators and individual systems applied is less
cumbersome as they can be found within the list of researchers that sign up to
organize or participate in the task. On the other hand, general publishing
avenues bespeak of a deluge of peer-reviewed scholarly publications
jinha2010article  and PDF preprints ahead (or even instead) of peer-reviewed
publications chiarelli2019accelerating . This high-volume publication trend
problem is only compounded by the diversity in empirical AI research where
Leaderboards can potentially be searched and tracked on research problems in
various fields such as Computer Vision, Time Series Analysis, Games, Software
engineering, Graphs, Medicine, Speech, Audio processing, Adversarial learning,
etc. Thus the problem of obtaining completed Leaderboard representations of
empirical research seems a tedious if not completely insurmountable task.

Regardless of the setup, i.e. from Shared Tasks or empirical AI research,
another problem in the current methodology is the information representation
of Leaderboards which is often via Github repositories, shared task websites,
or researchers’ personal websites. Some well-known websites that exist to this
end are: PapersWithCode (PwC) PWC ,555https://paperswithcode.com NLP-Progress
np , AI-metrics aim , SQUaD explorer squad-exp , Reddit SOTA reddit . The
problem with leveraging websites for storing Leaderboards is the resulting
rich data’s lack of machine-actionability and integrability. In other words,
unstructured, non-machine-actionable information from scholarly articles is
converted to semi-structured information on the websites which still
unfortunately remain non-machine-actionable. In the broader context of
scholarly knowledge, the FAIR guiding principles for scientific data
management and stewardship wilkinson2016fair  identify general guidelines for
making data and metadata machine-actionable by making them maximally Findable,
Accessible, Interoperable, and Reusable for machines and humans alike.
Semantic Web technologies such as the W3C recommendations Resource Description
Framework (RDF) and Web Ontology Language (OWL) are the most widely-accepted
choice for implementing the FAIR guiding principles jacobsen2019fair . In this
context, the Open Research Knowledge Graph (ORKG) orkg  https://orkg.org/ as a
next-generation library for digitalized scholarly knowledge publishing
presents a framework fitted with the necessary Semantic Web technologies to
enable the encoding of Leaderboards as FAIR, machine-actionable data. Adopting
semantic standards to represent Leaderboards not just Task-Dataset-Metric but
also related information such as code links, pre-trained models, and so on can
be made machine-actionable and consequently queryable. This would directly
address the lack of transparency and integration of various results’ problems
identified in current methods of recording empirical research sharedtask3 ;
sharedtask1 ; sharedtask2 .

This work, taking note of the two main problems around Leaderboard
construction, i.e. information capture and information representation,
proposes solutions to address them directly. First, regarding information
capture, we recognize due to the overwhelming volume of data, now more than
ever, that it is of paramount importance to empower scientists with automated
methods to generate the Leaderboards oversight. The community could greatly
benefit from an automatic system that can generate a Leaderboard as a Task-
Dataset-Metric tuple over large collections of scholarly publications both
covering empirical AI, at large and encapsulating Shared Tasks, specifically.
Thus, we empirically tackle the Leaderboard knowledge mining machine learning
(ML) task via a detailed set of evaluations involving large datasets for the
two main publishing workflows, i.e. as LaTeX source and PDF, with several ML
models. For this purpose, we extend the experimental settings from our prior
work kabongo2021automated  by adding support for information extraction from
LaTeX code source and compared empirical evaluations on longer input sequences
(beyond 512 tokens) for both XLNet and BigBird zaheer2020big . Our ultimate
goal with this study is to help the Digital Library (DL) stakeholders to
select the optimal tool to implement knowledge-based scientific information
flows w.r.t. Leaderboards. To this end, we evaluate four state-of-art
transformer models, viz. BERT, SciBERT, XLNet, and BigBird, each of which has
its own unique strengths. Second, regarding information representation, orkg-
Leaderboards workflow, is integrated in the knowledge-graph-based DL
infrastructure of the ORKG orkg . Thus the resulting data will be made
machine-actionable and served via the dynamic ORKG Frontend views
666https://orkg.org/benchmarks and further queryable via structured queries
over the larger scholarly KG using SPARQL777https://orkg.org/triplestore or
https://orkg.org/sparql/.

In summary, the contributions of our work are:

  1. 1.

we construct a large empirical corpus containing over 4,000 scholarly articles
and 1,548 leaderboards TDM triples for the development of text mining systems;

  2. 2.

we empirically evaluate three different transformer models and leverage the
best model, i.e. orkg-LeaderboardsXLNet, for the ORKG benchmarks curation
platform;

  3. 3.

produced a pipeline that works both with the raw PDF and the LaTeX code source
of a research publication.

  4. 4.

we extended our previous work kabongo2021automated  by empirically
investigating our approach with longer input beyond the traditional 512
sequence length limit by BERT-based models, and added support for both
mainstreams of research publication PDFs and LaTeX code source.

  5. 5.

in a comprehensive empirical evaluation of orkg-Leaderboards for both LaTeX
and PDFs based pipelines, we obtain around 93% micro and 92% macro F1 scores
which outperform existing systems by over 20 points.

To the best of our knowledge, the orkg-Leaderboards system obtains state-of-
the-art results for the Leaderboard extraction defined as Task-Dataset-Metric
triples extraction from empirical AI research articles handling both LaTeX and
PDF formats. Thus orkg-Leaderboards can be readily leveraged within KG-based
DLs and be used to comprehensively construct Leaderboards with more concepts
beyond the TDM triples. To facilitate further research, our
data888https://doi.org/10.5281/zenodo.7419877 and
code999https://github.com/Kabongosalomon/task-dataset-metric-nli-
extraction/tree/latex are made publicly available.

##  2 Definitions

This section defines the central concepts in the Task-Dataset-Metric
extraction schema of orkg-Leaderboards. Furthermore, the semantic concepts
used in the information representation for the data in the ORKG are defined.

##### Task.

It is a natural language mention phrase of the theme of the investigation in a
scholarly article. Alternatively referred to as research problem d2022computer
or focus ftd . An article can address one or more tasks. Task mentions being
often found in the article Title, Abstract, Introduction, or Results tables
and discussion. E.g., question answering, image classification, drug
discovery, etc.

##### Dataset.

A mention phrase of the dataset encapsulates a particular Task used in the
machine learning experiments reported in the respective empirical scholarly
articles. An article can report experiments on one or more datasets. Dataset
mentions are found in similar places in the article as Task mentions. E.g.,
HIV
dataset101010https://wiki.nci.nih.gov/display/NCIDTPdata/AIDS+Antiviral+Screen+Data,
MNIST lecun1998gradient , Freebase 15K bordes2013translating , etc.

##### Metric.

Phrasal mentions of the standard of measurement111111https://www.merriam-
webster.com/dictionary/metric used to evaluate and track the performance of
machine learning models optimizing a Dataset objective based on a Task. An
article can report performance evaluations on one or more metrics. Metrics are
generally found in Results tables and discussion sections in scholarly
articles. E.g., BLEU (bilingual evaluation understudy) papineni2002bleu  used
to evaluate “machine translation” tasks, F-measure sasaki2007truth  used
widely in “classification” tasks, MRR (mean reciprocal rank) voorhees1999trec
used to evaluate the correct ordering of a list of possible responses in
“information retrieval” or “question answering” tasks, etc.

##### Benchmark.

ORKG Benchmarks (https://orkg.org/benchmarks) organize the state-of-the-art
empirical research within ORKG research fields121212https://orkg.org/fields
and are powered in part by automated information extraction supported by the
orkg-Leaderboards software within a human-in-the-loop curation model. A
benchmark per research field is fully described in terms of the following
elements: research problem or Task, Dataset, Metric, Model, and Code. E.g., a
specific instance of an ORKG Benchmark
131313https://orkg.org/benchmark/R121022/problem/R120872 on the “Language
Modelling” Task, evaluated on the “WikiText-2” Dataset, evaluated by
“Validation perplexity” Metric with a listing of various reported Models with
respective Model scores.

##### Leaderboard.

Is a dynamically computed trend-line chart on respective ORKG Benchmark pages
leveraging their underlying machine-actionable data from the Knowledge Graph.
Thus, Leaderboards depict the performance trend-line of models developed over
time based on specific evaluation Metrics.

##  3 Related Work

There is a wealth of research in the NLP community on specifying a collection
of extraction targets as a unified information-encapsulating unit from
scholarly publications. The two main related lines of work that are at the
forefront are: 1) extracting instructional scientific content that captures
the experimental process bioassay ; chemrecipes ; labprotocols ;
mysore2019materials ; kuniyoshi2020annotating ; and 2) extracting terminology
as named entity recognition objectives ftd ; handschuh2014acl ;
augenstein2017semeval ; Luan2018MultiTaskIO ; lrec2020  to generally obtain a
concise representation of the scholarly article which also includes the
Leaderboard information unit hou2019identification ; scirex ; mondal2021end .

Starting with the capture of the experimental process, bioassay  proposed an
AI-based clustering method for the automatic semantification of bioassays
based on the specification of the BAO
ontology141414https://github.com/BioAssayOntology/BAO. In labprotocols , they
annotate wet lab protocols, covering a large spectrum of experimental biology
w.r.t. lab procedures and their attributes including materials, instruments,
and devices used to perform specific actions as a prespecified machine-
readable format as opposed to the ad-hoc documentation norm. Within scholarly
articles, such instructions are typically published in the Materials and
Method section in Biology and Chemistry fields. Similarly, in chemrecipes ;
mysore2019materials , to facilitate machine learning models for automatic
extraction of materials syntheses reactions and procedures from text, they
present datasets of synthesis procedures annotated with semantic structure by
domain experts in Materials Science. The types of information captured include
synthesis operations (i.e. predicates), and the materials, conditions,
apparatus, and other entities participating in each synthesis step.

In terms of extracting terminology to obtain a concise representation of the
article, an early dataset called the FTD corpus ftd  defined focus, technique,
and domain entity types which were leveraged to examine the influence between
research communities. Another dataset, the ACL RD-TEC corpus handschuh2014acl
identified seven conceptual classes for terms in the full-text of scholarly
publications in Computational Linguistics, viz. Technology and Method, Tool
and Library, Language Resource, Language Resource Product, Models, Measures
and Measurements, and Other to generate terminology lists. Similarly,
terminology mining is the task of scientific keyphrase extraction. Extracting
keyphrases is an important task in publishing platforms as they help recommend
articles to readers, highlight missing citations to authors, identify
potential reviewers for submissions, and analyze research trends over time.
Scientific keyphrases, in particular, of type Processes, Tasks and Materials
were the focus of the SemEval17 corpus annotations augenstein2017semeval
which included full-text articles in Computer Science, Material Sciences, and
Physics. The SciERC corpus Luan2018MultiTaskIO  provided a resource of
annotated abstracts in Artificial Intelligence which annotations for six
concepts, viz. Task, Method, Metric, Material, Other-Scientific Term, and
Generic to facilitate the downstream task of generating a searchable KG of
these entities. On the other hand, the STEM-ECR corpus lrec2020  notable for
its multidisciplinarity included 10 different STEM domains annotated with four
generic concept types, viz. Process, Method, Material, and Data that mapped
across all domains, and further with terms grounded in the real world via
Wikipedia/Wiktionary links. Finally, several works have recently emerged
targeting the task of Leaderboard extraction, with the TDM-IE pioneering work
hou2019identification  also addressing the much harder Score element as an
extraction target. Later works attempted the document-level information
extraction task by defining explicit relations evaluatedOn between Task and
Dataset elements and evaluatedBy between Task and Metric scirex ;
mondal2021end . In contrast, in our prior orkg-TDM system kabongo2021automated
and in this present extended orkg-Leaderboards experimental report, we attempt
the Task-Dataset-Metric tuple extraction objective assuming implicitly encoded
relations. This simplifies the pipelined entity and relation extraction
objectives as a single tuple inference task operating over the entire
document. Nevertheless, scirex ; mondal2021end  also defined coreference
relations between similar term mentions, which can be leveraged
complementarily in our work to enrich the respective Task-Dataset-Metric
mentions.

##  4 The ORKG-Leaderboards Task Dataset

###  4.1 Task Definition

The Leaderboard extraction task addressed in orkg-Leaderboards can be
formalized as follows. Let $p$ be a paper in the collection $P$. Each $p$ is
annotated with at least one triple $(t_{i},d_{j},m_{k})$ where $t_{i}$ is the
$i^{th}$ Task defined, $d_{j}$ the $j^{th}$ Dataset that encapsulates Task
$t_{i}$, and $m_{k}$ is the $k^{th}$ evaluation Metric used to evaluate a
system performance on a Task’s Dataset. While each paper has a varying number
of Task-Dataset-Metric triples, they occur at an average of roughly 4 triples
per paper.

In the supervised inference task, the input data instance corresponds to the
pair: a paper $p$ represented as the DocTAET context feature $p_{DocTAET}$ and
its Task-Dataset-Metric triple $(t,d,m)$. The inference data instance, then is
$(c\mathchar 24635\relax\;[(t,d,m),p_{DocTAET}])$ where $c\in\\{true,false\\}$
is the inference label. Thus, specifically, our Leaderboard extraction problem
is formulated as a natural language inference task between the DocTAET context
feature $p_{DocTAET}$ and the $(t,d,m)$ triple annotation. $(t,d,m)$ is $true$
if it is among the paper’s Task-Dataset-Metric triples, where they are
implicitly assumed to be related, otherwise $false$. The $false$ instances are
artificially created by a random selection of inapplicable $(t,d,m)$
annotations from other papers. Cumulatively, Leaderboard construction is a
multi-label, multi-class inference problem.

####  4.1.1 DocTAET Context Feature

The DocTAET context feature representation hou2019identification  selects only
the parts of a paper where the Task-Dataset-Metric mentions are most likely to
be found. While the Leaderboard extraction task is applicable on the full
scholarly paper content, feeding a machine learning model with the full
article is disadvantageous since the model will be fed with a large chunk of
text which would be mostly noise as it is redundant to the extraction task.
Consequently, an inference model fed with large amounts of noise as contextual
input cannot generalize well. Instead, the DocTAET feature was designed to
heuristically select only those parts of an article that are more likely to
contain Task-Dataset-Metric mentions as true contextual information signals.
Specifically, as informative contextual input to the machine learning model,
DocTAET captures sentences from four specific places in the article that are
most likely to contain Task-Dataset-Metric mentions, viz. the Document Title,
Abstract, first few lines of the Experimental setup section and Table content
and captions.

###  4.2 Task Dataset

To facilitate supervised system development for the extraction of Leaderboards
from scholarly articles, we built an empirical corpus that encapsulates the
task. Leaderboard extraction is essentially an inference task over the
document. To alleviate the otherwise time-consuming and expensive corpus
annotation task involving expert annotators, we leverage distant supervision
from the available crowdsourced metadata in the PwC
(https://paperswithcode.com/) KB. In the remainder of this section, we explain
our corpus creation and annotation process.

####  4.2.1 Scholarly Papers and Metadata from the PwC Knowledge Base.

We created a new corpus as a collection of scholarly papers with their Task-
Dataset-Metric triple annotations for evaluating the Leaderboards extraction
task inspired by the original IBM science result extractor
hou2019identification  corpus. The collection of scholarly articles for
defining our Leaderboard extraction objective is obtained from the publicly
available crowdsourced leaderboards PwC. It predominantly represents articles
in the Natural Language Processing and Computer Vision domains, among other AI
domains such as Robotics, Graphs, Reasoning, etc. Thus, the corpus is
representative for empirical AI research. The original downloaded collection
(timestamp 2021-05-10 at 12:30:21)151515Our corpus was downloaded from the PwC
GitHub repository https://github.com/paperswithcode/paperswithcode-data and
was constructed by combining the information in the files All papers with
abstracts and Evaluation tables which included article urls and TDM
crowdsourced annotation metadata. was pre-processed to be ready for analysis.
While we use the same method here as the science result extractor, our corpus
is different in terms of both labels and size, i.e. number of papers, as many
more Leaderboards have been crowdsourced and added to PwC since the original
work. Furthermore, as an extension to our previous work kabongo2021automated
on this theme, based on the two main scholarly publishing workflows i.e. as
LaTeX or PDF, correspondingly two variants of our corpus are created and their
models respectively developed.

Recently, publishers are increasingly encouraging paper authors to provide the
supporting LaTeX files accompanying the corresponding PDF article. The
advantage of having the LaTeX source files is that they contain the original
article in plain-text format and thus result in cleaner data in downstream
analysis tasks. Our prior orkg-TDM kabongo2021automated  model was finetuned
only on the parsed plain-text output of PDF articles wherein the plain text
was scraped from the PDF which results in partial information loss. Thus, in
this work, we modify our previous workflow deciding to tune one model on LaTeX
source files as input data, given the increasing impetus of authors also
submitting the LaTeX source code; and a second model following our previous
work on plain-text scraped from PDF articles.

  1. 1.

LaTeX pre-processed corpus. To obtain the LaTeX sources, we queried arXiv
based on the paper titles from the 5361 articles of our original corpus
leveraged to developed orkg-TDM kabongo2021automated . Resultingly, LaTeX
sources for roughly 79% of the papers from the training and test datasets in
our original work were obtained. Thus the training set size was reduced from
3,753 papers in the original work to 2,951 papers in this work with
corresponding LaTeX sources. Similarly, the test set size was reduced from
1,608 papers in the original work to 1,258 papers in this work for which LaTeX
sources could be obtained. Thus the total size of our corpus reduced from
5,361 papers to 4,209 papers. Once the LaTeX sources were respectively
gathered for the training and test sets, the data had to undergo one
additional step of preprocessing. With the help of
pandoc161616https://pandoc.org/, latex format files were converted into the
XML TEI171717https://tei-c.org/ markup format files. This is the required
input for the heuristics-based script that produces the DocTAET feature. Thus
the resulting XML files were then fed as input to the DocTAET feature
extraction script. The pipeline to reproduce this process is released in our
code repository 181818https://github.com/Kabongosalomon/task-dataset-metric-
nli-extraction/tree/main/data.

  2. 2.

PDF pre-processed corpus. For the 4,209 papers with LaTeX sources, we created
an equivalent corpus but this time using the PDF files. This is the second
experimental corpus variant of this work. To convert PDF to plain text,
following along the lines of our previous work kabongo2021automated , the
GROBID parser GROBID  was applied. The resulting files in XML TEI markup
format were then fed into the DocTAET feature extraction script similar to the
LaTeX document processing workflow.

####  4.2.2 Task-Dataset-Metric Annotations

Since the two corpus variants used in the empirical investigations in this
work are a subset of the corpus in our earlier work kabongo2021automated , the
4,209 papers in our present corpus, regardless of the variant, i.e. LaTeX or
PDF, retained their originally obtained Task-Dataset-Metric labels via distant
labeling supervision on the PwC knowledge base (KB).

###  4.3 Task Dataset Statistics

Our overall corpus statistics are shown in Table 1. The column “Ours-Prior”
reports the dataset statistics of our prior work kabongo2021automated  for
comparison purposes. The column “Ours-Present” reports the dataset statistics
of the subset corpus used in the empirical investigations reported in this
paper. The corpus size is the same for both the LaTeX and PDF corpus variants.
In all, our corpus contains 4,208 papers split as 2,946 as training data and
1,262 papers as test data. There were 1,724 unique TDM-triples overall. Note
that since the test labels were a subset of the training labels, the unique
labels overall can be considered as those in the training data. Table 1 also
shows the distinct Tasks, Datasets, Metrics in the last three rows. Our corpus
contains 262 Tasks defined on 853 Datasets and evaluated by 528 Metrics. This
is significantly larger than the original corpus which had 18 Tasks defined on
44 Datasets and evaluated by 31 Metrics.

| Ours-Prior | Ours-Present | Original
---|---|---|---
| Train | Test | Train | Test | Train | Test
Papers | 3,753 | 1,608 | 2,946 | 1,262 | 170 | 167
“unknown” annotations | 922 | 380 | 2,359 | 992 | 46 | 45
Total TDM-triples | 11,724 | 5,060 | 9,614 | 4,096 | 327 | 294
Avg. number of TDM-triples per paper | 4.1 | 4.1 | 4.3 | 4.2 | 2.64 | 2.41
Distinct TDM-triples | 1,806 | 1,548 | 1,668 | 1,377 | 78 | 78
Distinct Tasks | 288 | 252 | 262 | 228 | 18 | 18
Distinct Datasets | 908 | 798 | 853 | 714 | 44 | 44
Distinct Metrics | 550 | 469 | 528 | 434 | 31 | 31
|  |  |  |  |  |
Table 1: Ours-Prior kabongo2021automated  vs. Ours-Present vs. the original
science result extractor hou2019identification  corpora statistics. The
“unknown” labels were assigned to papers with no TDM-triples after the label
filtering stage.

####  4.3.1 DocTAET Context Feature Statistics

Figure 1 shows in detail the variance of the DocTAET Context Feature over
three datasets proposed for Leaderboard extraction as Task-Dataset-Metric
triples: 1) 1(a) for the dataset from the pioneering science result extractor
system hou2019identification ; 2) 1(b) for the dataset from our prior ORKG-TDM
work kabongo2021automated ; 3) 1(c) and 1(d) for the dataset in our present
paper from the Grobid and LaTeX workflows, respectively (column “Ours-Present”
in Table 1)).

(a) DocTAET feature length in the original science result extractor corpus
hou2019identification  had a max, min, and mean length of 546, 81 and 309.45,
respectively

(b) DocTAET feature length in the dataset in our prior work
kabongo2021automated  had a max, min, and mean length of 2161, 5 and 378.88,
respectively

(c) DocTAET feature length in the dataset from the Grobid workflow in our
present paper has a max, min, and mean length of 2686, 101 and 513.37,
respectively

(d) DocTAET feature length in the dataset from the LaTeX workflow in our
present paper has a max, min, and mean length of 7374, 100 and 685.25,
respectively

Figure 1: DocTAET feature length of papers in the original science result
extractor dataset hou2019identification  1(a), the dataset used in our prior
ORKG-TDM experiements kabongo2021automated  1(a), the dataset from the Grobid
workflow in our present work 1(c) and the dataset from the LaTeX workflow in
our present work 1(d).

Both the prior datasets, i.e., the original science result extractor dataset
hou2019identification  and the ORKG-TDM dataset kabongo2021automated ,
followed the Grobid processing workflow and reported roughly the same average
length of the DocTAET feature. This reflects the consistency preserved in the
method of computing the DocTAET feature of between 300 to 400 tokens. Note the
ORKG-TDM corpus was significantly larger than the original science result
extractor corpus; hence their DocTAET feature length statistics do not match
exactly.

In our present paper, as reported earlier, we use a subset of papers from the
ORKG-TDM dataset for which the corresponding LaTeX sources could be obtained
to ensure similar experimental settings between the Grobid and LaTeX
processing workflows. This is why the DocTAET feature length statistics
between the ORKG-TDM dataset (1(b)) and our present dataset in the Grobid
processing workflow (1(c)) do not match exactly. Still, we see that they are
roughly in similar ranges. Finally, of particular interest is observing the
DocTAET feature length statistics that could be obtained from the LaTeX
processing workflow introduced in this work (1(d)). Since from the LaTeX
processing workflow cleaner plain-text output could be obtained, the
corresponding DocTAET feature lengths in many of the papers were longer than
all the rest of the datasets considered, which operated in the Grobid
processing workflow over PDFs.

##  5 The ORKG-Leaderboards System

This section depicts the overall end-to-end orkg-Leaderboards, including
details on the deep learning models used in our Natural Language Inference
(NLI) task formulation.

###  5.1 Workflow

Figure 2: The orkg-Leaderboards end-to-end system workflow in the context of
the Open Research Knowledge Graph (ORKG) digital library https://orkg.org/

The overall orkg-Leaderboards workflow as depicted in Figure 2 includes the
following steps:

  1. 1.

A user provides the article input as either the main ‘.tex’ file or a PDF
file.

  2. 2.

If the input is provided as a ‘.tex’ file, the pandoc script is applied to
convert the LaTeX to the corresponding XML TEI marked-up format.

  3. 3.

Alternatively, if the input is provided as a PDF file, the Grobid parser is
applied to obtain the corresponding scraped plain text in the XML xxxx marked-
up format.

  4. 4.

Once the XML xxx marked-up files are obtained, the DocTAET feature extraction
script is applied to obtain the paper context representations.

  5. 5.

Furthermore, if in the training phase, the collection of papers in the
training set is assigned their respective $true$ Task-Dataset-Metric labels
and a random set of ”False” Task-Dataset-Metric labels.

  6. 6.

Otherwise, if in the test phase, the query paper is assigned all the Task-
Dataset-Metric inference targets as candidate labels.

  7. 7.

Finally, on the one hand, for the training phase, for each of the input file
formats i.e., ‘.tex’ or PDF, an optimal inference model is trained by testing
four transformer model variants, viz. BERT, SciBERT, XLNet, and BigBird.

  8. 8.

On the hand, for the test phase, depending on the input file format i.e.,
‘.tex’ or PDF, the corresponding trained optimal model is applied to the query
instance.

  9. 9.

Finally, from the test phase, the predicted Task-Dataset-Metric tuples output
are integrated in the ORKG.

###  5.2 Leaderboards Natural Language Inference (NLI)

To support Leaderboard inference hou2019identification , we employ deep
transfer learning modeling architectures that rely on a recently popularized
neural architecture – the transformer vaswani2017attention . Transformers are
arguably the most important architecture for natural language processing (NLP)
today since they have shown and continue to show impressive results in several
NLP tasks devlin2018bert . Owing to the self-attention mechanism in these
models, they can be fine-tuned on many downstream tasks. These models have
thus crucially popularized the transfer learning paradigm in NLP. We
investigate three transformer-based model variants for leaderboard extraction
in a Natural Language Inference configuration.

Natural language inference (NLI), generally, is the task of determining
whether a “hypothesis” is true (entailment), false (contradiction), or
undetermined (neutral) given a “premise” ref_paperswithcode_nli . For
leaderboard extraction, the slightly adapted NLI task is to determine that the
(task, dataset, metric) “hypothesis” is true (entailed) or false (not
entailed) for a paper given the “premise” as the DocTAET context feature
representation of the paper.

Currently, there exist several transformer-based models. In our experiments,
we investigated four core models: three variants of BERT, i.e., the vanilla
BERT devlin2018bert , scientific BERT (SciBERT) beltagy2019scibert , and
BigBird zaheer2020big . We also tried a different type of transformer model
than BERT called XLNet yang2019xlnet , which employs Transformer-XL as the
backbone model. Next, we briefly describe the four variants we use.

##### BERT Models

BERT (i.e., Bidirectional Encoder Representations from Transformers), is a
bidirectional autoencoder (AE) language model. As a pre-trained language
representation built on the deep neural technology of transformers, it
provides NLP practitioners with high-quality language features from text data
simply out of the box and thus improves performance on many NLP tasks. These
models return contextualized word embeddings that can be directly employed as
features for downstream tasks jiang2020improving .

The first BERT model we employ is BERTbase (12 layers, 12 attention heads, and
110 million parameters), which was pre-trained on billions of words from the
BooksCorpus (800M words) and the English Wikipedia (2,500M words).

The second BERT model we employ is the pre-trained scientific BERT called
SciBERT beltagy2019scibert . SciBERT was pretrained on a large corpus of
scientific text. In particular, the pre-training corpus is a random sample of
1.14M papers from Semantic Scholar191919https://semanticscholar.org consisting
of full texts of 18% of the papers from the computer science domain and 82%
from the broad biomedical field. We used their uncased variants for both
BERTbase and SciBERT.

##### XLNet

XLNet is an autoregressive (AR) language model yang2019xlnet  that enables
learning bidirectional contexts using Permutation Language Modeling. This is
unlike BERT’s Masked Language Modeling strategy. Thus in PLM, all tokens are
predicted but in random order, whereas in MLM, only the masked (15%) tokens
are predicted. This is also in contrast to the traditional language models,
where all tokens are predicted in sequential order instead of randomly. Random
order prediction helps the model to learn bidirectional relationships and,
therefore, better handle dependencies and relations between words. In
addition, it uses Transformer XL dai2019transformer  as the base architecture,
which models long contexts, unlike the BERT models with contexts limited to
512 tokens. Since only cased models are available for XLNet, we used the cased
XLNetbase (12 layers, 12 attention heads, and 110 million parameters).

##### BigBird

BigBird is a sparse-attention-based transformer that extends Transformer based
models, such as BERT, to much longer sequences. Moreover, BigBird comes along
with a theoretical understanding of the capabilities of a complete transformer
that the sparse model can handle zaheer2020big . BigBird takes inspiration
from graph sparsification methods by relaxing the need for the attention to
fully attend to all the input tokens. Formally the model first builds a set of
$g$ global tokens attending on all parts of the sequence, then all tokens
attend to a set of $w$ local neighboring tokens, and finally, all tokens
attend to a set of $r$ random tokens. The empirical configuration explained in
the last paragraph leads to a high-performing attention mechanism scaling to
much longer sequence lengths (8x) zaheer2020big .

##  6 ORKG-Leaderboards System Experiments

###  6.1 Experimental Setup

##### Parameter Tuning

We use the Hugging Transformer libraries
(202020https://github.com/huggingface/transformers) with their BERT variants
and XLNet implementations. In addition to the standard fine-tuned setup for
NLI, the transformer models were trained with a learning rate of $1e^{-5}$ for
14 epochs; and used the $AdamW$ optimizer with a weight decay of 0 for bias,
gamma, beta and 0.01 for the others. Our models’ hyperparameters details can
be found in our code repository online at
212121https://github.com/Kabongosalomon/task-dataset-metric-nli-
extraction/blob/main/train_tdm.py.

In addition, we introduced a task-specific parameter that was crucial in
obtaining optimal task performance from the models. It was the number of
$false$ triples per paper. This parameter controls the discriminatory ability
of the model. The original science result extractor system
hou2019identification  considered
n-t$\textit{false}instancesforeachpaper,where$n$wasthedistinctsetoftriplesoveralland$t$wasthenumberof$true$\textit{leaderboard}triplesperpaper.Thisapproachwouldnotgeneralizetoourlargercorpuswithover1,724distincttriples.Inotherwords,consideringthateachpaperhadonaverage4\textit{true}triples,itwouldhavealargersetof\textit{false}tripleswhichwouldstronglybiastheclassifierlearningtowardonly\textit{false}inferences.Thus,wetunedthisparameterinarangeofvaluesintheset\\{10,50,100\\}whichateachexperimentrunwasfixedforallpapers.\par
Finally,weimposedanartificialtrimmingoftheDocTAETfeaturetoaccountforBERTandSciBERT^{\prime}smaximumtokenlengthof512.Forthis,thetokenlengthsoftheexperimentalsetupandtableinfowereinitiallytruncatedtoroughly150tokens,afterwhichtheDocTAETfeatureistrimmedattherightto512tokens.Whereas,XLNetandBigBirdarespecificallydesignedtohandlelongercontextsofundefinedlengths.Nevertheless,tooptimizefortrainingspeed,weincorporatedacontextlengthof2000tokens.\par$

##### Evaluation

Similar to our prior work kabongo2021automated , all experiments are performed
via two-fold cross-validation. Within the two-fold experimental settings, we
report macro- and micro-averaged precision, recall, and F1 scores for our
Leaderboard extraction task on the test dataset. The macro scores capture the
averaged class-level task evaluations, whereas the micro scores represent
fine-grained instance-level task evaluations. Further, the macro and micro
evaluation metrics for the overall task have two evaluation settings: 1)
considers papers with Task-Dataset-Metric and papers with “unknown” in the
metric computations; and 2) only papers with Task-Dataset-Metric are
considered while the papers with “unknown” are excluded. In general, we focus
on the model performances in the first evaluation setting as it directly
emulates the real-world application setting that includes papers that do not
report empirical research and therefore for which the Leaderboard model does
not apply. In the second setting, however, the reader still can gain insights
into the model performances when given only papers with Leaderboards.

###  6.2 Experimental Results

In this section, we discuss new experimental findings shown in Table 2, Table
2 v2, Table 3 v1, and Table 3 v2 with respect to four research questions
elicited as RQ1, RQ2, RQ3, and RQ4 respectively.

#### RQ1: Which is the best model in the real-world setting when considering a
dataset of both kinds of papers: those with Leaderboards and those without
Leaderboards therefore labeled as “Unknown”?

For these results, we refer the reader to the first four results’ rows in both
Table 2 and Table 2 v2, respectively. Note, Table 2 reports results from the
Grobid processing workflow and Table 2 v2 reports results from the LaTeX
processing workflow. In both cases, it can be observed that orkg-
LeaderboardsXLNet is the best transformer model for the Leaderboard inference
task in terms of micro-F1. In the case of the Grobid processing workflow, the
best micro-F1 from this model is 94.8%. Whereas in the case of LaTeX
processing workflow, the best micro-F1 from orkg-LeaderboardsXLNet is 93.0%.
Note in selecting the best model we prefer the micro evaluations since they
reflect the fine-grained discriminative ability of the models at the instance
level. The macro scores are seen simply as supplementary measures in this
regard to observing the performance of the models at the class level.

#### RQ2: How do the models in two processing workflows, i.e. Grobid producing
plaintext with some noise and the clean plaintext from LaTeX, compare, both in
general and specifically for the best orkg-LeaderboardsXLNet model?

The model trained on the plain-text obtained from LaTeX contrary to our
intuition shows a lower performance compared to the one trained on the noisy
Grobid produced plain-text. One possible cause, maybe related to the context
length as the LaTeX produced dataset has an average length of $685.25$
compared to $512.37$ for the Grobid produced data, as shown in 1(d) and 1(c).
In this case, we hypothesize that for the LaTeX processing workflow to be
implemented with the most effective model, experiments with a much larger
dataset are warranted. There may be one of two outcomes: 1) the model from the
LaTeX workflow still performs worse than the model from the Grobid workflow in
which case we can conclude that longer contexts regardless of whether they are
from a clean source or noisy source are difficult to generalize from, or 2)
the model from the LaTeX workflow indeed begins to outperform the model from
the Grobid workflow in which case we can safely conclude that for the
transformer models to generalize on longer contexts a much larger training
dataset is needed. We relegate these further detailed experiments to future
work.

#### RQ3: Which insights can be gleaned from the BERT and SciBERT models
operating on shorter context lengths of 512 tokens versus the more advanced
models, viz. XLNet and BigBird, operating on longer context lengths of 2000
tokens?

We observed that BERT and SciBERT models show lower performance compared to
the XLNet transformer model operating on 2000 tokens. This we hypothesized as
expected behavior since the longer contextual information can capture richer
signals for the model to learn from, which is highly likely to be lost when
imposing the 512 tokens limit. Contrary to this intuition, however, the
BigBird model with the longer context is not able to outperform BERT and
SciBERT. We suspect the specific attention mechanism in the BigBird model
zaheer2020big  needs further examination over a much larger dataset to
conclude that it is ineffective for Task-Dataset-Metric extraction task
compared to other transformer-based models. Ma-P111Macro Precision Ma-R Ma-F1
Mi-P222Micro Precision. Mi-R Mi-F1 Average Evaluation Accross 2-fold orkg-
LeaderboardsBERT 93.2 95.7 93.5 95.4 93.9 94.7 orkg-LeaderboardsSciBERT 92.6
94.3 92.2 95.4 91.1 93.2 orkg-LeaderboardsXLNet 93.1 96.4 93.7 95.1 94.6 94.8
orkg-LeaderboardsBigBird 93.2 94.9 93.0 95.7 92.4 94.0 Average Evaluation
Across 2-fold (without ”Unknown” annotation) orkg-LeaderboardsBERT 91.3 94.4
91.8 94.8 93.9 94.3 orkg-LeaderboardsSciBERT 90.5 92.5 90.3 94.8 90.6 92.7
orkg-LeaderboardsXLNet 91.3 95.0 92.0 94.3 93.5 93.9 orkg-LeaderboardsBigBird
91.5 93.3 91.3 95.2 92.2 93.6 Table 2: BERT512, SciBERT512, XLNet2000 and
BigBird2000 results, trained on the subset of the dataset released by
kabongo2021automated  from the Grobid workflow Ma-P111Macro Precision Ma-R
Ma-F1 Mi-P222Micro Precision. Mi-R Mi-F1 Average Evaluation Across 2-fold
orkg-LeaderboardsBERT 93.5 94.2 92.8 96.0 90.0 92.9 orkg-LeaderboardsSciBERT
91.7 93.9 91.6 94.6 88.6 91.5 orkg-LeaderboardsXLNet 91.9 94.4 92.0 94.9 91.2
93.0 orkg-LeaderboardsBigBird 90.7 91.6 89.7 94.6 87.2 90.7 Average Evaluation
Accross 2-fold (without ”Unknown” annotation) orkg-LeaderboardsBERT 91.2 92.3
90.6 95.4 88.0 91.5 orkg-LeaderboardsSciBERT 89.4 91.7 89.2 93.7 86.0 89.7
orkg-LeaderboardsXLNet 89.5 92.4 89.8 94.2 89.4 91.7 orkg-LeaderboardsBigBird
87.5 88.7 86.6 93.6 85.3 89.3 Table 2 v2: BERT512, SciBERT512, XLNet2000 and
BigBird2000 results, based on DocTEAT from LaTeX code source. Entity Macro
Micro $\mathrm{P}$ $\mathrm{R}$ $\mathrm{F}_{1}$ $\mathrm{P}$ $\mathrm{R}$
$\mathrm{F}_{1}$ TDM 93.1 96.4 93.7 95.1 94.6 94.8 Task 94.3 97.2 95.0 96.8
95.9 96.4 Dataset 93.8 96.7 94.4 96.2 95.4 95.8 Metric 93.7 96.9 94.4 96.0
95.3 95.6 Table 3 v1: Performance of our best model, i.e. orkg-
LeaderboardsXLNet, for Task, Dataset, and Metric concept extraction of the
leaderboard for the grobid workflow Entity Macro Micro $\mathrm{P}$
$\mathrm{R}$ $\mathrm{F}_{1}$ $\mathrm{P}$ $\mathrm{R}$ $\mathrm{F}_{1}$ TDM
91.9 94.4 92.0 94.9 91.2 93.0 Task 94.3 97.2 95.0 96.8 95.9 96.4 Dataset 93.8
96.7 94.4 96.2 95.4 95.8 Metric 93.7 96.9 94.4 96.0 95.3 95.6 Table 3 v2:
Performance of our best model, i.e. orkg-LeaderboardsXLNet, for Task, Dataset,
and Metric concept extraction of the leaderboard for the latex workflow

#### RQ4: Which of the three Leaderboard Task-Dataset-Metric concepts are easy
or challenging to extract?

As a fine-grained examination of our best model, i.e. orkg-LeaderboardsXLNet,
we examined its performance for extracting each of three concepts
$(Task,Dataset,Metric)$ separately. These results are shown in Table 3 v1 and
Table 2 v2. From the results, we observe that Task is the easiest concept to
extract, followed by Metric, and then Dataset. We ascribe the low performance
for extracting the Dataset concept due to the variability in its naming seen
across papers even when referring to the same real-world entity. For example,
the real-world dataset entity ‘CIFAR-10’ is labeled as ‘CIFAR-10, 4000 Labels’
in some papers and ‘CIFAR-10, 250 Labels’ in others. This phenomenon is less
prevalent for Task and the Metric concepts. For example, the Task ‘Question
Answering’ is rarely referenced differently across papers addressing the task.
Similarly, for Metric, ‘accuracy’ as an example, has very few variations.
Figure 3: A contrastive view of Task-Dataset-Metric information in the
traditional PDF format of publishing as non-machine-actionable data (on the
left) versus as machine-actionable data with Task-Dataset-Metric annotations
obtained from orkg-Leaderboards and integrated in the next-generation
scholarly knowledge platform as the ORKG Benchmarks view (on the right).

##  7 Integrating ORKG-Leaderboards in the Open Research Knowledge Graph

In this era of the publications deluge worldwide jinha2010article ;
chiarelli2019accelerating ; stm , researchers are faced with a critical
dilemma: How to stay on track with the past and the current rapid-evolving
research progress? With this work, our main aim is to propose a solution to
this problem. And with the orkg-Leaderboards software, we have concretely made
advances toward our aim in the domain of empirical AI research. Furthermore,
with the software integrated into the next-generation digitalized publishing
platform, viz. https://orkg.org/, the machine-actionable Task-Dataset-Metric
data represented as a Knowledge Graph with the help of the Semantic Web’s RDF
language makes the information skimmable for the scientific community. This is
achieved via the dynamic Frontend views of the ORKG Benchmarks feature
https://orkg.org/benchmarks. This is illustrated via Figure 3. On the left
side of Figure 3 is shown the traditional PDF-based paper format. Highlighted
within the view are the Task, Dataset, and Metric phrases. As evident, the
phrases are mentioned in several places in the paper. Thus in this traditional
model of publishing via non-machine-actionable PDFs, a researcher interested
in this critical information would need to scan the full paper content. They
are then faced with the intense cognitive burden of repeating such a task over
a large collection of articles. On the right side of the Figure 3 is presented
a dynamic ORKG Frontend view of the same information, however over machine-
actionable RDF semantically represented information of the Task, Dataset, and
Metric elements. To generate such a view, the orkg-Leaderboard software would
simply be applied on a large collection of articles either in LaTeX or PDF
format, and the resulting Task-Dataset-Metric tuples uploaded in the ORKG.
Note, however, orkg-Leaderboard does not attempt extraction of the Score
element. We observed from some preliminary experiments that the Score element
poses a particularly hard extraction target. This is owing to the fact that
the underlying contextual data supporting Score extraction is especially
noisy–clean table data extraction from PDFs are a challenging problem in the
research community that would need to be addressed first to develop promising
Score extractors. Nevertheless, in the context of this missing data in the
ORKG Benchmarks pages, its human-in-the-loop curation model is relied on. In
such a setting, respective article authors with their Task-Dataset-Metric
model information being automatically extracted to the KG can simply edit
their corresponding model scores in the graph. Thus as concretely shown on the
right screen of Figure 3, empirical results are made skimmable and easy to
browse for researchers interested in gaining an overview of empirical research
progress via a ranked list of papers proposing models and a performance
progress trend chart computed over time. Although the experiments of our study
targeted empirical AI research, we are confident, that the approach is
transferable to similar scholarly knowledge extraction tasks in other domains.
For example in Chemistry or Material Sciences, experimentally observed
properties of substances or materials under certain conditions could be
obtained from various papers.

##  8 Conclusion and Future Work

In this work we experimented with the empirical construction of Leaderboards,
using four recent transformer-based models (BERT, SciBERT, XLNet, BigBird)
that have achieved state-of-the-art performance in several tasks and domains
in the literature. Leveraging the two main streams of information acquisition
used in scholarly communication i.e (Pdf, LaTeX), our work published two
models to accurately extract Task, Dataset, and Metric entities from an
empirical AI research publication. Therefore as a next step, we will extend
the current triples (task, dataset, metric) model with additional concepts
that are suitable candidates for a Leaderboard such as score or code URLs,
etc. We also envision the task-dataset-metric extraction approach to be
transferable to other domains (such as materials science, engineering
simulations, etc.). Our ultimate target is to create a comprehensive
structured knowledge graph tracking scientific progress in various scientific
domains, which can be leveraged for novel machine-assistance measures in
scholarly communication, such as question answering, faceted exploration, and
contribution correlation tracing.

#### Acknowledgments

This work was co-funded by the Federal Ministry of Education and Research
(BMBF) of Germany for the project LeibnizKILabor (grant no. 01DD20003), BMBF
project SCINEXT (GA ID: 01lS22070), NFDI4DataScience (grant no. 460234259) and
by the European Research Council for the project ScienceGRAPH (Grant agreement
ID: 819536).

## References

  *   * (1) Parra Escartín, C., Reijers, W., Lynn, T., Moorkens, J., Way, A., Liu, C.-H.: Ethical considerations in nlp shared tasks. (2017). Association for Computational Linguistics
  * (2) Nissim, M., Abzianidze, L., Evang, K., van der Goot, R., Haagsma, H., Plank, B., Wieling, M.: Last words: Sharing is caring: The future of shared tasks. Computational Linguistics 43(4), 897–904 (2017)
  * (3) Kim, J.-D., Pyysalo, S.: In: Dubitzky, W., Wolkenhauer, O., Cho, K.-H., Yokota, H. (eds.) BioNLP Shared Task, pp. 138–141. Springer, New York, NY (2013). https://doi.org/10.1007/978-1-4419-9863-7_138. https://doi.org/10.1007/978-1-4419-9863-7_138
  * (4) Jinha, A.E.: Article 50 million: an estimate of the number of scholarly articles in existence. Learned Publishing 23(3), 258–263 (2010)
  * (5) Chiarelli, A., Johnson, R., Richens, E., Pinfield, S.: Accelerating scholarly communication: The transformative role of preprints (2019)
  * (6) paperswithcode.com. https://paperswithcode.com/. Accessed: 2021-04-26
  * (7) NLP-progress. http://nlpprogress.com/. Accessed: 2021-04-26
  * (8) AI metrics. https://www.eff.org/ai/metrics. Accessed: 2021-04-26
  * (9) SQuAD Explorer. https://rajpurkar.github.io/SQuAD-explorer/. Accessed: 2021-04-26
  * (10) Reddit Sota. https://github.com/RedditSota/state-of-the-art-result-for-machine-learning-problems. Accessed: 2021-04-26
  * (11) Wilkinson, M.D., Dumontier, M., Aalbersberg, I.J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L.B., Bourne, P.E., et al.: The fair guiding principles for scientific data management and stewardship. Scientific data 3 (2016)
  * (12) Jacobsen, A., de Miranda Azevedo, R., Juty, N., Batista, D., Coles, S., Cornet, R., Courtot, M., Crosas, M., Dumontier, M., Evelo, C.T., et al.: FAIR Principles: Interpretations and Implementation Considerations. MIT Press (2019)
  * (13) Auer, S., Oelen, A., Haris, M., Stocker, M., D’Souza, J., Farfar, K.E., Vogt, L., Prinz, M., Wiens, V., Jaradeh, M.Y.: Improving access to scientific literature with knowledge graphs. Bibliothek Forschung und Praxis 44(3), 516–529 (2020)
  * (14) Escartín, C.P., Lynn, T., Moorkens, J., Dunne, J.: Towards transparency in nlp shared tasks. arXiv preprint arXiv:2105.05020 (2021)
  * (15) Kabongo, S., D’Souza, J., Auer, S.: Automated mining of leaderboards for empirical ai research. In: International Conference on Asian Digital Libraries, pp. 453–470 (2021). Springer
  * (16) Zaheer, M., Guruganesh, G., Dubey, K.A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al.: Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems 33, 17283–17297 (2020)
  * (17) D’Souza, J., Auer, S.: Computer science named entity recognition in the open research knowledge graph. In: From Born-Physical to Born-Virtual: Augmenting Intelligence in Digital Libraries: 24th International Conference on Asian Digital Libraries, ICADL 2022, Hanoi, Vietnam, November 30–December 2, 2022, Proceedings, pp. 35–45 (2022). Springer
  * (18) Gupta, S., Manning, C.: Analyzing the dynamics of research by extracting key aspects of scientific papers. In: Proceedings of 5th International Joint Conference on Natural Language Processing, pp. 1–9. Asian Federation of Natural Language Processing, Chiang Mai, Thailand (2011). https://aclanthology.org/I11-1001
  * (19) LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11), 2278–2324 (1998)
  * (20) Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. Advances in neural information processing systems 26 (2013)
  * (21) Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318 (2002)
  * (22) Sasaki, Y., et al.: The truth of the f-measure. Teach tutor mater 1(5), 1–5 (2007)
  * (23) Voorhees, E.M., et al.: The trec-8 question answering track report. In: Trec, vol. 99, pp. 77–82 (1999)
  * (24) Anteghini, M., D’Souza, J., dos Santos, V.A., Auer, S.: Easy semantification of bioassays. In: International Conference of the Italian Association for Artificial Intelligence, pp. 198–212 (2022). Springer
  * (25) Kononova, O., Huo, H., He, T., Rong, Z., Botari, T., Sun, W., Tshitoyan, V., Ceder, G.: Text-mined dataset of inorganic materials synthesis recipes. Scientific data 6(1), 1–11 (2019)
  * (26) Kulkarni, C., Xu, W., Ritter, A., Machiraju, R.: An annotated corpus for machine reading of instructions in wet lab protocols. In: NAACL: HLT, Volume 2 (Short Papers), New Orleans, Louisiana, pp. 97–106 (2018). https://doi.org/10.18653/v1/N18-2016
  * (27) Mysore, S., Jensen, Z., Kim, E., Huang, K., Chang, H.-S., Strubell, E., Flanigan, J., McCallum, A., Olivetti, E.: The materials science procedural text corpus: Annotating materials synthesis procedures with shallow semantic structures. In: Proceedings of the 13th Linguistic Annotation Workshop, pp. 56–64 (2019)
  * (28) Kuniyoshi, F., Makino, K., Ozawa, J., Miwa, M.: Annotating and extracting synthesis process of all-solid-state batteries from scientific literature. In: LREC, pp. 1941–1950 (2020)
  * (29) Handschuh, S., QasemiZadeh, B.: The acl rd-tec: a dataset for benchmarking terminology extraction and classification in computational linguistics. In: COLING 2014: 4th International Workshop on Computational Terminology (2014)
  * (30) Augenstein, I., Das, M., Riedel, S., Vikraman, L., McCallum, A.: Semeval 2017 task 10: Scienceie - extracting keyphrases and relations from scientific publications. In: SemEval@ACL (2017)
  * (31) Luan, Y., He, L., Ostendorf, M., Hajishirzi, H.: Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In: EMNLP (2018)
  * (32) D’Souza, J., Hoppe, A., Brack, A., Jaradeh, M.Y., Auer, S., Ewerth, R.: The stem-ecr dataset: Grounding scientific entity references in stem scholarly content to authoritative encyclopedic and lexicographic sources. In: LREC, Marseille, France, pp. 2192–2203 (2020)
  * (33) Hou, Y., Jochim, C., Gleize, M., Bonin, F., Ganguly, D.: Identification of tasks, datasets, evaluation metrics, and numeric scores for scientific leaderboards construction. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5203–5213. Association for Computational Linguistics, Florence, Italy (2019). https://doi.org/10.18653/v1/P19-1513. https://aclanthology.org/P19-1513
  * (34) Jain, S., van Zuylen, M., Hajishirzi, H., Beltagy, I.: Scirex: A challenge dataset for document-level information extraction. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7506–7516 (2020)
  * (35) Mondal, I., Hou, Y., Jochim, C.: End-to-end construction of nlp knowledge graph. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1885–1895 (2021)
  * (36) GROBID. GitHub (2008–2022)
  * (37) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)
  * (38) Kenton, J.D.M.-W.C., Toutanova, L.K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of naacL-HLT, pp. 4171–4186 (2019)
  * (39) Natural Language Inference. https://paperswithcode.com/task/natural-language-inference. Accessed: 22 April 2021
  * (40) Beltagy, I., Lo, K., Cohan, A.: SciBERT: A pretrained language model for scientific text. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3615–3620. Association for Computational Linguistics, Hong Kong, China (2019). https://doi.org/10.18653/v1/D19-1371. https://aclanthology.org/D19-1371
  * (41) Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V.: Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems 32 (2019)
  * (42) Jiang, M., D’Souza, J., Auer, S., Downie, J.S.: Improving scholarly knowledge representation: Evaluating bert-based models for scientific relation classification. In: International Conference on Asian Digital Libraries, pp. 3–19 (2020). Springer
  * (43) Dai, Z., Yang, Z., Yang, Y., Carbonell, J.G., Le, Q., Salakhutdinov, R.: Transformer-xl: Attentive language models beyond a fixed-length context. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2978–2988 (2019)
  * (44) Ware, M., Mabe, M.: The stm report: An overview of scientific and scholarly journal publishing (2015)


# Dissociative recombination of CF+

Joshua Forer1, Jeoffrey Boffelli2, Mehdi Ayouz3, Dávid Hvizdoš4, Viatcheslav
Kokoouline5, Ioan F. Schneider2, Chris H. Greene4 1Columbia Astrophysics
Laboratory, Columbia University, New York, New York 10027, USA
2Laboratoire Ondes et Milieux Complexes, CNRS, Universiteé le Havre Normandie,
53 rue Prony, Le Havre, 76058, France.
3Université Paris-Saclay, CentraleSupélec, Laboratoire de Génie des Procédés
et Matériaux, 91190, Gif-sur-Yvette, France.
4Department of Physics and Astronomy and Purdue Quantum Science and
Engineering Institute, Purdue University, West Lafayette, Indiana 47907, USA
5Department of Physics, University of Central Florida, 32816, Florida, USA

(August 28, 2024)

###### Abstract

This work presents our theoretical study of the dissociative recombination
(DR) of the closed-shell diatomic system CF+ based on an approach recently
applied to the CH+ molecule. Our extended treatment uses the UK R-matrix
theory and the multichannel quantum-defect theory procedure to uniformly
resolve the direct and indirect mechanisms of DR while bypassing explicit
dissociative state and electronic coupling calculations. The theoretical
results exhibit good overall agreement with previous experimental
measurements. At lower scattering energies, good agreement is found only if
the rotational structure of the ion is included and the theoretical cross
sections are averaged over initial rotational levels corresponding to the
temperature at which the experimental measurements were made. At higher
scattering energies, our rotationally resolved results are very similar to
those obtained without including the ion’s rotational structure.

Relevance to fluorine-containing molecular plasmas. Fluorine-containing
molecular species play a distinct role, compared to other species, in the
chemistry of gases that are subjected to ultra-violet (UV) radiation, such as
in the interstellar medium (ISM)[1, 2], owing to a combination of the high
ionization potential of atomic fluorine and the large dissociation energies of
HF and HF+. The ubiquitous hydrogen atoms shield atomic fluorine in
interstellar clouds from ionizing UV radiation because the ionization
potential of F is higher than that of H, which results in a mostly neutral
population of atomic fluorine. Therefore, the fluorine chemical network in the
ISM is quite simple[1], when compared to other chemical networks involving
significant populations of ionized atoms, and can be used to benchmark
astrophysical models of interstellar clouds. Atomic fluorine can collide with
H2 to form HF, which then reacts with C+ to produce CF+. The CF+ ion, in turn,
can be destroyed by dissociative recombination (DR) with free electrons,
producing atomic fluorine and closing the network loop:

$\begin{gathered}\ce{F+H2->FH+H},\\\ \ce{HF+C+->CF++H},\\\
\ce{CF++e-->C+F}.\end{gathered}$ (1)

The above reactions are the major steps in the fluorine cycle in the ISM,
which makes HF and CF+ the first and second most abundant fluorine-containing
molecules in that environment, where their relative abundances are determined
by the rate coefficients of the two last reactions in (1). To understand
fluorine chemistry in the ISM and the above-mentioned benchmarking of the
astrophysical models by comparing them with observations, it is essential to
have accurate rate coefficients for the above reactions.

One astrophysical model [1] of the fluorine network predicted CF+ abundances
that are more than an order of magnitude larger than the abundances inferred
from observations [3, 1]. It is likely that one or both rate coefficients for
the two last reactions in (1) used in the model are not accurate. The rate
coefficient for the second reaction in (1) was not experimentally inferred.
Different theoretical values, obtained by a modified Langevin method [1] and a
quasi-classical trajectory method [4] differ from each other by about an order
of magnitude. The rate coefficient for the third reaction (DR) in (1) used in
the model, $5.2\times 10^{-8}(T/300\leavevmode\nobreak\ \mathrm{K})^{-0.8}$
cm3/s, was derived from the ASTRID and CRYRING storage-ring experiments of
Novotny  _et al._ [5]. In those experiments, the rovibrational temperature of
the CF+ ions were very high, around 6000 K, according to the study [5].
Therefore, the DR rate coefficient derived from the experiments may not be
appropriate to model the cold (30–300 K) interstellar environment where the
ions are all expected to be in one of the lowest rovibronic (rotational,
vibrational, and electronic) state. The present study is devoted to an
accurate determination of the DR rate coefficient for CF+.

CF+ was first detected by Neufeld  _et al._ [3] and later found in different
ISM environments in the Milky Way, as well as in other galaxies [6, 7, 8] (see
further references in the cited works). With its relatively simple chemical
network in the ISM, the detection of CF+ can help constrain astrochemical
models, particularly those in which C+ plays a significant role. In addition
to the experimental data, there are theoretical DR cross sections reported in
the same study by Novotný  _et al._ [9]. The theoretical results agree very
well overall with the experiment. We note that no thermally averaged rate
coefficients were computed from the DR cross sections obtained by the authors
of that study. However, because the theoretical (taking into account the
anisotropic collisional energy distribution of the experiment) rate
coefficient agrees well with the experimental coefficient in the interval of
energies 0.001-0.6 eV, the thermal rate coefficients should also agree with
each other for temperatures of 10–3000 K.

For accurate modeling of the cold interstellar environment, one would need
rate coefficients obtained for the lowest rotational states of CF+. As
mentioned above, the storage ring experiments were performed at a high
rovibrational temperature of the ions. The theoretical calculations did not
account for the rotational structure and, therefore, cannot provide the
information about the dependence of the rate coefficients on rotational
excitation of CF+. In addition, the previous theory did not account for the
effect of the electronic Rydberg resonances associated to the first excited
state, a${}^{3}\Pi$, of the ion. If the incident electron is captured in one
of these resonances, it should enhance the DR cross section. Therefore, the
inclusion of the coupling of the incident electron with a${}^{3}\Pi$ Rydberg
states may be important at electron energies at and above 1 eV, where these
resonances should appear in the DR spectrum.

We have recently [10, 11, 12] developed a theoretical method that allows us to
account for the direct and indirect (via rotational, vibrational, or
electronic resonances) mechanisms [13] of DR, while accounting for the
rovibronic structure of the target ion and the coupling of the incident
electron with a large number of rovibronic ionic states to the dissociative
continuum of the neutral system. The present study applies the method to the
DR of CF+. Accurate DR cross sections are also important for understanding and
modeling processes in the plasma etching of semiconductors. Fluorocarbon
species are commonly used for etching; after a radio-frequency discharge, a
molecular plasma is formed in which CF+ is one of the major ions [14], with DR
being the major mechanism of plasma neutralization.

Because the theoretical method employed here has already been described by
Forer  _et al._ [12], the following discussion presents only the details
specific to the CF+ DR process.

Electronic states of CF+and CF. Born-Oppenheimer energies of several low-lying
singlet and triplet states of the CF+ ion were determined as functions of the
internuclear distance $R$ by Petsalakis and Theodorakopoulos [15] using the
multireference configuration interaction method. The energies of the three
lowest states (X${}^{1}\Sigma^{+}$, a${}^{3}\Pi$, A${}^{1}\Pi$) used in this
study are shown in Fig. 1. The a${}^{3}\Pi$ state has a relatively shallow
minimum, when compared to the ground state. The A${}^{1}\Pi$ state has an even
shallower minimum, which lies above the C+(^2P) + F(^2P) dissociation limit,
thereby possessing only predissociated vibrational levels. The CF potential
energy curves were later computed by the same authors [16]. While the present
study does not require an explicit form of the potential curves of the neutral
molecule, we note here that the lowest CF curves of the ${}^{2}\Sigma^{+}$,
${}^{2}\Sigma^{-}$, ${}^{2}\Pi$ and ${}^{2}\Delta$ symmetries cross the ground
state of the ion near its equilibrium geometry. In particular, the potential
curve of the lowest state of ${}^{2}\Sigma^{-}$ symmetry crosses the ionic
ground state almost exactly at the equilibrium and one of the
${}^{2}\Sigma^{+}$ curves of the neutral crosses the ionic ground state
slightly to the left from the equilibrium. This means that the direct DR
mechanism, whereby CF rovibrational Rydberg resonances are not excited, is
expected to have a significant contribution.

Figure 1:  The potential energy curves for the lowest three electronic states
of CF+, obtained using the work of Petsalakis and Theodorakopoulos [15]. The
dashed line represents the complex absorbing potential used for the ionic
ground state in the vibrational frame transformation. The straight lines
through the repulsive part of the X${}^{1}\Sigma$ curve represent the real
part of the energy of the vibrational levels used in our approach. The
dissociation limit for the displayed potential energy curves is 0 eV.

Ab initio CF+ + e- scattering calculations. The first step in the approach
involves quantum chemistry calculations of the target (CF+). This is followed
by building the scattering matrix ($S(R)$) in the Born-Oppenheimer
approximation with a fixed-nuclei R-matrix method implemented in the UKRMol+
suite [17], called with the UKRmol scripts [18], for several values of the
internuclear coordinate $R$. Before performing the electron-scattering
calculations, the electronic states of the target are constructed with the
Psi4 quantum chemistry package [19] in the $C_{2v}$ point group with the cc-
pVTZ basis set, using 2 frozen orbitals (1–2$A_{1}$) and 8 active orbitals
(3–6$A_{1}$, 1–2$B_{2}$, 1–2$B_{2}$). Scattering calculations are performed
with a close-coupling expansion, using the target’s active space supplemented
with 6 virtual orbitals (7–9$A_{1}$, 3$B_{1}$, 3$B_{2}$, 1$A_{2}$), as
described in the work of Mašín  _et al._ [17]. The two lowest electronic
states of the ion are separated by about 5 eV at the equilibrium distance
(2.216 bohr). This excited state is low enough in energy to generate a Rydberg
series of electronic resonances that affect low-energy electron scattering.
These Rydberg states provide an indirect pathway to the dissociative states of
the neutral molecule (CF). Although we mentioned in the introduction that we
expect direct DR to contribute significantly, both mechanisms can occur
simultaneously and can interfere.

The body frame reaction ($K(R)$)-matrices are obtained in the same manner as
in the previous study [12]: a K-matrix is obtained for each geometry with the
R-matrix calculations, using an R-matrix radius of 13 bohr, and is evaluated
at $\sim 5\times 10^{-3}$ eV above the ground state with channels attached to
the excited states treated as strongly closed. The scattering calculation at
the internuclear coordinate $R$ at equilibrium $R=R_{e}$ was performed with
the origin for the partial-wave expansion at the target’s center of charge (at
a distance $d_{C_{e}}$ from the center of charge to the C nucleus).
Calculations for all other values of $R$ were performed with the origin
situated at a distance $d_{C}$ from C proportional to $d_{C_{e}}$ and $R$:
$d_{C}=d_{C_{e}}R/R_{e}$. The center-of-charge frame is preferable because
scattering calculations for dipolar targets converge faster with respect to
the partial wave sum of the scattering electron. We identify the electronic
state of the ion with $n$, the orbital angular momentum quantum number of the
electron with $l$, and the projection of $\vec{l}$ on the molecular axis with
$\lambda$. Performing the closed-channel elimination on the R-matrix, as
described by Hvizdoš  _et al._ [20], produces resonances in the K-matrices, so
care must be taken to sample K-matrices at energies where the energy
dependence is smooth. We consider 9 partial waves ranging from $l=0$–$2$ in
the scattering calculations, each attached to the $X^{1}\Sigma^{+}$,
a${}^{3}\Pi$, and A${}^{1}\Pi$ states of CF+. The K-matrices are then
converted to S-matrices via the same method presented in the previous study
[12].

Scattering matrix with rovibronic channels. The next step in our approach, the
rovibrational frame transformation, is carried out in two steps: an energy-
independent vibrational frame transformation followed by a rotational frame
transformation. Compared to the previous study [12], the frame transformation
has two differences: (i) we solve the vibrational Schrödinger equation using a
basis of B-splines [21] instead of a discrete variable representation (DVR)
method and (ii) the complex absorbing potential’s (CAP) imaginary component
takes the cubic form

$V_{CAP}=\begin{cases}-2i\eta(R-R_{0})^{3}&R>R_{0}\\\ 0&R\leq
R_{0}\end{cases},$ (2)

instead of the previously used exponential form. In (2), $R_{0}$ is the
starting point of the CAP (4.0 bohr), $\eta$ is the CAP strength (0.005561
atomic units), and $L$ is the CAP length (2.0 bohr). The vibrational quantum
number $v$ runs from $0$–$69$. Similarly to previous studies [22, 23, 12],
bound states and quasi-continuum states must be represented by vibrational
wave functions used in the transformation. Quasi-continuum states are chosen
to satisfy outgoing-wave boundary conditions, which represent outgoing
dissociative flux. The resulting vibronic S-matrices, now with no
$R$-dependence, are expressed in a basis of 630 vibronic channels.

The vibronic S-matrices undergo a rotational frame transformation as described
in the previous study [12]. Channels of the resulting S-matrices are numerated
by rovibronic quantum numbers, $n$-the index of the electronic state of the
ion, $v,j$\- the vibrational and rotational quantum states in the electronic
state $n$. If one is not interested in the rotational structure of the final
cross sections (for example, to compare with experimental data where the
rotational structure was not resolved), the rotational frame transformation
can be omitted.

The energy of the initial state of CF+, $E_{nv(j\mu)}$, and the energy of the
incident electron with respect to $E_{nv(j\mu)}$, $E_{\text{el}}$, define the
total energy of the system, $E_{\text{tot}}$:

$E_{\text{tot}}=E_{nv(j\mu)}+E_{\text{el}}.$ (3)

We use parentheses around the quantum numbers $j$ and $\mu$ to indicate the
optional inclusion of the rotational resolution from the rotational frame
transformation ; the quantum numbers $j$ and $\mu$ are absent if it is
bypassed. For total energies where some scattering channels are closed, the
closed-channel elimination procedure, (CCEP) equation (2.50) in the work of
Aymar  _et al._ [24], is applied [25, 12].

The CCEP produces the scattering matrix – we call it the physical S-matrix –
$S^{phys}(E_{\mathrm{tot}})$, which is strongly energy-dependent due to the
Rydberg resonances associated with closed rovibronic (or vibronic, if the
rotational structure is neglected) channels. The open and closed channels of
the S-matrix with the rotational structure are labeled with quantum numbers
$\\{n,v,j,\mu,l\\}$, mentioned above. In addition, there is a quantum number
of the total angular momentum $J$ and its projection of the molecular axis,
over which the S-matrix is diagonal. We use the notation $S^{J,phys}_{nvj\mu
l,n^{\prime}v^{\prime}j^{\prime}\mu^{\prime}l^{\prime}}$ for the S-matrix
elements. The channels of the S-matrix without the rotational structure are
enumerated by the set of quantum numbers $\\{n,v,l,\lambda,\Lambda\\}$, where
$\Lambda$ is the projection of the angular electronic angular momentum (target
+ incident electron) on the molecular axis. We use notation
$S^{\Lambda,phys}_{n^{\prime}v^{\prime}l^{\prime}\lambda^{\prime},nvl\lambda}$
for these elements. Here and throughout, primed quantities refer to quantum
numbers or indices of final states, while their unprimed counterparts refer to
initial states.

DR cross sections and rate coefficients. Equipped with the physical
(ro)vibronic S-matrices, we can compute the total DR cross section from an
initial state of the system, $nv(j\mu)$:

$\displaystyle\sigma_{nv}(E_{\text{el}})=\frac{\pi}{2m_{e}E_{\text{el}}}\sum\limits_{l\lambda}\big{[}1-\sum\limits_{l^{\prime}\lambda^{\prime}}\sum\limits_{n^{\prime}v^{\prime}}S^{\Lambda,phys}_{n^{\prime}v^{\prime}l^{\prime}\lambda^{\prime},nvl\lambda}S^{\Lambda,phys\ddagger}_{nvl\lambda,n^{\prime}v^{\prime}l^{\prime}\lambda^{\prime}}\Big{]},$
(4)
$\displaystyle\sigma_{nvj\mu}(E_{\text{el}})=\frac{\pi}{2m_{e}E_{\text{el}}}\sum\limits_{J}\frac{2J+1}{2j+1}\sum\limits_{l}\Big{[}1-\sum\limits_{l^{\prime}}\sum\limits_{n^{\prime}v^{\prime}j^{\prime}\mu^{\prime}}S^{J,phys}_{n^{\prime}v^{\prime}j^{\prime}\mu^{\prime}l^{\prime},nvj\mu
l}S^{J,phys\ddagger}_{nvj\mu
l,n^{\prime}v^{\prime}j^{\prime}\mu^{\prime}l^{\prime}}\Big{]}.$ (5)

where $m_{e}$ is the mass of the electron. Calculations with rotational
resolution use (5), while those without it use (4).

Results. Fig. 2 presents the calculated DR cross sections for different
rovibronic levels ($n=1$; $v=0$; $j=0$, 1, 2) of CF+ and one calculation
without rotational resolution ($n=1$; $v=0$). We restrict the calculation here
to low-scattering energy ($10^{-4}-10^{-1}$ eV) to stress the rotational
effects on the cross section. The overall behavior of the cross sections at
the lowest energies is $1/E_{\text{el}}$. Many resonances are present and can
be grouped into two different classes depending on their origin. The first
class consists of sharp and narrow predissociated Rydberg resonances driving
the indirect mechanism. Some of these are shown in the inset of Fig. 2. The
second class are broad valence resonances driving the direct mechanism.

Figure 2:  CF+ DR cross sections. Cross sections with rotational structure are
shown from the three lowest states of the ion, $j=0,1,2$. Cross sections from
the ground vibronic state, without the rotational structure, are also shown
$v=0$.  Figure 3:  CF+ rate coefficients. Solid curves indicate results from
the present theory, the dashed curve represents the results of the an MQDT
calculation, and the points are experimental results obtained at the TSR. The
MQDT calculation and TSR results are from the same study [9]. Two of the solid
curves are state-selected data convolved with experimental parameters, but not
thermally averaged over initial states: $v=0$ and $j=0$. The other solid
curves represent data convolved with experimental parameters and averaged over
populations of initial states at the indicated rotational temperatures. The
inset shows the same data in the energy range 0.1–1 eV.

Fig. 3 compares the results obtained, with and without rotational resolution
($v=0$ and $j=0$, respectively), a previous MQDT calculation, and experimental
measurements made at the TSR — the latter two are from the work of Novotný
_et al._ [9]. Various convolutions of our rotational results are also
displayed in Fig. 3. The cross sections are convolved with an anisotropic
Maxwell-Boltzmann distribution, which accounts for different populations of
initial states of CF+ as well as the longitudinal and transverse energy
spreads in the TSR experiment, as described by equations 5 and 6 in the work
of Fonseca dos Santos  _et al._ [26] and Forer  _et al._ [27]. Our $v=0$ and
$j=0$ results are only convolved with the TSR collisional energy spreads;
these results remain state-selected. The remaining curves were averaged over
the initial states of CF+ at several rotational temperatures, up to 700 K.
Measurements were made at the TSR experiment [9] while the ions cooled from
$\sim$ 6000 K to 700 K, which suggests that a convolution of our results at
only 700 K may not be entirely appropriate for comparison to the TSR results.

As is evident in Fig. 3, the present thermally averaged results approach the
experimental data in the range of collision energies between $10^{-2}$ and
$6\times 10^{-2}$ eV as the rotational temperature of the thermal average
increases. The discrepancy between our $v=0$ results and the TSR results seems
to diminish as more rotational states are included at rotational temperatures
approaching those of the experiment. Our non-thermally averaged rovibronic
($j=0$) and vibronic ($v=0$) results are most different at lower collision
energies, but are much more similar above $\sim 6\times 10^{-2}$ eV. Fig. 3
gives an idea of the ranges where rotation can be reasonably neglected to
study DR in the case of CF+. As can be seen in the inset, the results are
overall very similar above $10^{-1}$ eV. As the collision energy approaches 1
eV, the theoretical results of Novotný  _et al._ [9] drop off significantly,
while our results continue to find good agreement with their experimental
measurements. This may be due to the lack of the a${}^{3}\Pi$ state of CF+ in
their theoretical mode, as mentioned in the introduction.

Figure 4:  DR thermally averaged rate coefficients. The “$j=0$” line is the
rate coefficient obtained assuming that the CF+ ions are in their ground
rovibrational state and that the electron energy distribution follows Maxwell-
Boltzmann statistics; the “Thermal average” line is obtained assuming a
Maxwell-Boltzmann distribution over initial rovibrational states of the ions
at the same temperature as the free electrons, i.e. averaged over the thermal
ensemble of the ions. The experimentally determined rate coefficient, with
error bars, of Novotny  _et al._ [5] is also shown as a function of
temperature.

Figure 4 compares the state-selected kinetic rate coefficients obtained in
this study to those reported by Novotny  _et al._ [5] and used in the
astrophysical model of the fluorine chemistry in the ISM [1]. Our results are
very similar in the temperature range 0-40 K.

In summary, we would like to stress the following key conclusions of the
study.

  * •

The present theoretical results are in excellent agreement with the TSR and
ASTRID storage ring data [9] of CF+ DR. Taking into account the anisotropic
electron energy distribution in the experiments, the present theoretical rate
coefficient (Fig. 3) agrees very well with the experiment at lower
temperatures.

  * •

The good agreement with two experiments demonstrates the validity and
performance of the theoretical approach to model the DR process.

  * •

We have performed complete DR calculations with and without accounting for the
rotational structure of the target ion and analyzed the obtained results with
the idea of understanding when the inclusion of the rotational structure is
needed and when it can be neglected.

  * •

The study has illustrated the importance of rotational resonances in DR
spectra, especially at low energies. This could be a quite general conclusion
for many target ions for which the coupling of the incident electron with
rotational motion of the target ion is significant: such a coupling produces
rotational Rydberg resonances if the corresponding rotational channel is
closed and, therefore, increases the time during which the ion-electron system
can undergo the dissociation process. The term “low energy” should be
understood as the energy below the excitation threshold of the lowest
rotational states of the target ion. The rotational effects are effectively
averaged out at higher collision energies.

  * •

The rotational structure of the ion is important to include for low-energy DR,
as evident in Fig. 3 for the low-energy $v=0$ and $j=0$ rate coefficients. The
figure also illustrates how one may well ignore the rotational resolution at
higher scattering energies, given the similarity in the $v=0$ and $j=0$ rate
coefficients above $6\times 10^{-2}$ eV.

  * •

The study has also demonstrated the role of rotational and, in general,
rovibronic structure in interpretation of experimental results from storage
rings, like those from the TSR and ASTRID [9]. DR cross sections (and
anisotropic rate coefficients) as functions of collision energy tend to show
typical drops when rotational, vibrational, or electronic thresholds are
opening: If the coupling between the incident electron and the corresponding
motion of the ion (rotational, vibrational, or electronic) is strong, below
such threshold energies electron-ion collisions lead to a large probability of
DR, while at energies above the thresholds, the collisions lead to the
excitation – rotational, vibrational, or electronic – of the ion.

##  I Acknowledgements

We acknowledge support from NSF award nos. 2303895, 2102187, and 2102188. M.
A. and V. K. also acknowledge support from the program “Accueil des chercheurs
étrangers” of CentraleSupélec. J. B. and I. F. S. acknowledge support from the
French consortiums LabEx EMC3, FR-IEPE, and PN-PCMI.

## References

  * Neufeld and Wolfire [2009] D. A. Neufeld and M. G. Wolfire, The chemistry of interstellar molecules containing the halogen elements, Ap. J. 706, 1594 (2009).
  * Dagdigian  _et al._ [2019] P. J. Dagdigian, J. Kłos, M. Wolfire, and D. A. Neufeld, Calculation of the rate of the C+\+ HF $\to$ CF+\+ H reaction: implications for fluorine chemistry in the interstellar medium, Ap. J. 872, 203 (2019).
  * Neufeld  _et al._ [2006] D. A. Neufeld, P. Schilke, K. M. Menten, M. G. Wolfire, J. H. Black, F. Schuller, H. S. Müller, S. Thorwirth, R. Güsten, and S. Philipp, Discovery of interstellar CF+, Astron. Astrophys. 454, L37 (2006).
  * Denis-Alpizar  _et al._ [2018] O. Denis-Alpizar, V. V. Guzmán, and N. Inostroza, Study of the formation of interstellar CF+ from the HF+ C${}^{+}\to$ CF+\+ H reaction, Mon. Not. R. Astron. Soc. 479, 753 (2018).
  * Novotny  _et al._ [2005] O. Novotny, J. Mitchell, J. LeGarrec, A. Florescu-Mitchell, C. Rebrion-Rowe, A. Svendsen, M. El Ghazaly, L. H. Andersen, A. Ehlerding, A. Viggiano, _et al._ , The dissociative recombination of fluorocarbon ions: II. CF+, J. Phys. B: At., Mol. Opt. Phys. 38, 1471 (2005).
  * Kalenskii and Johansson [2010a] S. Kalenskii and L. E. Johansson, Spectral survey of the star-forming region W51 e1/e2 at 3 mm, Astron. Rep. 54, 1084 (2010a).
  * Kalenskii and Johansson [2010b] S. Kalenskii and L. E. Johansson, Spectral scan of the star-forming region DR21 (OH). Observations and LTE analysis, Astron. Rep. 54, 295 (2010b).
  * Gong  _et al._ [2024] Y. Gong, K. M. Menten, A. M. Jacob, C. Henkel, and C.-H. R. Chen, First detection of CF+ in the Large Magellanic Cloud, Astron. Astrophys. 10.1051/0004-6361/202449429 (2024).
  * Novotný  _et al._ [2009] O. Novotný, O. Motapon, M. H. Berg, D. Bing, H. Buhr, H. Fadil, M. Grieser, J. Hoffmann, A. Jaroshevich, B. Jordon-Thaden, C. Krantz, M. Lange, M. Lestinsky, M. Mendes, S. Novotny, D. A. Orlov, A. Petrignani, I. F. Schneider, A. Orel, and A. Wolf, Dissociative recombination of CF+: Experiment and theory, J. Phys. Conf. Ser. 192, 012021 (2009).
  * Jiang  _et al._ [2019] X. Jiang, C. H. Yuen, P. Cortona, M. Ayouz, and V. Kokoouline, Cross sections for vibronic excitation of CH+ by low-energy electron impact, Phys. Rev. A 100, 062711 (2019).
  * Jiang  _et al._ [2021] X. Jiang, J. Forer, C. H. Yuen, M. Ayouz, and V. Kokoouline, Theory of dissociative recombination of molecular ions with competing direct and indirect mechanisms applied to CH+, Phys. Rev. A 104, 042801 (2021).
  * Forer  _et al._ [2023] J. Forer, D. Hvizdoš, X. Jiang, M. Ayouz, C. H. Greene, and V. Kokoouline, Unified treatment of resonant and nonresonant mechanisms in dissociative recombination: Benchmark study of CH+, Phys. Rev. A 107, 042801 (2023).
  * Bardsley [1968] J. N. Bardsley, Theory of Dissociative Recombination, J. Phys. B: At. Mol. Phys. 1, 365 (1968).
  * Goyette  _et al._ [2000] A. Goyette, Y. Wang, M. Misakian, and J. K. Olthoff, Ion fluxes and energies in inductively coupled radio-frequency discharges containing C2F6 and c-C4F8, J. Vac. SCI Technol. A 18, 2785 (2000).
  * Petsalakis and Theodorakopoulos [2000] I. D. Petsalakis and G. Theodorakopoulos, Electronic states of cf+, Chem. Phys. 254, 181 (2000).
  * Petsalakis and Theodorakopoulos [2011] I. D. Petsalakis and G. Theodorakopoulos, Theoretical calculations on the potential energy curves of electronic states of CF: Rydberg states of CF above the lowest ionization limit, Chem. Phys. Lett. 508, 17 (2011).
  * Mašín  _et al._ [2020] Z. Mašín, J. Benda, J. D. Gorfinkiel, A. G. Harvey, and J. Tennyson, UKRmol+: A suite for modelling electronic processes in molecules interacting with electrons, positrons and photons using the R-matrix method, Comput. Phys. Commun. 249, 107092 (2020).
  * Houfek  _et al._ [2024] K. Houfek, J. Benda, Z. Mašín, A. Harvey, T. Meltzer, V. Graves, and J. D. Gorfinkiel, UKRmol-scripts: a Perl-based system for the automated operation of the photoionization and electron/positron scattering suite UKRmol+, Comput. Phys. Commun. , 109113 (2024).
  * Turney  _et al._ [2012] J. M. Turney, A. C. Simmonett, R. M. Parrish, E. G. Hohenstein, F. A. Evangelista, J. T. Fermann, B. J. Mintz, L. A. Burns, J. J. Wilke, M. L. Abrams, _et al._ , Psi4: an open-source ab initio electronic structure program, WIREs Comput. Mol. Sci. 2, 556 (2012).
  * Hvizdoš  _et al._ [2023] D. Hvizdoš, J. Forer, V. Kokoouline, and C. H. Greene, Bound and autoionizing potential energy curves in the CH molecule, J Chem. Phys. 159 (2023).
  * Forer and Kokoouline [2024] J. Forer and V. Kokoouline, BSPRVSE, https://github.com/banana-bred/bsprvse (2024).
  * Kokoouline and Greene [2005] V. Kokoouline and C. H. Greene, Theoretical study of dissociative recombination of ${C}_{2v}$ triatomic ions: Application to H2D+ and D2H+, Phys. Rev. A 72, 022712 (2005).
  * Čurík and Greene [2008] R. Čurík and C. H. Greene, Rates for dissociative recombination of LiH+ ions, J. Phys. Conf. Ser. 115, 012016 (2008).
  * Aymar  _et al._ [1996] M. Aymar, C. H. Greene, and E. Luc-Koenig, Multichannel Rydberg spectroscopy of complex atoms, Rev. Mod. Phys. 68, 1015 (1996).
  * Kokoouline and Greene [2003] V. Kokoouline and C. H. Greene, Unified theoretical treatment of dissociative recombination of LiH+ triatomic ions: application to H3+ and D3+, Phys. Rev. A 68, 012703 (2003).
  * Fonseca dos Santos  _et al._ [2007] S. Fonseca dos Santos, V. Kokoouline, and C. H. Greene, Dissociative recombination of H3+ in the ground and excited vibrational states, J. Chem. Phys. 127, 124309 (2007).
  * Forer  _et al._ [2024] J. Forer, D. Hvizdoš, M. Ayouz, C. H. Greene, and V. Kokoouline, Kinetic rate coefficients for electron-driven collisions with CH+: dissociative recombination and rovibronic excitation, Mon. Not. R. Astron. Soc. 527, 5238 (2024).