date01052022 date115022022

[orcid=0000-0002-8330-2703]

[1]

Data curation, Formal analysis, Investigation, Methodology, Software,
Validation, Visualization and Roles/Writing - original draft

1]organization=School of Mathematical and Computational Sciences, Indian
Association for the Cultivation of Science ,city=Kolkata, postcode=700032,
country=India

[orcid=0000-0003-0050-7138] Conceptualization, Formal analysis, Investigation,
Methodology, Project administration, Supervision, Validation and Writing -
review and editing

2]organization=School of Computing Science, University of Glasgow,
city=Glasgow, postcode=G12 8QQ, country=UK

[orcid=0000-0001-7655-7591] Conceptualization, Formal analysis, Funding
acquisition, Project administration, Resources, Supervision, Validation and
Writing - review and editing

[orcid=0000-0003-1588-4665]

Conceptualization, Formal analysis, Project administration, Supervision,
Validation and Writing - review and editing

3]organization=Department of Computer Science and Engineering, Jadavpur
University, city=Kolkata, postcode=700032, country=India

[cor1]Corresponding author

# Enhancing AI Research Paper Analysis: Methodology Component Extraction using
Factored Transformer-based Sequence Modeling Approach

Madhusudan Ghosh<EMAIL_ADDRESS>[  Debasis Ganguly
<EMAIL_ADDRESS>[  Partha Basuchowdhuri
<EMAIL_ADDRESS>Sudip Kumar Naskar
<EMAIL_ADDRESS>[

###### Abstract

Research in scientific disciplines evolves, often rapidly, over time with the
emergence of novel methodologies and their associated terminologies. While
methodologies themselves being conceptual in nature and rather difficult to
automatically extract and characterise, in this paper, we seek to develop
supervised models for automatic extraction of the names of the various
constituents of a methodology, e.g., ‘R-CNN’, ‘ELMo’ etc. The main research
challenge for this task is effectively modeling the contexts around these
methodology component names in a few-shot or even a zero-shot setting. The
main contributions of this paper towards effectively identifying new evolving
scientific methodology names are as follows: 1) we propose a factored approach
to sequence modeling, which leverages a broad-level category information of
methodology domains, e.g., ‘NLP’, ‘RL’ etc.; 2) to demonstrate the feasibility
of our proposed approach of identifying methodology component names under a
practical setting of fast evolving AI literature, we conduct experiments
following a simulated chronological setup (newer methodologies not seen during
the training process); 3) our experiments demonstrate that the factored
approach outperforms state-of-the-art baselines by margins of up to 9.257% for
the methodology extraction task with the few-shot setup.

###### keywords:

Information Extraction Transformer Models Distant Supervision Factored
Modelling Scientific Literature Chronological Evaluation Framework

We propose a novel transformer-based factored model for our extraction task.

We propose a novel chronological evaluation setup proposed as opposed to
static one.

We explore the viability of retraining technique for new methodology
extraction task.

Experimental results show that our novel model achieves state-of-the-art
performance.

##  1 Introduction

A general trend observed in the scientific literature of any discipline is
that it grows at a rapid rate embracing new theories, methodologies and their
empirical validations or refuting. Some disciplines, such as the biological
science or behavioural science, often rely on a technique called meta-analysis
that combines evidences from a number of studies (Walker et al., 2008). The
steep increase of scientific publications makes it difficult to manually
conduct evidence synthesis. Moreover, such rapid advances in scientific
methodologies 111Since ‘methodology’ is more of an abstract concept, for the
purpose of this paper, we consider the names of the more concrete methodology
components as our objects of interest, e.g., ‘BERT’ (a transformer), ‘Adam’
(an optimisation method) etc. From hereon, we use the word ‘methodology’ to
actually refer to the various methodology components encountered in AI
research papers. create difficulty for researchers to maintain a comprehensive
and updated knowledge of the recent literature, which is critical for academic
tasks, such as developing novel research ideas, selecting the correct
baselines (Bedi et al., 2022), or peer-reviewing others’ research. As attempts
to provide automated support for these activities, previous research has
applied automated information extraction (IE) approaches for identifying key
properties from a scientific article, e.g., while (Hou et al., 2019) proposed
a supervised sequence labeling approach to identify task and dataset names
from papers, (Scells and Zuccon, 2018) and (Singh et al., 2017a) applied an
information retrieval based approach for analysis of systematic reviews.

A limitation of the existing studies that seek to automatically extract key
concepts, such as tasks, datasets and method names from scientific articles,
is that either these have been empirically validated to work well only on the
abstracts of papers (Gábor et al., 2017), or no attempt has been made to
evaluate how effectively the models _transition_ to newer concepts and methods
introduced into the literature. For instance, the context in which the word
‘transformer’ is used in the recent years is substantially different from the
conventional sense of the term; earlier it used to refer to an electrical
device (Engineering) or an operator function (Mathematics), whereas these days
it mostly refers to self-attention based neural architecture. Figure 1 shows
more examples from the NLP domain on how the dominant methodologies and their
contexts can change over time.

Excerpt from paper Vaswani et al. (2017)
---
The Transformer follows this overall architecture using stacked multi-head
attention and point-wise, fully connected layers for both the encoder and
decoder…
Excerpt from paper Devlin et al. (2019)
BERT’s model architecture is a multi-layer bidirectional Transformer encoder
based on the original implementation described in Vaswani et al. (2017) …
Table 1:  A demonstration on how the contexts around the methodology name -
‘transformer’ evolved over time. During its inception, the word ‘transformer’
needed to be explained in terms of stacked ‘multi-head attention’, ‘point-wise
fully connected layers’ etc. After 2 years, with its frequent use in NLP and
image classification, researchers simply refer to a ‘transformer’ as a known
concept, and use it as a building block to develop novel architectures. This
is the reason why the contexts around the word ‘transformer’ in more recent
papers, such as the BERT paper shown in the example, are different from the
ones in 2017. These differences in contexts are likely to pose a challenge in
automatically detecting the methodology name ‘transformer’ from papers that
appeared from around 2019 using a model that is trained on data annotated till
2017.

While standard BERT-based (Devlin et al., 2019) sequence labeling approaches,
e.g. (Beltagy et al., 2019; Jain et al., 2020) etc., when fine-tuned with the
example mentions of the word ‘transformer’ from scientific articles that
appeared after the publication of the paper “Attention is all you need”
(Vaswani et al., 2017) in the year 2017 is expected to work fairly well to
identify the neural network sense of the term, a critical pragmatic question
to ask is: how well can we identify methodology names that are either not seen
during the training process, or even if seen their meanings were different?

This is an important question to ask, because an affirmative answer to this
question implies that newer (domain-specific) methodologies of the future can
then potentially be detected only by leveraging the information from the
contexts of mentions of similar methodology names, e.g., the word
‘transformer’ can still be detected as a _methodology component_
(corresponding to the neural network sense of the term) given that the
contexts around its mentions are similar to the contexts of other similar
methodology components existing before 2017 used to solve similar downstream
tasks, e.g. LSTMs and CNNs. However, this is a challenging task because it is
not only the introduction of new terminologies but also the evolution of
common interpretations of terminologies that can pose further challenges. For
instance, as Table 1 illustrates how the context terms around the word
‘transformer’ has changed since the inception of the methodology in 2017 from
a formal definition to a more common usage.

Existing work has either focused on extracting a set of relatively well-
defined entities, the contexts of which are less likely to evolve over time
(e.g., tasks and datasets as investigated in (Hou et al., 2019, 2021)), or
they have investigated the more challenging task of methodology extraction
from within a static collection (Jain et al., 2020). In contrast, we follow a
more pragmatic course in our experiments where we investigate the feasibility
of how effectively can methodology component names be identified under a zero
or a few-shot setting.

###  1.1 Practical Implications

In this section, we highlight the practical implications of our research. Due
to the fast evolving nature of scientific discipline, it is difficult for
researchers to manually keep a comprehensive track of the state-of-the-art
knowledge of the scientific literature. Different task-specific tools have
been proposed to mitigate this effort, which includes work in selecting
correct baselines, peer-reviewing other research work (Bedi et al., 2022) and
also scientific paper recommendation system (Chaudhuri et al., 2022). For
developing such types of automated systems, emerging methodology extraction
from scientific articles plays a significant role, which is in fact, the
objective of this paper.

Towards addressing the more challenging extraction task with the zero or few-
shot setup, we, as a novel contribution in this paper, propose a factored
approach of sequence labeling by leveraging the broad level domain name
categories of AI papers, e.g., NLP, RL (reinforcement learning) etc. More
concretely speaking, we partition either the input space constructing an
ensemble of domain specific models, or we partition the label space allowing a
multi-task oriented learning of predicting the text spans of relevant mentions
along with the domain categories. To tackle the task of practical interest,
i.e., the downstream prediction task of learning from the past data and
allowing the model to update itself from more recent data, we investigate how
well the proposed factored models can be updated with incremental training on
new silver-standard data - those accumulated from the model’s predictions on
recent scientific articles. To illustrate this with an example, consider
training a model with gold-standard data till time $t$ (e.g., the year 2017),
making predictions with this model on $t+1$ (e.g., the year 2018), feed back
this predicted data of identified mentions as a silver-standard data to update
the model, and then further use it on more recent data at time $t+2$ (e.g.,
the year 2019) (Zeng et al., 2018; Qu et al., 2022; Luo et al., 2020). Our
experiments show that this strategy works particularly well for our proposed
factored approaches.

(a)

(b)

Figure 1: Word cloud showing the relative change in the dominant methodology
names between papers published in pre- and post-2017 (data collected from the
online repository of research articles - paperswithcode.com).

###  1.2 Our Contributions

To summarise, the following are our contributions in this paper.

  1. 1.

We propose two factored approaches to transformer-based sequence labeling -
one that uses partitioning of the input space, and the other that uses
partitioning of the label space. Both these models are shown to perform better
than SciREX (Jain et al., 2020), a state-of-the-art extractor of scientific
concepts.

  2. 2.

We propose our novel evaluation framework by partitioning the data
chronologically. We carry out a more pragmatic approach of methodology
extraction from AI papers, as opposed to the more conventional static setup of
previous works (Hou et al., 2019, 2021; Luan et al., 2017; Jain et al., 2020).
In particular, we follow a chronological train-test split of the data, and
investigate the effectiveness of various models in identifying new methodology
names.

The structure of the paper is as follows. Section 2 provides an overview of
related work. Section 3 describes our proposed novel factored transformer-
based approach for methodology name extraction. In Section 5, we describe the
experiment setup. Section 6 presents and analyses the observations from the
results of our experiments. Finally, Section 7 concludes the paper with
directions for future work.

##  2 Related Work

Since our work mainly investigates deep metric learning based entity
extraction approaches from scientific literature, we now first discuss the
state-of-the-art techniques used for the named-entity recognition (NER) task,
and then follow it up with how these NER models are adapted particularly for
the scientific concept extraction task, a task which we, in fact, address in
this paper.

#####  Generic NER:

Our work revolves around solving the NER task, as we extract scientific
entities, such as AI methodology names. Over the years, researchers have used
many different techniques for entity extraction from different domains (Li et
al., 2020b; Nasar et al., 2021; Ji et al., 2020; Zhang et al., 2019).
Generally, NER problem is treated as a sequence labeling task (Fei et al.,
2019; Ji et al., 2020). Popular deep learning based approaches such as LSTM
based models (Chiu and Nichols, 2016; Cho and Lee, 2019) and CNN based models
(Sung et al., 2021; Ma and Hovy, 2016) and along with that attention module
based frameworks (Patra and Moniz, 2019; Li et al., 2020c; Lin et al., 2020;
Dutta et al., 2020) are widely used in literature to handle the sequence
labeling task. Recently, neural network based approaches also incorporate
probabilistic graph models such as Conditional Random Fields (CRF) in
conjunction with BiLSTM as BiLSTM-CRF (Mayhew et al., 2020). Currently
pretrained language model based approaches have become very popular. Some of
the most important ones include intra-span and inter-span information (Wang et
al., 2020b), few-shot slot tagging (Ma et al., 2021), decoupled NER model with
two stage training (Hu et al., 2021), noise aware training mechanism (Huang et
al., 2021), contextual embedding model (Hoory et al., 2021), Trigger Matching
Network which encodes and softly grounds the _entity triggers_ of unlabeled
sentences (Lin et al., 2020), meta self-training framework for few-shot
sequence labeling (Wang et al., 2021b), and subsequence-based deep active
learning (Radmard et al., 2021). Other extensions of NER include biomedical
NER (Tong et al., 2021; Fan et al., 2020; Liu et al., 2021; Asghari et al.,
2022), nested NER (Xu et al., 2021; Li et al., 2020a), joint entity and
relation extraction (Dai et al., 2019; Zeng et al., 2020; Nayak and Ng, 2020;
Xiao et al., 2020; Sun et al., 2021; Li et al., 2021b), named entity
normalization (Ji et al., 2021), unified NER (Li et al., 2021a), NER for low-
resource languages (Das et al., 2017), document level NER using multitask
learning approach (Wang et al., 2021a), NER with multi-level topic aware
attention mechanism (Ma et al., 2023), information extraction module using
multi-modal approach (Toledo et al., 2019). Another approach is to align the
entities by combining both embedding and symbol based techniques (Jiang et
al., 2022). Additionally, an effective methodology, named FLAIR, was developed
in (Akbik et al., 2018) that allows users to fine-tune any word embedding and
any PLM to yield improved results on the NER task.

#####  Weakly Supervised NER:

In the present era of data-hungry deep learning models, availability of
annotated data in adequate quantities is a problem. Preparing annotated data
in a distantly supervised manner (Mintz et al., 2009; Riedel et al., 2010)
alleviates this problem to some extent. In our work, we have generated weakly
supervised silver-standard annotations for NER task to incrementally re-train
our model. It prompted us to explore the state-of-the-art techniques for NER
using weak supervision. Mintz et al. (2009) and Riedel et al. (2010) addressed
the problem of relation extraction and mention identification without labelled
text. Recently, gazetteer based distant supervision technique had been applied
to generate data for entity extraction task (Mengge et al., 2020; Yang et al.,
2018; Cao et al., 2019; Peng et al., 2019). They used Wikipedia anchors or
gazetteers and other existing knowledge bases (KBs) (i.e., MEDLINE, WikiData
etc.) for this task. Ali et al. (2020) proposed FG-NET to obtain fine-grained
named entity typing over distantly supervised data. Furthermore, Liu et al.
(2020) proposed a headword amplified multi-span distantly supervised method
for NER. Heist and Paulheim (2021) employed rule mining to populate Wikipedia-
based KBs, where the rules were derived with a weak supervision approach.

#####  Emerging NER:

For our scientific entity extraction task, we often come across newly coined
terms (methodology names) or rarely used terms that eventually emerged as a
frequently used term at a later time. Related works in emerging NER help us to
understand how we could design our models to mitigate the challenge derived
from the emerging nature of a number of entities. Allan et al. (1998)
conducted online identification of new topics, and also tracked how the
meanings of these topics change over time. Recently, some important works have
been done on the topic of language dynamics i.e., to how word meanings change
over time (Mitra et al., 2014; Mukherjee et al., 2011; Maity et al., 2012;
Loreto et al., 2012; Endris et al., 2017; Singh et al., 2017b; Kelley and
Gillan, 2022). Recently, Kabongo et al. (2023) addressed emerging TDM
($\langle$Task, Dataset, Metric$\rangle$) extraction task from the scholarly
documents of AI literature by considering it as textual entailment recognition
task. In contrast to identifying emerging tasks and datasets (Kabongo et al.,
2023), our work focuses on extracting newly introduced methodology components
from AI literature, which is more challenging because emergence of new
methodologies (models) is often more common than the emergence of new tasks
and datasets. Moreover, in contrast to modeling this as a textual entailment
task (Kabongo et al., 2023), we propose a factored sequence labeling model.

#####  Scientific NER:

In recent years, IE from scientific papers has become a popular area of study
in NLP. Our work deals with extraction of AI methodologies from scientific
literature and led us to subsequent exploration of related works. Athar and
Teufel (2012); Viswanathan et al. (2021) used citations and Jurgens et al.
(2018) used topic trends to extract information from scientific papers.
Recently, Safder et al. (2020) extracted algorithmic metadata from scholarly
articles by applying Bi-LSTM. Tsai et al. (2013) applied unsupervised
bootstrapping method to identify and cluster the main concepts of a paper. In
SemEval 2017 (Task 10), a novel key-phrase boundary classification method was
presented to identify entities with class labels - process, materials and task
(Augenstein and Søgaard, 2017).

Gábor et al. (2017) proposed the task of semantic relation extraction and
classification from scientific papers (SemEval 2018 Task 7) using a dataset of
350 annotated abstracts. These datasets had been used to develop their neural
models for IE on scientific literature (Ammar et al., 2017; Luan et al., 2017;
Augenstein and Søgaard, 2017). Luan et al. (2018) used relation types and
cross-sentence relations to construct a framework called Scientific
Information Extractor (SciERC) for extraction of scientific entities of six
types $\langle$Task, Method, Metric, Material, Other-Scientific Term, Other-
Generic$\rangle$ and seven relation types. Jain et al. (2020) extended this
work and released a dataset on document level IE which covers extraction of
scientific entities of four types $\langle$Dataset , Method, Task ,
Metric$\rangle$. Hou et al. (2021) and Kabongo et al. (2021) proposed a TDM
tagger using a novel data augmentation technique. Hou et al. (2021) applied
this tagger to around 30,000 NLP papers from the ACL anthology and
demonstrated its efficacy to construct an NLP TDM knowledge graph. Recently,
an end to end large scientific knowledge graph has been been proposed by
introducing ‘evaluatedby’ and ‘evaluatedon’ relationships (Mondal et al.,
2021). Recently, Ghosh et al. (2022) devised transfer learning strategy by
finetuning the mT5 language model for downstream astronomical entity
extraction task from astrophysics literature. Kardas et al. (2020) linked
automatically extracted results from tables to leaderboards. Recently, Lo et
al. (2020) and Saier and Färber (2020) proposed two versions of large datasets
namely ‘S2ORC’, ‘unarXive’ from different scientific disciplines.
Additionally, citation context plays an important role for scientific
information extraction task. In this context, Cohan et al. (2019) explored
structural scaffolds based approach for citation intent classification task
from scientific literature. Lahiri et al. (2023) used propmt based approach
using GPT-2 for the detection of citation context from the scientific article.

In addition to the extracting information from computer science articles as
cited above, information extraction from articles of social science is
comparatively under-explored. Works in this direction include those of Shen et
al. (2023) who apply a pre-trained language model specifically designed for
social science texts called SsciBERT. Additionally, scientific context
generation is also an important area of research along with the scientific IE
task. Recently, Chen et al. (2021) explored the contributions of context in
scientific text generation by proposing two primary tasks, i.e., context-aware
description generation and context-aware paragraph generation. Due to fast
evolving nature of scientific literature, another interesting area, which
remains under-explored, is scientific paper recommendations. In relation to
this context, Chaudhuri et al. (2022) proposed a novel scientific paper
recommendation system.

The main difference between our work and the existing studies of scientific
entity extraction from research papers is that we study this problem in a
realistic few-shot setup, where the objective is to identify new and emerging
methodologies without a significant presence of such entities in historical
data. Moreover, we propose a factored sequential modeling approach for the
task which is also novel.

##  3 Proposed Methodology

This section starts with the problem definition considering the methodology
extraction as a sequence labeling task, which is then followed by details on
the neural architecture of our proposed factored models.

#####  Sequence Labeling:

Given a sentence from a document, we follow a sequence labeling approach for
method component extraction. Formally, given a sentence (word sequence)
$x=(\mathrm{w_{\\_}1,w_{\\_}2...w_{\\_}n})$, the objective is to learn a
function parameterized by $\theta$, $f_{\\_}{\theta}$, that maps an observed
sequence of embedded vectors to a sequence of labels
$f_{\\_}{\theta}:\mathbf{w_{\\_}1},\ldots,\mathbf{w_{\\_}n}\rightarrow
y_{\\_}1,\ldots,y_{\\_}n$, where each $\mathbf{w_{\\_}i}\in\mathbb{R}^{d}$ is
an embedded vector of the token $w_{\\_}i$, and each
$y_{\\_}i\in\\{\mathrm{B},\mathrm{I},\mathrm{O}\\}$ denotes a label which, as
per the standard NER terminology, indicates if it is the beginning,
continuation or the end of a text span (in the context of our work - an
identified methodology name). Given a set of examples of such
$\mathcal{D}=\\{(x,y)\\}$ sequence pairs, the parameters $\theta$ of a
sequence classification model are learned by optimizing

$z=\underset{\theta}{\mathrm{argmin}}\,\sum_{\\_}{(x,y)\in\mathcal{D}}\mathcal{L}(y,f(x,\theta)),$
(1)

where $\mathcal{L}$ is a standard loss function, e.g., the cross-entropy.

We employ the state-of-the-art BERT-based transformer architecture which
leverages the information obtained during the pre-training phase, transferring
this knowledge to facilitate a downstream task Wang et al. (2020a). In
particular, following Jain et al. (2020), we use SciBERT (Beltagy et al.,
2019) as our pre-trained transformer model and fine-tune it on the sequence
labeling objective of Equation 1.

o incorporate the temporal essence into the task of extracting new
methodological terminologies from the recent scientific articles, we split our
whole data corpus into two parts based on a specific year. We call these two
datasets in-domain data and out-domain data. During dataset construction, we
also prepare a mapping of methodological terms and their corresponding year
from the PwC knowledge base (KB) (i.e., we take that year of a method name
whenever it first time appears in PWC KB). We are using that mapping to create
our in-domain and out-domain datasets. We first split the method components
for a given year into two parts. If a method appears before or in that
particular given year, we keep it in one part and vice versa. Then we apply
some search heuristics using those two sets of method components on our
distantly supervised dataset to prepare our in and out-domain sentence
instances accordingly.

###  3.1 Factored Model for Sequence Labeling

We propose two factored modelling approaches for the task of methodology names
extraction by leveraging their broad-level categorical information (more
details in Section 5.1.2). Both the proposed models involve the common
processing steps of fine-tuning a transformer model and applying a standard
CRF decoder layer similar to Jain et al. (2020). Both these models make use of
partitions in the data induced by the category information; while one approach
partitions the input data, the other partitions the label space. More details
follow.

####  3.1.1 Model with Partitioned Input Space

Let there be a total of $m$ broad-level categories that may be associated with
the methodology names, e.g., ‘GPT’ for ‘NLP’, ‘ResNet’ for ‘computer vision’
etc. (Section 5.1.2 describes how these categories are obtained for the
PapersWithCode dataset). In this factored approach, we partition each sentence
of the dataset into $m$ categories, and then train $m$ different sequence
labeling models for each of these partitions. Formally speaking,

$f_{\\_}{\theta_{\\_}i}(x\in\mathcal{D}_{\\_}{i})\mapsto
y\in\\{\mathrm{B},\mathrm{I},\mathrm{O}\\},\,\,\,i=1,\ldots,m,\,\,\text{and}\,\,\mathcal{D}=\bigcup\limits_{\\_}{i=1}^{m}\mathcal{D}_{\\_}{i}.$
(2)

In other words, we $m$ different models
$f_{\\_}{\theta_{\\_}i},\ldots,f_{\\_}{\theta_{\\_}m}$ models, where each
$f_{\\_}{\theta_{\\_}i}$ takes as input instances of the $i^{\text{th}}$
category and tunes the corresponding $\theta_{\\_}i$ as required model
parameters using the sequence labeling loss as defined in Equation 1.

The common intuition behind applying this approach is that a single model may
not essentially capture the inherent diversities of the contexts of different
types of methodology names across different domains. For instance, the context
of the methodology component ‘transformer’ is expected to be different for
image and text categories - the former expected to be associated with contexts
relevant to image/video classification, whereas the latter being associated
with terminologies such as summarization, question answering, etc. As
described later in Section 5.1.2, the value of the category level information
is available at the level of documents (research articles). The document-level
category is then assigned to each of its constituent sentences during the
model training phase.

After training $m$ different models, during inference phase we assume that the
category of a test document is known. This allows determining which model, out
of the $m$ choices, is to be used for prediction. We call this DFG based
approach (as presented in Figure LABEL:fig:CG-model) to indicate that it uses
a partition of the input Data to model the Fine-Grained information.

Figure 2: Schematic diagram of retraining strategies on chronological data
partitions.

####  3.1.2 Model with Partitioned Label Space

This model instead of using the three coarse-grained labels
$\\{\mathrm{B},\mathrm{I},\mathrm{O}\\}$ (indicators of the beginning,
continuation and end of a text span), uses the category information to create
more fine-grained labels. Specifically speaking, if $c$ denotes one of the $m$
categories, for each coarse-grained indicator label (B/I/O), we create its
fine-grained counterpart. Formally, a label’s value $y$ belongs to the
following set:

$\\{\langle Z,c\rangle:\,\,Z\in\\{\mathrm{B},\mathrm{I},\mathrm{O}\\}\land
c\in\\{1,\ldots,m\\}\\}.$ (3)

To give an example, if ‘NLP’ represents a category, then the B/I/O labels
corresponding to this category are $\langle B,\mathrm{NLP}\rangle$ etc. We
then use the labels as defined in Equation 3 to train the sequence labeling
objective. We name this approach LFG to indicate that it uses a set of Labels
that are Fine-Grained (see Figure LABEL:fig:fac-CG). Note that in contrast to
the DFG model, the LFG one involves training only a single model overall,
which means that it is not required to know the category information during
the inference phase. Since the downstream task involves identifying the text
segments only, the fine-grained labels post-inference are trivially mapped
back to their coarse-grained interpretations, i.e., $\langle Z,c\rangle\mapsto
Z$, e.g., $\langle\text{B},\mathrm{NLP}\rangle$ and
$\langle\text{B},\mathrm{GRAPH}\rangle$ are treated as identical by stripping
off the categories ‘NLP’ and ‘GRAPH’ after the inference step.

##  4 Chronological Evaluation Framework

In this section, we propose a framework to evaluate various methodology
extraction models in a way different from the conventional one-time evaluation
using a train:test split of the data Hou et al. (2019); Jain et al. (2020).
The main motivation of proposing a new evaluation framework is to investigate
the practical use-case on how effectively can models trained on historical
data can adapt themselves to keep pace with the fast evolving literature of a
scientific discipline. This is particularly important because obtaining
annotated data for gold-standard model training takes considerable manual
effort Liang et al. (2020); Zhao et al. (2021). From a pragmatic point-of-
view, a scientific concept extraction model should thus be robust enough to be
trained incrementally with its own predicted entities (silver-standard aka
weakly supervised training) over a period of time.

More concretely speaking, the core idea of the proposed evaluation framework
is to first induce a chronological split of the dataset comprising scientific
articles. The first (earlier in time) part of the split is treated as the
existing literature and a sequence labeling model $M$ to identify scientific
concepts is trained on this gold-standard data (denoted by $M_{\\_}t$ in
Figure 2). Next, $M_{\\_}t$ is applied on the documents published in the year
$t+1$ (denoted as $D_{\\_}{t+1}$ in Figure 2) to extract a set of predicted
scientific concept names $Pred_{\\_}{t+1}$. This data is then fed back to
$M_{\\_}t$ as silver-standard data, and $M_{\\_}t$ is incrementally retrained
with these automatically extracted entities to yield the next version of the
model, $M^{\prime}_{\\_}{t+1}$ Luo et al. (2020). Subsequently,
$M^{\prime}_{\\_}{t+1}$ is applied on the next year’s data $D_{\\_}{t+2}$ and
the cycle continues.

An expected observation from an effective model is that retraining it with
silver-standard data should be better than not updating the model, e.g.,
$V(M^{\prime}_{\\_}{t+1},D_{\\_}{t+2})>V(M_{\\_}t,D_{\\_}{t+2})$, where the
notation $V(M,D)$ denotes some evaluation metric (e.g., precision, recall
etc.) obtained with model $M$ on dataset $D$. However, training with silver-
standard data for a subsequent time-period (i.e., in our case with a model’s
own predictions) is also expected to be worse than the ideal situation of the
availability of annotated (gold-standard data) over this period. In other
words, following our terminology, it is expected that
$V(M^{\prime}_{\\_}{t+1},D_{\\_}{t+2})<V(M_{\\_}{t+1},D_{\\_}{t+2})$. Later in
our experiments in Section 6.4, we show that our proposed models indeed
exhibit this behaviour, i.e., they may be updated with silver-standard data
(its own predictions) to improve its effectiveness on future research papers.

In particular, the chronological split of the data to simulate the delineation
of the data into the past history and the future, we choose the year 2017,
i.e., papers dated up to 2017 (inclusive) serving as the training split, and
papers that are published from 2018 to date serving as the evaluation or test
data. The specific reason for choosing $t=2017$ is to capture the general
effect of the transition in several domains around the year 2017 after the
introduction of transformers, explainable AI etc. For example, the NLP
community itself has witnessed a paradigm shift from the pre-BERT era (2018)
to the recent times.

The decision to assign a paper to one of the splits was determined by its
timestamp. The timestamp of a paper is the most recent timestamp of its
constituent methodology names. Thus, if there exists at least one methodology
name in a paper the timestamp of which is past 2017, it is allocated to the
evaluation set. For example, although a paper may have used LSTMs (a pre-2018
technology) to model a sequence of BERT (a post-2017 technology) vectors of
passages, the paper is still allocated to the evaluation set.

Figure LABEL:fig:percat-stats shows the number of methodology name tags per-
category across the training and the test data splits. It can be seen that the
prediction task is likely to be challenging because the number of unique tags
appearing post-2017 is often higher than the number of tags appearing
pre-2018.

##  5 Experiment Setup

(a)

(b)

Figure 3: Screenshot of a PwC webpage showing a) the broad categories or the
topics of a paper, and also its b) methodology component tag names as uploaded
by the authors.

In this section, we first describe the details of preparing the dataset used
in our experiments from the PapersWithCode (PwC) collection, and then we
follow it up with the research questions pertaining to the task of methodology
extraction, and the methods investigated towards addressing those questions.

###  5.1 Dataset Preparation

We employ a realistic setup for evaluating our proposed models in a ‘few-shot’
manner. Specifically, we employ a chronological splitting strategy of the
entire dataset to constitute a training set of older technologies and a newer
(more recent) evaluation set comprised mostly of new methodology names that
are non-existent in the training set (zero-shot setup). In situations where
the methodology name phrase exists in the training set, it is mostly the case
that the semantic meaning of the string has evolved significantly, e.g., the
word ‘transformer’. This section outlines the dataset preparation strategy and
provides detailed statistics on the prepared dataset.

| Std. dev | Mean
---|---|---
The distribution of papers over time | 3709.4 | 2152.1
Distribution of number of method tags per paper | 4.6 | 6.7
Distribution of distinct tags present in the dataset | 510.7 | 98.7
Table 2: Detailed statistics of the dataset used in our experiments.

####  5.1.1 Selecting and parsing Papers from PwC

As a first step, we downloaded the PwC JSON dump222https://production-
media.paperswithcode.com/about/papers-with-abstracts.json.gz as updated on
date comprised of a total of 291,503 papers. Each paper is accompanied with
metadata that includes the title, abstract, a list of tags indicating
methodology components such as ‘RoBERTa’ and ‘GPT-2’ etc., and the Arxiv links
to the paper. Since the information most important to us is the mentions of
the methodology components for a paper (as these correspond to the $y$ values
used to train our sequence labeling models as per Equations 1 to 3), we remove
those papers which do not possess this metadata field. This leaves us with a
total of 34,560 papers to work with. Table 2 presents some statistics of the
dataset. As an example, Figure 3(b) shows the set of methodology names tagged
by the authors from a sample PwC page.

As the next step, we use the SciPDF
parser333https://github.com/titipata/scipdfparser.git, a parser implementation
tailored for scientific PDFs, to parse the text from each PDF file. The parser
outputs a structured view of the document text separated out into segments
comprising sections, figures and tables. In contrast to the previous work of
Hou et al. (2019), we exclude the captions of figures and tables because it is
unlikely that a methodology name will only appear in a figure or a table
caption without occurring within the text of a document.

####  5.1.2 Method Categories

The broad-level method categories that our factored models use (Equations 2
and 3) is obtained from the PwC metadata. In particular, each document in the
PwC corpus is associated with a broad-level category or topic (in our case
$m=7$ as mentioned in Section 3.1.1 and 3.1.2), which is one of General (GEN),
Computer Vision (CV), Sequence-to-Sequence Modeling (SEQ), Reinforcement
Learning (RL), NLP, Audio & Speech (AUDIO), and Graph-based Modeling (GRAPH).
These document-level tag values are assigned to each constituent sentence of a
document. Figure 3(a) shows an example of methodology instances from the
‘General’ category of the PwC KB.

####  5.1.3 Document Representation and Weak Labeling for Training

We first segment an article into different sections, such as ‘Introduction’,
‘Experiments’, etc. The text from some of these sections is more likely to
contain the entities that are sought to be extracted. Prior work used the
content from four different sections of a paper, namely the title, abstract,
experiments, and table data to identify tasks, datasets and evaluation
measures and their values Hou et al. (2019); they named their document
representation as DocTAET: Title, Abstract, ExpSetup, and TableInfo. However,
different from Hou et al. (2019), our task is to extract the methodology
components from a paper; we, therefore, represent a paper as concatenated
chunks of text from the following sections: ‘Abstract’, ‘Introduction’,
‘Methodology’, ‘Experiments’ and ‘Results’, and use their initials as per the
convention of Hou et al. (2019) to name the representation as DocAIMER. Note
that different from DocTAET, in DocAIMER we exclude the TableInfo field and
include the content from the Introduction; this is because the method names
are more likely to be present in the abstract, introduction, methodology, the
experiment and the result sections of a paper, and are not likely to occur
exclusively in the captions (without these concepts being mentioned in the
text).

Different from DocTAET, we do not include data from tables because our
intention is not to extract the evaluation metrics or scores. Instead,
different from the DocTAET representation, we include the ‘Introduction’
section since it is likely that it develops the motivation for a paper’s
methodology using references to other method names in the literature.
Following the argument of Jain et al. (2020), we also leave out the ‘Related
Works’ since it primarily describes earlier published works rather than the
current paper’s work. Similarly, we include the experiment setup section
because it is likely to be comprised of the more fine-grained architectural
details of a proposed methodology, e.g., information on the optimizer,
regularization, loss function, etc.

To identify the textual content of each paper section, we use a simple rule-
based lookup to exclude sections that are not likely to be one of the AIMER-
type sections, e.g., we exclude sections with names ‘Background’, ‘Related
work’, ‘Conclusions’, ‘Conclusions and Future work’ etc. For reproducibility,
we release the dataset for research purposes.

| Model |  90:10 split on $\leq$2017 |  Train: $\leq$ 2017, Test: $>$ 2017
---|---|---|---
(r)3-5 (r)6-8 | Prec. | Recall | F-score | Prec. | Recall | F-score
Baselines | BiLSTM | 0.7281 | 0.5018 | 0.5941 | 0.6095 | 0.1960 | 0.2966
BiLSTMCRF | 0.8257 | 0.5666 | 0.6720 | 0.6460 | 0.1933 | 0.2976
SciBERT | 0.9632 | 0.9863 | 0.9746 | 0.3722 | 0.2380 | 0.2903
SciBERTCRF | 0.9832 | 0.9881 | 0.9857 | 0.4383 | 0.2452 | 0.3145
| SciREX | 0.9831 | 0.9887 | 0.9859 | 0.4604 | 0.2177 | 0.2956
Ours |  SciBERTCRF-DFG | 0.9821 | 0.9894 | 0.9856 | 0.5789 | 0.1907 | 0.2869
SciBERTCRF-DFGB | 0.9790 | 0.9836 | 0.9813 | 0.5784 | 0.2365 | 0.3357
SciBERTCRF-LFG | 0.9821 | 0.9887 | 0.9854 | 0.5110 | 0.2236 | 0.3111
SciBERTCRF-LFGB | 0.9816 | 0.9891 | 0.9853 | 0.4742 | 0.2690 | 0.3433
Ablations | SciBERT-DFG | 0.9658 | 0.9886 | 0.9771 | 0.4930 | 0.1418 | 0.2203
SciBERT-DFGB | 0.9646 | 0.9754 | 0.9700 | 0.4363 | 0.1065 | 0.1712
SciBERT-LFG | 0.9616 | 0.9845 | 0.9729 | 0.4429 | 0.1938 | 0.2696
SciBERT-LFGB | 0.9628 | 0.9859 | 0.9742 | 0.4271 | 0.2590 | 0.3225
|  |  |  |  |  |  |
Table 3: A comparison between the models investigated for the methodology
extraction task. Evaluated conducted with two different train-test splits -
(i) conventional 90:10 split, and (ii) chronological splitting corresponding
to a few-shot setup. In few-shot setup, the improvements of the best results
obtained with SciBERTCRF-LFGB(bold-faced underlined) vs. the best performing
baseline, SciBERTCRF(bold-faced), are significant (t-test with 95%
confidence).

###  5.2 Research Questions

As discussed in Section 1, the rapid growth of scientific disciplines has led
to a frequent introduction of new methodologies in the literature. In context
of this, we investigate our first research question,

  * •

RQ-1:How effectively can methodology extraction be conducted under the
challenging settings of few-shot data for training (unseen or infrequently
seen methodology names during training), and evolving contexts of the
mentions?

We compare the performance of various models (baselines, our proposed and
their ablations; see Section 5.3.2) to answer this question.

Our second research question is directed towards finding answer to the
question on what variant of the factored approach - input data partitioned or
label partitioned (Figure LABEL:fig:cg_model), works particularly well for the
methodology extraction task. In particular,

  * •

RQ-2: What is the most effective way of leveraging the domain or category of a
scientific article for training a sequence labeling model in a factorised
manner?

In the next research question, we aim to understand whether including silver-
standard data may help to improve a model (see Section 4).

  * •

RQ-3: How effectively can models be updated with their own predictions
included as silver-standard data for the task of methodology extraction from
scientific documents?

###  5.3 Methods Investigated

In this section, we describe the details of each method investigated in our
experiments.

####  5.3.1 Baselines

In order to evaluate the effectiveness of our proposed novel factored-based
models, we compare our methodologies with the following baseline methods. The
baselines we have chosen are well-established and widely used in this field,
providing a solid point of reference for our experiments (Beltagy et al.,
2019; Schweter and Akbik, 2020; Jain et al., 2020).

  * •

BiLSTM: Since we approach our methodology extraction task as a sequence
labeling task (as discussed in Section 3), we employ a BiLSTM-based approach
as it is considered to a standard technique for supervised information
extraction widely used in the literature (Cho and Lee, 2019). In particular,
we employ a standard BiLSTM-based approach that operates with word embedded
vectors as input. Specifically, for word embedding we employed GloVe
(Pennington et al., 2014) with the dimensionality set to 100.

  * •

BiLSTMCRF (Panchendrarajan and Amaresan, 2018): This a variant of BiLSTM where
an additional CRF decoder layer is applied for the methodology extraction task
with the other settings of BiLSTM remaining identical. It has been shown in
the literature that CRF used in combination with BiLSTM produces better
results for sequence labeling tasks, such as NER (Zeng et al., 2017).

  * •

SciBERT (Beltagy et al., 2019): SciBERT is a widely used pretrained language
model trained on scientific documents collections, and has been shown to yield
state-of-the-art performance in information extraction tasks for the
scientific domain Beltagy et al. (2019). For our experiments, we fine-tune the
standard SciBERT model on our labeled data (Section 5.1.3) with the sequence
labeling loss (Equation 1).

  * •

SciBERTCRF (Schweter and Akbik, 2020): Similar to augmenting BiLSTM with CRF,
in this approach we augment SciBERT with a CRF layer.

  * •

SciREX (Jain et al., 2020): This approach, as proposed in Jain et al. (2020),
augments the SciBERT model with a layer of BiLSTM and CRF. The overall model
is trained in an end-to-end manner with the sequence labeling loss of Equation
1.

The difference of our proposed approach with the baseline methods enumerated
above is that none of them use the category label information towards
addressing the methodology extraction task. This means that better results
obtained with our methods can hence be attributed to this missing factor.

####  5.3.2 Our Proposed Methodologies

For all our proposed factor approaches, we employ the CRF layer in the model
(the ones without the CRF layer are presented as ablations). We also employ
different levels of granularity for a thorough investigation of the factored
models; details follow.

  * •

SciBERTCRF-DFG: As per Section 3.1.1, we fine-tune $m=7$ of different
$f_{\\_}{\theta_{\\_}i}$s (Equation 2) using 7 individual categories (Section
5.1.2) of data instances. This approach routes a test instance sentence to the
appropriate model and hence the assumption is that the category information is
known during inference time (in our experiment setting, this is available from
the PwC metadata).

  * •

SciBERTCRF-DFGB: This approach is similar to SciBERTCRF-DFG, except that it
uses the category information in a more coarse-grained manner. To be more
precise, we merge together the 6 non-generic categories into one category.
This means that along with the ‘GEN’ category, we work with $m=2$ categories
(hence the suffix B to denote binary). This approach thus follow a middle
course between SciBERTCRF and SciBERTCRF-DFG in terms of the granularity of
the category information usage. This actually leads to a more balanced
partitioning of the data input space as can be seen from Figure
LABEL:fig:percat-stats that the ‘GEN’ category alone comprises a large
proportion of the data instances.

  * •

SciBERTCRF-LFG: Different from SciBERTCRF-DFG (which involves input
partitioning), this approach is the fine-grained version of the label
partitioning model as discussed in Section 3.1.2. Similar to SciBERTCRF-DFG ,
this uses more $m=7$ fine-grained labels, one for each category (Equation 3)
to fine-tune a sequence labeling model in an end-to-end manner.

  * •

SciBERTCRF-LFGB 444Our source code and dataset are available at
https://github.com/Madhu000/ie-nlplit.git: Again, similar to SciBERTCRF-DFGB,
this takes a middle path between SciBERTCRF and SciBERTCRF-LFG in the sense
that $m=2$ types of labels are used (GEN vs. the rest merged into one).
Similar to SciBERTCRF-DFGB, this method also takes a more balanced course
towards addressing the task specific to the PwC dataset, which is dominated by
the GEN category.

For each proposed method outlined above we derive its corresponding ablated
version by removing the CRF layer. We name each approach by removing the ‘CRF’
subscript from its name, e.g., SciBERT-LFGB instead of SciBERTCRF-LFGB.

#####  Implementation Details:

Implementation of the baselines and the proposed approaches uses the FLAIR
framework555https://github.com/flairNLP/flair. All the models were trained on
Google Colab PRO plus. In terms of the common neural network settings, we used
AdamW (Loshchilov and Hutter, 2019) as the optimizer with a learning rate of
$5e^{-5}$ and a stopping criterion as mentioned in Conneau et al. (2020); the
training batch size used was 32.

##  6 Results

In this section, we present a comprehensive analysis of our research questions
as mentioned in Section 5.2. We mainly focus on providing detailed
observations and insights for our research questions.

###  6.1 Model Comparisons

#####  Main results.

In relation to RQ-1, Table 3 presents a comparison between the models
investigated (best results in each group, i.e., the baselines or the novel
models, which also includes the ablations, both bold-faced; additionally the
overall best results across the two groups are underlined). It can be observed
from Table 3 that all baseline models along with our proposed factored
extensions work satisfactorily well with the standard percentage-based
train:test split over the dataset of papers till 2017 (which is an easier
setup and widely reported in the literature, e.g., Hou et al. (2019); Beltagy
et al. (2019); Hou et al. (2021); Jain et al. (2020)). The baseline models
mostly works better than the proposed factored versions when it comes to the
standard train:test (in-domain) evaluation.

However, under the challenging situation of a more realistic out-domain setup
(chronologically partitioned data), it is observed that the factored versions
of the models (our proposed ones) work better (this is seen from the fact that
the recall and the F-score values in the lower group are underlined). As a
general observation, it is also seen that the factored models lead to better
recall values. An additional observation is that the CRF-based models work
consistently better than their non-CRF counterparts.

#####  Granularity of the factored models.

Turning our attention to RQ-2 (which explores what levels of granularity in
the factored models work most effectively), we first observe from Table 3 that
the binary version of the label-based factoring (SciBERTCRF-LFGB) performs the
best for the few-shot (chronological split) setup, which means that a balanced
factoring works particularly well for this more realistic experiment setup.
Further analysis presented in Figure LABEL:fig:percat-stats shows that this
effectiveness measure of SciBERTCRF-LFGB is actually computed over a
substantial proportion of new methodologies encountered during the inference
phase (zero-shot setup). In fact, for the SEQ and RL categories, this
proportion is even higher than $0.75$.

#####  Category-specific analysis.

Figure LABEL:fig:result_analysis presents a category specific analysis of the
results for all the proposed models including the baselines. A few key
observations are as follows.

  * •

SciBERTCRF-LFGB consistently performs well across all the 7 different
categories.

  * •

SciBERTCRF-LFGB performs very well on ‘Audio’ domain (cf. Figure
LABEL:fig:percat-stats) in spite of introduction of a significant number of
new methodologies in the post-2017 era.

  * •

Proposed SciBERTCRF-DFGB produces high F-scores across each category. A common
intuition behind it is that we develop the SciBERTCRF-DFGB framework by
partitioning the input data and that might have enabled it to learn more
category specific information from the partitioned data samples.

These observations suggest that the factored extension of SciBERTCRF is able
to generalize well for most scientific disciplines.

###  6.2 Feedback-based Model Updates

To address RQ-3 we employ the retraining strategy of Section 4, i.e., we use a
model’s own predictions on the test set as feedback information to
incrementally update its parameters. In particular, we use the best performing
model of Table 3, namely SciBERTCRF-LFGB for this set of experiments. To gain
further insights into the model’s behaviour under the zero-shot setup, we
partition the set of entities in our test-set (post-2017 as outlined in
Section 5.1) into two groups - one where the timestamp of the first occurrence
of the entity precedes 2018, and the other includes only those entities which
appear from 2018. Moreover, for an oracle-based comparison we also include
results where the models are updated with gold-standard data.

From Figure LABEL:fig:result_rolling we observe that the model SciBERTCRF-LFGB
trained with silver-standard data continually improve over the years (as can
be seen from the monotonically increasing values of the middle lines). This
line corresponding to the silver-standard setup, as expected, is sandwiched
between the two extremes of updates with gold-standard data (the top-line) and
no updates (the bottom line). Particularly interesting is the observation for
the zero-shot setup (Figure LABEL:fig:result_rolling_2), where the middle line
is seen to be more closely following the top line, thus indicating the
effectiveness of the model updates for new methodology names.

Additionally, to see if this trend is similar across the intermediate years
(and not only 2021) we also evaluate the models over the test sets
corresponding to the intermediate timestamps (i.e., years 2019 and 2020).
Figure LABEL:fig:result_rolling_second shows that this sandwiching trend is
consistent for the intermediate years as well, again indicating the robustness
of the feedback method as measured across different test sets. One important
point to note is that the chart heights across the different intermediate
years are not comparable because the evaluation uses different test sets.

###  6.3 Additional Analysis

#####  Changes in contexts of scientific methods.

To illustrate that it is indeed challenging for the models to predict newer
entities and to demonstrate the necessity for model updates, we now
demonstrate how does the context of a specific word, namely ‘transformer’,
changes over time. While it is seen that the prevalent context terms in the
training data (pre-2018 entities) comprises terms such as ‘architecture’,
‘self-attention’, ‘encoder’ etc., the ones in the test set, on the other hand,
correspond to ‘BERT’, ‘VIT’, ‘GPT-2’ etc. This shows that there is a
substantial change in the contexts around the mentions of the word
‘transformer’, which in turn shows that firstly, the base model SciBERTCRF-
LFGB(trained till 2017) performs well in a challenging setup (Table 3), and
secondly, that the feedback with the predicted entities enable the model to
capture some of this new context.

#####  Post-hoc manual analysis.

Additionally, we created a small manually annotated benchmark of data samples
from the post-2017 partition of the dataset comprising a total of 285
sentences (again, we will release this as a part of the dataset). We then
evaluate the best performing model of Table 3, namely SciBERTCRF-LFGB, with
this small manually annotated dataset. We observed that the model yields a
precision of 0.623, a recall of 0.2099 and an F-score of 0.314, the numbers
being comparable to those reported in Table 3.

###  6.4 Methodological Implications

From the investigations carried out in this work, we observe from Table 3 that
our proposed novel factored-based approach, SciBERTCRF-LFGB, achieves state-
of-the-art performance compared to the previous model for extracting emerging
methodology names from scientific AI literature in our proposed chronological
evaluation setup. Furthermore, we have demonstrated that the models trained on
historical data can be retrained incrementally using the predictions of a
previously trained model for newly published articles. This approach reduces
the burden of training a model from scratch and lowers annotation costs. As
discussed in Section , our incremental training strategy significantly
improves the extraction of newer methodology names, which are typically
introduced in recent literature.

##  7 Conclusions

In this paper we presented a thorough investigation of the feasibility of
applying supervised approaches trained on historical data in automatically
extracting novel methodology component names from scientific articles.
Focusing specifically on the AI domain, we propose two ways of leveraging the
category information of these papers via a factored approach. More
specifically, while one of these methods trains an individual model for each
category, the other approach constructs more fine-grained labels for training.
Our experiments show that both these variants outperform a standard BERT-based
fine-tuning approach. While the input data partitioning approach helps
increase precision, training with fine-grained labels, on the other hand,
leads to increase in the recall and the F-score measures. To the best of our
knowledge, this is the first contribution towards extracting emerging
methodology names from AI articles in zero-shot and few-shot scenario.

In future, we will expand our work beyond the AI domain to see how the method
generalizes to a diverse range of characteristically different domains of
study, e.g., economics, physics, etc. We will also explore clustering based
approaches to propose a data-driven factored model. Another direction of
investigation would be to find out effective ways of filtering the silver-
standard data used for incrementally updating the extraction models.

## Declaration of Competing Interest

We do not have any conflict of interests regarding our submission.

## References

  * Akbik et al. (2018) Akbik, A., Blythe, D., Vollgraf, R., 2018.  Contextual string embeddings for sequence labeling, in: Proceedings of the 27th International Conference on Computational Linguistics, Association for Computational Linguistics, Santa Fe, New Mexico, USA. pp. 1638–1649.  URL: https://aclanthology.org/C18-1139.
  * Ali et al. (2020) Ali, M.A., Sun, Y., Li, B., Wang, W., 2020.  Fine-grained named entity typing over distantly supervised data based on refined representations, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7391–7398.
  * Allan et al. (1998) Allan, J., Papka, R., Lavrenko, V., 1998.  On-line new event detection and tracking, in: Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, pp. 37–45.
  * Ammar et al. (2017) Ammar, W., Peters, M.E., Bhagavatula, C., Power, R., 2017.  The ai2 system at semeval-2017 task 10 (scienceie): semi-supervised end-to-end entity and relation extraction, in: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pp. 592–596.
  * Asghari et al. (2022) Asghari, M., Sierra-Sosa, D., Elmaghraby, A.S., 2022.  Biner: A low-cost biomedical named entity recognition.  Information Sciences 602, 184–200.
  * Athar and Teufel (2012) Athar, A., Teufel, S., 2012.  Context-enhanced citation sentiment detection, in: Proceedings of the 2012 conference of the North American chapter of the Association for Computational Linguistics: Human language technologies, pp. 597–601.
  * Augenstein and Søgaard (2017) Augenstein, I., Søgaard, A., 2017.  Multi-task learning of keyphrase boundary classification, in: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, Vancouver, Canada. pp. 341–346.  URL: https://aclanthology.org/P17-2054, doi:10.18653/v1/P17-2054.
  * Bedi et al. (2022) Bedi, M., Pandey, T., Bhatia, S., Chakraborty, T., 2022.  Why did you not compare with that? identifying papers for use as baselines, in: ECIR (1), Springer. pp. 51–64.
  * Beltagy et al. (2019) Beltagy, I., Lo, K., Cohan, A., 2019.  SciBERT: A pretrained language model for scientific text, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China. pp. 3615–3620.  URL: https://aclanthology.org/D19-1371, doi:10.18653/v1/D19-1371.
  * Cao et al. (2019) Cao, Y., Hu, Z., Chua, T.s., Liu, Z., Ji, H., 2019.  Low-resource name tagging learned with weakly labeled data, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China.  URL: https://aclanthology.org/D19-1025, doi:10.18653/v1/D19-1025.
  * Chaudhuri et al. (2022) Chaudhuri, A., Sarma, M., Samanta, D., 2022.  Share: Designing multiple criteria-based personalized research paper recommendation system.  Information Sciences 617, 41–64.
  * Chen et al. (2021) Chen, H., Takamura, H., Nakayama, H., 2021.  Scixgen: A scientific paper dataset for context-aware text generation, in: Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 1483–1492.
  * Chiu and Nichols (2016) Chiu, J.P., Nichols, E., 2016.  Named entity recognition with bidirectional lstm-cnns.  Transactions of the association for computational linguistics 4, 357–370.
  * Cho and Lee (2019) Cho, H., Lee, H., 2019.  Biomedical named entity recognition using deep neural networks with contextual information.  BMC bioinformatics 20, 1–11.
  * Cohan et al. (2019) Cohan, A., Ammar, W., van Zuylen, M., Cady, F., 2019.  Structural scaffolds for citation intent classification in scientific publications, in: Proceedings of NAACL-HLT, pp. 3586–3596.
  * Conneau et al. (2020) Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., Stoyanov, V., 2020.  Unsupervised cross-lingual representation learning at scale, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online. pp. 8440–8451.  URL: https://aclanthology.org/2020.acl-main.747, doi:10.18653/v1/2020.acl-main.747.
  * Dai et al. (2019) Dai, D., Xiao, X., Lyu, Y., Dou, S., She, Q., Wang, H., 2019.  Joint extraction of entities and overlapping relations using position-attentive sequence labeling, in: Proceedings of the AAAI conference on artificial intelligence, pp. 6300–6308.
  * Das et al. (2017) Das, A., Ganguly, D., Garain, U., 2017.  Named entity recognition with word embeddings and wikipedia categories for a low-resource language.  ACM Trans. Asian Low Resour. Lang. Inf. Process. 16, 18:1–18:19.  URL: https://doi.org/10.1145/3015467, doi:10.1145/3015467.
  * Devlin et al. (2019) Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota. pp. 4171–4186.  URL: https://aclanthology.org/N19-1423, doi:10.18653/v1/N19-1423.
  * Dutta et al. (2020) Dutta, S., Das, D., Chakraborty, T., 2020.  Changing views: Persuasion modeling and argument extraction from online discussions.  Information Processing & Management 57, 102085.
  * Endris et al. (2017) Endris, K.M., Giménez-García, J.M., Thakkar, H., Demidova, E., Zimmermann, A., Lange, C., Simperl, E., 2017.  Dataset reuse: an analysis of references in community discussions, publications and data, in: Proceedings of the Knowledge Capture Conference, pp. 1–4.
  * Fan et al. (2020) Fan, B., Fan, W., Smith, C., et al., 2020.  Adverse drug event detection and extraction from open data: A deep learning approach.  Information Processing & Management 57, 102131.
  * Fei et al. (2019) Fei, H., Ren, Y., Ji, D., 2019.  Recognizing nested named entity in biomedical texts: a neural network model with multi-task learning, in: 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), IEEE. pp. 376–381.
  * Gábor et al. (2017) Gábor, K., Buscaldi, D., Schumann, A.K., QasemiZadeh, B., Zargayouna, H., Charnois, T., 2017.  Semeval-2018 task 7: Semantic relation extraction and classification in scientific papers, in: International Workshop on Semantic Evaluation (SemEval-2018).
  * Ghosh et al. (2022) Ghosh, M., Santra, P., Iqbal, S.A., Basuchowdhuri, P., 2022.  Astro-mt5: Entity extraction from astrophysics literature using mt5 language model, in: Proceedings of the first Workshop on Information Extraction from Scientific Publications, pp. 100–104.
  * Heist and Paulheim (2021) Heist, N., Paulheim, H., 2021.  Information extraction from co-occurring similar entities, in: Proceedings of the Web Conference 2021, pp. 3999–4009.
  * Hoory et al. (2021) Hoory, S., Feder, A., Tendler, A., Erell, S., Peled-Cohen, A., Laish, I., Nakhost, H., Stemmer, U., Benjamini, A., Hassidim, A., et al., 2021.  Learning and evaluating a differentially private pre-trained language model, in: Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 1178–1189.
  * Hou et al. (2019) Hou, Y., Jochim, C., Gleize, M., Bonin, F., Ganguly, D., 2019.  Identification of tasks, datasets, evaluation metrics, and numeric scores for scientific leaderboards construction, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy. pp. 5203–5213.  URL: https://aclanthology.org/P19-1513, doi:10.18653/v1/P19-1513.
  * Hou et al. (2021) Hou, Y., Jochim, C., Gleize, M., Bonin, F., Ganguly, D., 2021.  TDMSci: A specialized corpus for scientific literature entity tagging of tasks datasets and metrics, in: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Association for Computational Linguistics, Online. pp. 707–714.  URL: https://aclanthology.org/2021.eacl-main.59, doi:10.18653/v1/2021.eacl-main.59.
  * Hu et al. (2021) Hu, Y., Zhu, Y., Zhang, J., Zheng, C., Zhou, J., 2021.  Toward fully exploiting heterogeneous corpus: A decoupled named entity recognition model with two-stage training, in: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1641–1652.
  * Huang et al. (2021) Huang, X., Chen, Y., Wu, S., Zhao, J., Xie, Y., Sun, W., 2021.  Named entity recognition via noise aware training mechanism with data filter, in: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4791–4803.
  * Jain et al. (2020) Jain, S., van Zuylen, M., Hajishirzi, H., Beltagy, I., 2020.  SciREX: A challenge dataset for document-level information extraction, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.
  * Ji et al. (2020) Ji, D., Tao, P., Fei, H., Ren, Y., 2020.  An end-to-end joint model for evidence information extraction from court record document.  Information Processing & Management 57, 102305.
  * Ji et al. (2021) Ji, Z., Xia, T., Han, M., Xiao, J., 2021.  A neural transition-based joint model for disease named entity recognition and normalization, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Association for Computational Linguistics, Online. pp. 2819–2827.  URL: https://aclanthology.org/2021.acl-long.219, doi:10.18653/v1/2021.acl-long.219.
  * Jiang et al. (2022) Jiang, T., Bu, C., Zhu, Y., Wu, X., 2022.  Combining embedding-based and symbol-based methods for entity alignment.  Pattern Recognition 124, 108433.
  * Jurgens et al. (2018) Jurgens, D., Kumar, S., Hoover, R., McFarland, D., Jurafsky, D., 2018.  Measuring the evolution of a scientific field through citation frames.  Transactions of the Association for Computational Linguistics 6, 391–406.
  * Kabongo et al. (2023) Kabongo, S., D’Souza, J., Auer, S., 2023.  Zero-shot entailment of leaderboards for empirical ai research.  arXiv preprint arXiv:2303.16835 .
  * Kabongo et al. (2021) Kabongo, S., D’Souza, J., Auer, S., 2021.  Automated mining of leaderboards for empirical ai research, in: Towards Open and Trustworthy Digital Societies: 23rd International Conference on Asia-Pacific Digital Libraries, ICADL 2021, Virtual Event, December 1–3, 2021, Proceedings 23, Springer. pp. 453–470.
  * Kardas et al. (2020) Kardas, M., Czapla, P., Stenetorp, P., Ruder, S., Riedel, S., Taylor, R., Stojnic, R., 2020.  AxCell: Automatic extraction of results from machine learning papers, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online. pp. 8580–8594.  URL: https://aclanthology.org/2020.emnlp-main.692, doi:10.18653/v1/2020.emnlp-main.692.
  * Kelley and Gillan (2022) Kelley, S.W., Gillan, C.M., 2022.  Using language in social media posts to study the network dynamics of depression longitudinally.  Nature communications 13, 1–11.
  * Lahiri et al. (2023) Lahiri, A., Sanyal, D.K., Mukherjee, I., 2023.  Citeprompt: Using prompts to identify citation intent in scientific papers.  arXiv:2304.12730.
  * Li et al. (2020a) Li, B., Liu, S., Sun, Y., Wang, W., Zhao, X., 2020a.  Recursively binary modification model for nested named entity recognition, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 8164--8171.
  * Li et al. (2021a) Li, J., Fei, H., Liu, J., Wu, S., Zhang, M., Teng, C., Ji, D., Li, F., 2021a.  Unified named entity recognition as word-word relation classification.  arXiv preprint arXiv:2112.10070 .
  * Li et al. (2020b) Li, J., Sun, A., Han, J., Li, C., 2020b.  A survey on deep learning for named entity recognition.  IEEE Transactions on Knowledge and Data Engineering 34, 50--70.
  * Li et al. (2020c) Li, P.H., Fu, T.J., Ma, W.Y., 2020c.  Why attention? analyze bilstm deficiency and its remedies in the case of ner, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 8236--8244.
  * Li et al. (2021b) Li, R., Li, D., Yang, J., Xiang, F., Ren, H., Jiang, S., Zhang, L., 2021b.  Joint extraction of entities and relations via an entity correlated attention neural model.  Information Sciences 581, 179--193.
  * Liang et al. (2020) Liang, C., Yu, Y., Jiang, H., Er, S., Wang, R., Zhao, T., Zhang, C., 2020.  Bond: Bert-assisted open-domain named entity recognition with distant supervision, in: Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1054--1064.
  * Lin et al. (2020) Lin, B.Y., Lee, D.H., Shen, M., Moreno, R., Huang, X., Shiralkar, P., Ren, X., 2020.  TriggerNER: Learning with entity triggers as explanations for named entity recognition, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online. pp. 8503--8511.  URL: https://aclanthology.org/2020.acl-main.752, doi:10.18653/v1/2020.acl-main.752.
  * Liu et al. (2021) Liu, J., Gao, L., Guo, S., Ding, R., Huang, X., Ye, L., Meng, Q., Nazari, A., Thiruvady, D., 2021.  A hybrid deep-learning approach for complex biochemical named entity recognition.  Knowledge-Based Systems 221, 106958.
  * Liu et al. (2020) Liu, S., Sun, Y., Li, B., Wang, W., Zhao, X., 2020.  Hamner: headword amplified multi-span distantly supervised method for domain specific named entity recognition, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 8401--8408.
  * Lo et al. (2020) Lo, K., Wang, L.L., Neumann, M., Kinney, R., Weld, D.S., 2020.  S2orc: The semantic scholar open research corpus, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4969--4983.
  * Loreto et al. (2012) Loreto, V., Mukherjee, A., Tria, F., 2012.  On the origin of the hierarchy of color names.  Proceedings of the National Academy of Sciences 109, 6819--6824.
  * Loshchilov and Hutter (2019) Loshchilov, I., Hutter, F., 2019.  Decoupled weight decay regularization, in: International Conference on Learning Representations.  URL: https://openreview.net/forum?id=Bkg6RiCqY7.
  * Luan et al. (2018) Luan, Y., He, L., Ostendorf, M., Hajishirzi, H., 2018.  Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Brussels, Belgium. pp. 3219--3232.  URL: https://aclanthology.org/D18-1360, doi:10.18653/v1/D18-1360.
  * Luan et al. (2017) Luan, Y., Ostendorf, M., Hajishirzi, H., 2017.  Scientific information extraction with semi-supervised neural tagging, in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Copenhagen, Denmark. pp. 2641--2651.  URL: https://aclanthology.org/D17-1279, doi:10.18653/v1/D17-1279.
  * Luo et al. (2020) Luo, Y., Yin, L., Bai, W., Mao, K., 2020.  An appraisal of incremental learning methods.  Entropy 22, 1190.
  * Ma et al. (2021) Ma, J., Yan, Z., Li, C., Zhang, Y., 2021.  Frustratingly simple few-shot slot tagging, in: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1028--1033.
  * Ma et al. (2023) Ma, Q., Yu, L., Chen, H., Yan, J., Lin, Z., 2023.  Sequence labeling with mlta: Multi-level topic-aware mechanism.  Information Sciences 637, 118934.
  * Ma and Hovy (2016) Ma, X., Hovy, E., 2016.  End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF, in: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Berlin, Germany. pp. 1064--1074.  URL: https://aclanthology.org/P16-1101, doi:10.18653/v1/P16-1101.
  * Maity et al. (2012) Maity, S.K., Manoj, T.V., Mukherjee, A., 2012.  Opinion formation in time-varying social networks: The case of the naming game.  Physical Review E 86, 036110.
  * Mayhew et al. (2020) Mayhew, S., Nitish, G., Roth, D., 2020.  Robust named entity recognition with truecasing pretraining, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 8480--8487.
  * Mengge et al. (2020) Mengge, X., Yu, B., Zhang, Z., Liu, T., Zhang, Y., Wang, B., 2020.  Coarse-to-Fine Pre-training for Named Entity Recognition, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online. pp. 6345--6354.  URL: https://aclanthology.org/2020.emnlp-main.514, doi:10.18653/v1/2020.emnlp-main.514.
  * Mintz et al. (2009) Mintz, M., Bills, S., Snow, R., Jurafsky, D., 2009.  Distant supervision for relation extraction without labeled data, in: Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, Association for Computational Linguistics, Suntec, Singapore. pp. 1003--1011.  URL: https://aclanthology.org/P09-1113.
  * Mitra et al. (2014) Mitra, S., Mitra, R., Riedl, M., Biemann, C., Mukherjee, A., Goyal, P., 2014.  That’s sick dude!: Automatic identification of word sense change across different timescales, in: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Baltimore, Maryland. pp. 1020--1029.  URL: https://aclanthology.org/P14-1096, doi:10.3115/v1/P14-1096.
  * Mondal et al. (2021) Mondal, I., Hou, Y., Jochim, C., 2021.  End-to-end construction of nlp knowledge graph, in: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1885--1895.
  * Mukherjee et al. (2011) Mukherjee, A., Tria, F., Baronchelli, A., Puglisi, A., Loreto, V., 2011.  Aging in language dynamics.  PLoS One 6, e16677.
  * Nasar et al. (2021) Nasar, Z., Jaffry, S.W., Malik, M.K., 2021.  Named entity recognition and relation extraction: State-of-the-art.  ACM Computing Surveys (CSUR) 54, 1--39.
  * Nayak and Ng (2020) Nayak, T., Ng, H.T., 2020.  Effective modeling of encoder-decoder architecture for joint entity and relation extraction, in: Proceedings of the AAAI conference on artificial intelligence, pp. 8528--8535.
  * Panchendrarajan and Amaresan (2018) Panchendrarajan, R., Amaresan, A., 2018.  Bidirectional lstm-crf for named entity recognition, in: Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation.
  * Patra and Moniz (2019) Patra, B., Moniz, J.R.A., 2019.  Weakly supervised attention networks for entity recognition, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 6268--6273.
  * Peng et al. (2019) Peng, M., Xing, X., Zhang, Q., Fu, J., Huang, X., 2019.  Distantly supervised named entity recognition using positive-unlabeled learning, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy. pp. 2409--2419.  URL: https://aclanthology.org/P19-1231, doi:10.18653/v1/P19-1231.
  * Pennington et al. (2014) Pennington, J., Socher, R., Manning, C., 2014.  GloVe: Global vectors for word representation, in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Doha, Qatar. pp. 1532--1543.  URL: https://aclanthology.org/D14-1162, doi:10.3115/v1/D14-1162.
  * Qu et al. (2022) Qu, X., Zeng, J., Liu, D., Wang, Z., Huai, B., Zhou, P., 2022.  Distantly-supervised named entity recognition with adaptive teacher learning and fine-grained student ensemble.  arXiv e-prints , arXiv--2212.
  * Radmard et al. (2021) Radmard, P., Fathullah, Y., Lipani, A., 2021.  Subsequence based deep active learning for named entity recognition, in: ACL/IJCNLP (1), Association for Computational Linguistics. pp. 4310--4321.
  * Riedel et al. (2010) Riedel, S., Yao, L., McCallum, A., 2010.  Modeling relations and their mentions without labeled text, in: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer. pp. 148--163.
  * Safder et al. (2020) Safder, I., Hassan, S.U., Visvizi, A., Noraset, T., Nawaz, R., Tuarob, S., 2020.  Deep learning-based extraction of algorithmic metadata in full-text scholarly documents.  Information processing & management 57, 102269.
  * Saier and Färber (2020) Saier, T., Färber, M., 2020.  unarxive: a large scholarly data set with publications’ full-text, annotated in-text citations, and links to metadata.  Scientometrics 125, 3085--3108.
  * Scells and Zuccon (2018) Scells, H., Zuccon, G., 2018.  Generating better queries for systematic reviews, in: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, pp. 475--484.
  * Schweter and Akbik (2020) Schweter, S., Akbik, A., 2020.  FLERT: Document-level features for named entity recognition.  arXiv:2011.06993.
  * Shen et al. (2023) Shen, S., Liu, J., Lin, L., Huang, Y., Zhang, L., Liu, C., Feng, Y., Wang, D., 2023.  Sscibert: A pre-trained language model for social science texts.  Scientometrics 128, 1241--1263.
  * Singh et al. (2017a) Singh, G., Marshall, I.J., Thomas, J., Shawe-Taylor, J., Wallace, B.C., 2017a.  A neural candidate-selector architecture for automatic structured clinical text annotation, in: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 1519--1528.
  * Singh et al. (2017b) Singh, J., Zerr, S., Siersdorfer, S., 2017b.  Structure-aware visualization of text corpora, in: Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval, pp. 107--116.
  * Sun et al. (2021) Sun, K., Zhang, R., Mensah, S., Mao, Y., Liu, X., 2021.  Progressive multitask learning with controlled information flow for joint entity and relation extraction.  Association for the Advancement of Artificial Intelligence (AAAI) .
  * Sung et al. (2021) Sung, C., Goel, V., Marcheret, E., Rennie, S., Nahamoo, D., 2021.  Cnnbif: Cnn-based bigram features for named entity recognition, in: Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 1016--1021.
  * Toledo et al. (2019) Toledo, J.I., Carbonell, M., Fornés, A., Lladós, J., 2019.  Information extraction from historical handwritten document images with a context-aware neural model.  Pattern Recognition 86, 27--36.
  * Tong et al. (2021) Tong, Y., Chen, Y., Shi, X., 2021.  A multi-task approach for improving biomedical named entity recognition by incorporating multi-granularity information, in: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4804--4813.
  * Tsai et al. (2013) Tsai, C.T., Kundu, G., Roth, D., 2013.  Concept-based analysis of scientific literature, in: Proceedings of the 22nd ACM international conference on information & knowledge management, pp. 1733--1738.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I., 2017.  Attention is all you need.  Advances in neural information processing systems 30.
  * Viswanathan et al. (2021) Viswanathan, V., Neubig, G., Liu, P., 2021.  Citationie: Leveraging the citation graph for scientific information extraction, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 719--731.
  * Walker et al. (2008) Walker, E., Hernandez, A., Kattan, M., 2008.  Meta-analysis: Its strengths and limitations.  Cleveland Clinic journal of medicine 75, 431--9.  doi:10.3949/ccjm.75.6.431.
  * Wang et al. (2021a) Wang, D., Fan, H., Liu, J., 2021a.  Learning with joint cross-document information via multi-task learning for named entity recognition.  Information Sciences 579, 454--467.
  * Wang et al. (2020a) Wang, Y., Mukherjee, S., Chu, H., Tu, Y., Wu, M., Gao, J., Awadallah, A.H., 2020a.  Adaptive self-training for few-shot neural sequence labeling.  arXiv preprint arXiv:2010.03680 .
  * Wang et al. (2021b) Wang, Y., Mukherjee, S.S., Chu, H., Tu, Y., Wu, M., Gao, J., Awadallah, A.H., 2021b.  Meta self-training for few-shot neural sequence labeling, in: SIGKDD 2021 (Research Track).
  * Wang et al. (2020b) Wang, Y., Sun, C., Wu, Y., Yan, J., Gao, P., Xie, G., 2020b.  Pre-training entity relation encoder with intra-span and inter-span information, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1692--1705.
  * Xiao et al. (2020) Xiao, Y., Tan, C., Fan, Z., Xu, Q., Zhu, W., 2020.  Joint entity and relation extraction with a hybrid transformer and reinforcement learning based model, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 9314--9321.
  * Xu et al. (2021) Xu, Y., Huang, H., Feng, C., Hu, Y., 2021.  A supervised multi-head self-attention network for nested named entity recognition, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 14185--14193.
  * Yang et al. (2018) Yang, Y., Chen, W., Li, Z., He, Z., Zhang, M., 2018.  Distantly supervised ner with partial annotation learning and reinforcement learning, in: Proceedings of the 27th International Conference on Computational Linguistics, pp. 2159--2169.
  * Zeng et al. (2017) Zeng, D., Sun, C., Lin, L., Liu, B., 2017.  Lstm-crf for drug-named entity recognition.  Entropy 19, 283.
  * Zeng et al. (2020) Zeng, D., Zhang, H., Liu, Q., 2020.  Copymtl: Copy mechanism for joint extraction of entities and relations with multi-task learning, in: Proceedings of the AAAI conference on artificial intelligence, pp. 9507--9514.
  * Zeng et al. (2018) Zeng, X., He, S., Liu, K., Zhao, J., 2018.  Large scaled relation extraction with reinforcement learning, in: Proceedings of the AAAI Conference on Artificial Intelligence.
  * Zhang et al. (2019) Zhang, H., Boons, F., Batista-Navarro, R., 2019.  Whose story is it anyway? automatic extraction of accounts from news articles.  Information processing & management 56, 1837--1848.
  * Zhao et al. (2021) Zhao, X., Ding, H., Feng, Z., 2021.  Glara: Graph-based labeling rule augmentation for weakly supervised named entity recognition, in: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 3636--3649.


EVOLUTIONARY CONSTRAINTS:

GLUING IN A TOY MODEL

Ingemar Bengtsson∗

Istvan Rácz†

∗Stockholms Universitet, AlbaNova,

SE-106 91 Stockholm, Sverige

†HUN-REN Wigner RCP, H-1121 Budapest,

Konkoly-Thege Miklós út 29-33, Hungary

Abstract:

It is possible to solve the Einstein constraint equations as an evolutionary
rather than an elliptic system. Here we consider the Gauss constraint in
electrodynamics as a toy model for this, and produce an initial data set for
an electromagnetic pulse surrounded by vacuum.

1\. Introduction

How do we find initial data for a sourceless electric field that is confined
inside a ball of radius $R_{1}$, and vanishes outside some concentric ball
with radius $R_{2}>R_{1}$? It should be possible to produce a fair
approximation of such a solution by means of a pulse from a free-electron
laser, so we expect the solution to exist. The question has been raised [1, 2,
3] as a toy example of the gluing constructions that have been made for
initial data of Einstein’s equations. Here it will be considered as a toy
example of the advantages that can ensue if the underdetermined partial
differential equations that constrain the initial data—in this case Gauss’
law—are regarded as evolution equations instead of as elliptic equations. This
approach arose in the context of Einstein’s equations [4], and has scored some
successes there [5, 6, 7, 8], but here we will simply be concerned with smooth
solutions to the equation

$\nabla\cdot{\bf E}=0$ (1)

in flat space. The idea is to prescribe the value of the electric field on
some sphere in space, and then integrate outwards (or inwards) from there.
Thus the evolution takes place in space, not in time; the outcome of this
evolution is a set of initial data for Maxwell’s equations. An obvious
objection is that we seem to lose control over the asymptotic behaviour of the
initial data. What saves the day is that the equations are underdetermined,
and a large part of the solution will consist of freely specifiable data. With
suitable choices this will allow us to do what we want. The same issue arises
for the Einstein constraints, but for this we have to refer to the literature
[5, 6, 7, 8].

To begin with, and to get some ideas for the freely specifiable data that we
need, we will follow the treatment of ref. [1]. We introduce two smooth
functions $\chi=\chi(r)$ and $\psi=\psi(r)$ such that $\chi$ equals 1 when
$r<R_{1}$, $\chi=0$ when $r>R_{2}$, while $\psi$ equals zero outside this
interval. Smooth functions having these properties are easy to find. Let us
also introduce two divergence free vector fields ${\bf E}^{(1)}$ and ${\bf
E}^{(2)}$, and a function $u$. Then we make the Ansatz

${\bf E}=\chi{\bf E}^{(1)}+(1-\chi){\bf E}^{(2)}+\psi^{2}\nabla u\ .$ (2)

This agrees with ${\bf E}^{(1)}$ inside the small ball and it agrees with
${\bf E}^{(2)}$ outside the larger ball, and setting its divergence to zero
yields

$\nabla\cdot{\bf E}=\nabla\chi\cdot({\bf E}^{(1)}-{\bf
E}^{(2)})+\nabla\cdot(\psi^{2}\nabla u)=0\ .$ (3)

Our problem is solved if we can solve this equation for the function $u$. The
necessary existence theorem has been proved [1, 2, 3], but it rests on quite
sophisticated functional analysis and does not lead directly to an actual
construction of the field.

This is where an evolutionary formulation of Gauss’ law can help. We foliate
the region between the spheres with round spheres of radius $r$, having normal
vectors $n_{i}=x_{i}/r$. Then we decompose our vector fields into normal and
tangential parts,

$E_{i}=\alpha n_{i}+L_{i}\ ,\hskip 22.76219pt\alpha=n_{i}E_{i}\ .$ (4)

A calculation valid for a general foliation [9] then shows that

$\nabla\cdot{\bf E}={\cal L}_{\vec{n}}\alpha+\alpha
K+\bar{\nabla}_{i}L^{i}-\dot{n}^{i}L_{i}\ ,$ (5)

where ${\cal L}_{\vec{n}}$ denotes the Lie derivative in the direction of the
normal vector, $K$ is the mean curvature of the leaves of foliation,
$\bar{\nabla}$ is the covariant derivative on the leaves, and
$\dot{n}^{i}=n^{j}\nabla_{j}n^{i}$. In our case the leaves are round spheres
with $K=2/r$, the ‘time’ function is $r$, $\dot{n}_{i}$ vanishes, and the
equation simplifies to

$\nabla\cdot{\bf
E}=\partial_{r}\alpha+\frac{2}{r}\alpha+\bar{\nabla}_{i}L^{i}=\frac{1}{r^{2}}\partial_{r}(r^{2}\alpha)+\bar{\nabla}_{i}L^{i}\
.$ (6)

This is to be set equal to zero. The resulting equation is an evolution
equation for $\alpha$, while the tangential part $L_{i}$ is to regarded as
freely specifiable data. The underlying idea is applicable to much more
complex constraint equations, indeed it was first put forward in order to deal
with the Einstein constraints [4], but here our objective is to test the idea
in the simple case of Gauss’ law.

2\. Gluing initial data together

The question that arises is how to choose $L_{i}$, the part of the field that
is tangential to the leaves of foliation. Here the answer comes from the
Ansatz (2). The fields ${\bf E}^{(1)}$ and ${\bf E}^{(2)}$ are assumed to be
known, and it is understood that they are divergence free, so we can write

$\bar{\nabla}^{i}L_{i}^{(1)}=-\frac{1}{r^{2}}\partial_{r}(r^{2}\alpha^{(1)})$
(7)

for some known function $\alpha^{(1)}$, and similarly for ${\bf E}^{(2)}$. The
function $\chi$ is a function of $r$ only, hence $\bar{\nabla}_{i}\chi=0$, and
similarly for $\psi$. With this understanding the evolution equation (6), to
be solved for the normal part $\alpha$, is

$\displaystyle\frac{1}{r^{2}}\partial_{r}(r^{2}\alpha)=-\bar{\nabla}_{i}L^{i}=-\chi\bar{\nabla}^{i}L^{(1)}_{i}-(1-\chi)\bar{\nabla}^{i}L_{i}^{(2)}-\psi^{2}\bar{\nabla}_{i}\bar{\nabla}^{i}u=$
(8)
$\displaystyle=\frac{\chi}{r^{2}}\partial_{r}(r^{2}\alpha^{(1)})+\frac{1-\chi}{r^{2}}\partial_{r}(r^{2}\alpha^{(2)})-\frac{\psi^{2}}{r^{2}}\triangle_{S}u\
,$

where $\triangle_{S}$ is the familiar Laplacian on the unit sphere and the
function $u$ remains to be choosen. Our initial condition is that the field
shall agree with ${\bf E}^{(1)}$ when $r=R_{1}$, so we set

$\alpha_{|_{R_{1}}}=\alpha^{(1)}_{|_{R_{1}}}\ .$ (9)

The obvious drawback with the evolutionary formulation is that we have no a
priori control of the asymptotics. In the present case this translates into
the problem of how to ensure that the field agrees with ${\bf E}^{(2)}$ when
$r=R_{2}$. There is still hope, because the freely specifiable data have not
yet been fully specified. To achieve the aim we have to choose the function
$u=u(\theta,\phi;r)$ in a suitable way.

We begin by integrating our equation, which in this case can be done exactly:

$r^{2}\alpha=\chi
r^{2}\alpha^{(1)}+(1-\chi)r^{2}\alpha^{(2)}-\int_{R_{1}}^{r}\left(r^{2}(\alpha^{(1)}-\alpha^{(2)})\partial_{r}\chi+\psi^{2}\triangle_{S}u\right){\rm
d}r\ .$ (10)

With this definition of $\alpha$ the electric field ${\bf E}$ obeys Gauss’
law, and because of the way $\chi$ and $\psi$ were defined the initial
condition that ${\bf E}={\bf E}^{(1)}$ at $r=R_{1}$ holds. We also find

$R_{2}^{2}\alpha_{|_{R_{2}}}=R_{2}^{2}\alpha^{(2)}_{|_{R_{2}}}-\int_{R_{1}}^{R_{2}}\left(r^{2}(\alpha^{(1)}-\alpha^{(2)})\partial_{r}\chi+\psi^{2}\triangle_{S}u\right){\rm
d}r\ .$ (11)

Our problem will be solved if we can choose the function $u=u(\theta,\phi;r)$
so that the integral vanishes.

Let us make the problem more definite by insisting that ${\bf E}^{(2)}={\bf
0}$, and let us also choose the smoothing functions so that

$\psi^{2}(r)=-\partial_{r}\chi(r)\ .$ (12)

Then we require that

$\int_{R_{1}}^{R_{2}}\psi^{2}\left(r^{2}\alpha^{(1)}-\triangle_{S}u\right){\rm
d}r=0\ .$ (13)

If ${\bf E}^{(1)}$ is a Coulomb field then
$\alpha^{(1)}=n_{i}E_{i}^{(1)}=1/r^{2}$, and we find a contradiction when
integrating the equation over the unit sphere, because the integral of the
Laplacian acting on a regular function of the sphere vanishes for each value
of $r$. This was to be expected; by construction our vector field is
divergence free in the shell between the spheres, and hence

$\int_{r=R_{1}}{\bf E}\cdot{\bf da}=\int_{r=R_{2}}{\bf E}\cdot{\bf da}\ .$
(14)

If ${\bf E}$ agrees with a Coulomb field at $r=R_{1}$ it cannot vanish at
$r=R_{2}$. But this is the only restriction. If the integral of $\alpha^{(1)}$
over a sphere at fixed $r$ vanishes, as it must if ${\bf E}^{(2)}$ is
sourceless, then it can be expanded using spherical harmonics with $\ell\neq
0$. It is then straightforward to give a similar expansion for a function $u$
such that

$r^{2}\alpha^{(1)}-\triangle_{S}u=0\ .$ (15)

This is stronger than necessary for equation (13) to hold, but it shows
without further ado that a solution exists. Addressing equation (3) directly
is a more difficult matter [1].

3\. An example

A further advantage provided by the evolutionary formulation of the constraint
is that we can easily get explicit about the solutions. As a (very) simple
example, let us consider a ball with radius $R_{1}=1$ containing constant
crossed electric and magnetic fields, and let the fields vanish outside a ball
with radius $R_{2}=2$. Say that

$E_{x}^{(1)}=B_{y}^{(1)}=a\ ,$ (16)

all other components vanishing. With ${\bf E}^{(2)}$ vanishing we have

$E_{i}=\alpha n_{i}+L_{i}=\alpha n_{i}+\chi
L_{i}^{(1)}+\psi^{2}\bar{\nabla}_{i}u\ .$ (17)

The tangential part consists of free data, and equation (12) is in place. We
choose the function $u$ to obey

$\triangle_{S}u=r^{2}\alpha^{(1)}=axr=ar^{2}\cos{\phi}\sin{\theta}\ .$ (18)

The solution is

$u=f(r)-\frac{a}{2}r^{2}\cos{\phi}\sin{\theta}=f(r)-\frac{a}{2}xr\ .$ (19)

The function $f=f(r)$ need not concern us because only the tangential
derivative of $u$ matters. When this choice is made our solution for the
normal part of the field reads $\alpha=\chi\alpha^{(1)}$. It follows that

$E_{i}=\chi\alpha^{(1)}n_{i}+\chi L_{i}^{(1)}+\psi^{2}\bar{\nabla}_{i}u=\chi
E^{(1)}_{i}+\psi^{2}\bar{\nabla}_{i}u\ .$ (20)

Explicitly, the solution is

${\bf E}=\frac{a}{2r}\left(\begin{array}[]{c}2r\chi-(y^{2}+z^{2})\psi^{2}\\\
xy\psi^{2}\\\ xz\psi^{2}\end{array}\right)\hskip 22.76219pt{\bf
B}=\frac{a}{2r}\left(\begin{array}[]{c}xy\psi^{2}\\\
2r\chi-(x^{2}+z^{2})\psi^{2}\\\ yz\psi^{2}\end{array}\right)$ (21)

The magnetic field was similarly derived.

Figure 1: The Poynting vector field in a simple solution. The pulse is clearly
about to move rightwards (in the $z$-direction).

To get a hint about the time evolution of this blob of radiation we consider
the Poynting vector field in Figure 1. For illustrative purposes we have
chosen

$\psi=(r-1)^{2}(2-r)^{2}\ ,$ (22)

leading to finite differentiability only, but a smooth solution is easily
obtained.

More interesting solutions, where the electric and magnetic fields are non-
constant within the inner sphere, can be obtained with more work. However,
since our purpose here was to highlight the advantages of the evolutionary
formulation of the constraints, rather than to study pulses of electromagnetic
radiation as such, we stop here.

Acknowledgement: This project was supported in part by NKFIH grant K-142423.

## References

  * [1] P. T. Chruściel, Anti-gravity $\grave{a}$ la Carlotto–Schoen (d’apr$\grave{e}$s Carlotto et Schoen), Astérisque 407 (2019) 1.
  * [2] J. Corvino and D. Pollack, Scalar curvature and the Einstein constraint equations, in H. Bray and W. P. Minicozzi: Surveys in Geometric Analysis and Relativity, Advanced Lectures in Mathematics 20, Boston 2011.
  * [3] E. Delay, Smooth compactly supported solutions of some underdetermined elliptic PDE, with gluing applications, Commun. Partial Differential Equations 37 (2012) 1689.
  * [4] I. Rácz, Constraints as evolutionary systems, Class. Quant. Grav. 33 (2016) 015014.
  * [5] K. Csukas and I. Rácz, Numerical investigations of the asymptotics of solutions to the evolutionary form of the constraints, Class. Quant. Grav. 37 (2020) 155006.
  * [6] F. Beyer, J. Frauendiener, and J. Ritchie, Asymptotically flat vacuum initial data sets from a modified parabolic-hyperbolic formulation of the Einstein vacuum constraint equations, Phys. Rev. D101 (2020) 084013.
  * [7] F. Beyer and J. Ritchie, Asymptotically hyperboloidal initial data sets from a parabolic-hyperbolic formulation of the Einstein vacuum constraints, Class. Quant. Grav. 39 (2022) 145012.
  * [8] K. Csukas and I. Rácz, Is it possible to construct asymptotically flat initial data using the evolutionary forms of the constraints?, Phys. Rev. D107 (2023) 084013.
  * [9] I. Rácz, On the evolutionary form of the constraints in electrodynamics, Symmetry 11(1) (2018) 10.