Generalized spectrum of second order differential operators: 3D problems

Ivana Pultarová111Address: Faculty of Civil Engineering, Czech Technical
University in Prague, Thákurova 7, Praha 6; e-mail<EMAIL_ADDRESS>

Abstract. Generalized spectra of differential operators can be related to
spectra of preconditioned discretized operators. Obtaining (estimates of) the
eigenvalues of the preconditioned discretized operators may lead to better
estimating of the quality of preconditioners. In this short paper, we answer
the open question posted in the recent paper Generalized spectrum of second
order differential operators, authored by Gergelits, Nielsen, and Strakoš. The
proof we present allows us to fully extend characterizing the generalized
spectra of $\nabla\cdot K\nabla u=\lambda\triangle u$ to problems of dimension
three or higher.

AMS subject classification. 65F08, 65F15.

##  1 Itroduction

Efficiency of numerical solution methods for differential equations with
symmetric operators usually depends on spectral properties of the underlying
matrices. Different preconditioning techniques transform the solved systems to
systems with better spectral characteristics. Recent progress in estimating
the spectrum of preconditioned systems was achieved in [1] and then in [3, 5],
where new methods for obtaining two sided-bounds on all eigenvalues of the
preconditioned matrices were introduced. These results, however, were
motivated by infinite-dimensional problems. In [4] it was proved that
eigenvalues of preconditioned operators with scalar coefficients defined in
infinite-dimensional spaces correspond to the coefficient throughout the
domain. In [2], problems with tensorial data were considered and spectra of
preconditioned operators were fully characterized. The proof, however, was
presented only for problems of two space variables. In this paper, we prove
the main statement of [2] for three-dimensional (3D) cases, which can be
easily generalized for problems of higher dimensions.

In the subsequent section, we first recall the notation and the main result of
[2]. Then in the second subsection, we introduce the proof of characterizing
the spectra of preconditioned operators in 3D. The main result is formulated
in Theorem 2.1, the proof of which is based on Lemma 2.2, Lemma 2.3, Lemma
2.4, and Lemma 2.5. A brief discussion concludes the paper.

##  2 Characterization of the spectrum

###  2.1 Notation

We study the generalized eigenvalue problem

$\displaystyle\nabla\cdot K(\bm{x})\nabla u(\bm{x})$ $\displaystyle=$
$\displaystyle\lambda\triangle u(\bm{x})\quad\text{for}\;\bm{x}\in\Omega,$ (1)
$\displaystyle u(\bm{x})$ $\displaystyle=$ $\displaystyle
0\quad\text{for}\;\bm{x}\in\partial\Omega,$

where $\Omega\subset{\mathbb{R}}^{d}$, $d=1,2,3$, is a bounded domain with
Lipschits boundary $\partial\Omega$ and $K:\Omega\to{\mathbb{R}}^{d\times d}$
is the coefficient matrix which is assumed to be diagonal,

$K(\bm{x})=\left[\begin{array}[]{ccc}\kappa_{1}(\bm{x})&&\\\ &\ddots&\\\
&&\kappa_{d}(\bm{x})\end{array}\right].$

Let us define the operators $\mathcal{L}$, $\mathcal{A}$:
$H_{0}^{1}(\Omega)\to H^{-1}(\Omega)$,

$\displaystyle\langle\mathcal{L}u,v\rangle$ $\displaystyle=$
$\displaystyle\int_{\Omega}\nabla v\cdot\nabla u,\quad u,v\in
H_{0}^{1}(\Omega),$ $\displaystyle\langle\mathcal{A}u,v\rangle$
$\displaystyle=$ $\displaystyle\int_{\Omega}\nabla v\cdot K\nabla u,\quad
u,v\in H_{0}^{1}(\Omega),$

and the induced norm $\|u\|^{2}_{\mathcal{L}}=\langle\mathcal{L}u,u\rangle$.
The spectrum of the preconditioned operator $\mathcal{L}^{-1}\mathcal{A}$:
$H_{0}^{1}(\Omega)\to H_{0}^{1}(\Omega)$ is defined as

$\text{sp}(\mathcal{L}^{-1}\mathcal{A})=\\{\lambda\in{\mathbb{C}};\;\lambda\mathcal{I}-\mathcal{L}^{-1}\mathcal{A}\;\text{does
not have a bounded inverse}\\}.$

Thus the weak form of problem (1) can be viewed as finding the spectrum of the
preconditioned operator $\mathcal{L}^{-1}\mathcal{A}$.

We assume that $K$ is Lebesgue integrable and essentially bounded in
${\Omega}$. Instead of the diagonal shape of $K$, we can also deal with
general symmetric matrices $K$. In such cases, the main results, spectral
estimates, remain valid. To see this, an orthogonal decomposition of $K$ can
be applied and the approach of [2, Section 4] can be followed.

The main result of [2] formulated for 3D case ($d=3$) reads (see [2, Theorem
1.1])

######  Theorem 2.1.

Let $K$ be continuous in $\overline{\Omega}$. Then the spectrum of the
operator $\mathcal{L}^{-1}\mathcal{A}$ equals

${\rm sp}(\mathcal{L}^{-1}\mathcal{A})={\rm
Conv}(\cup_{i=1}^{3}\kappa_{i}(\overline{\Omega})).$

###### Proof.

The proof of the first inclusion consists of Lemmas 2.2, 2.3, and 2.4 which
are presented in the subsequent section. The inverse inclusion follows from
Lemma 2.5. ∎

###  2.2 Proof of the 3D case

To prove Theorem 2.1 we first introduce or recall four auxiliary lemmas.

######  Lemma 2.2.

Let $i\in\\{1,2,3\\}$ and let $\kappa_{i}$ be continuous in
$\bm{x}^{0}\in\Omega$. Then $\kappa_{i}(\bm{x}^{0})\in{\rm
sp}(\mathcal{L}^{-1}\mathcal{A})$.

###### Proof.

Without any loss of generality, let $i=1$ and let $\kappa_{1}$ be continuous
in $\bm{x}^{0}\in\Omega$. Denote $\kappa_{1}(\bm{x}^{0})=\lambda$. We shall
construct parametrized functions $v_{r}\in H^{1}(\Omega)$ such that

$\lim_{r\to 0}\|v_{r}\|_{\mathcal{L}}\neq 0\quad\text{and}\quad\lim_{r\to
0}\|(\lambda\mathcal{I}-\mathcal{L}^{-1}\mathcal{A})v_{r}\|_{\mathcal{L}}=0.$
(2)

For $r\in(0,1)$ define a two-dimensional disc $D_{r}$, its neighborhood
$R_{r}$, and a cylinder $C_{r}$

$\displaystyle D_{r}$ $\displaystyle=$
$\displaystyle\\{\bm{x}\in\mathbb{R}^{3};\,x_{1}=x_{1}^{0},\,d([x_{2},x_{3}],[x_{2}^{0},x_{3}^{0}])\leq
r\\}$ $\displaystyle R_{r}$ $\displaystyle=$
$\displaystyle\\{\bm{x}\in\mathbb{R}^{3};\,d(\bm{x},D_{r})\leq r^{2}\\}$
$\displaystyle C_{r}$ $\displaystyle=$
$\displaystyle\\{\bm{x}\in\mathbb{R}^{3};\,x_{1}\in(x_{1}^{0}-r^{2},x_{1}^{0}+r^{2}),\,d([x_{2},x_{3}],[x_{2}^{0},x_{3}^{0}])\leq
r\\},$

where $d(A,B)$ is Euclidean distance between objects $A$ and $B$. Note that
$D_{r}\subset C_{r}\subset R_{r}$. Let us choose some $r^{0}\in(0,1)$ such
that $R_{r^{0}}\subset\Omega$. For every $r\in(0,r^{0})$ define a function
$v_{r}$ by

$\displaystyle v_{r}(\bm{x})$ $\displaystyle=$ $\displaystyle
0,\qquad\bm{x}\in\Omega\setminus R_{r}$ $\displaystyle=$ $\displaystyle
1,\qquad\bm{x}\in D_{r}$ $\displaystyle=$ $\displaystyle
1-\frac{d(\bm{x},D_{r})}{r^{2}}\in\langle 0,1),\quad\bm{x}\in R_{r}\setminus
D_{r}.$

Note that $v_{r}$ is continuous in $\Omega$ and

$\nabla v_{r}(\bm{x})=\left(\frac{\pm 1}{r^{2}},0,0\right),\quad\bm{x}\in
C_{r}\setminus D_{r}.$

The part $R_{r}\setminus C_{r}$ is a part of a torus. Equipotential surfaces
of $v_{r}$ defined in $R_{r}\setminus C_{r}$ are of the shape of toroidal
surfaces again. Thus the gradient of $v_{r}$ in $\bm{x}\in R_{r}\setminus
C_{r}$ directs to the nearest point $\bm{x}_{D}$ of $D_{r}$ to $\bm{x}$. Due
to the equidistant distribution of the equipotential surfaces the gradients
have the same norms for any $\bm{x}$ of the same equipotential surface. Then

$\left|\frac{\partial v_{r}}{\partial x_{i}}\right|\leq\frac{1}{r^{2}},\quad
i=1,2,3,\quad\bm{x}\in R_{r}\setminus C_{r}.$

Volume of $R_{r}$ is less than $2\pi r^{2}(r+r^{2})^{2}$, volume of $C_{r}$ is
$2\pi r^{4}$, thus volume of $R_{r}\setminus C_{r}$ is less than $2\pi
r^{5}(2+r)$. Then

$\displaystyle\int_{C_{r}}\left(\frac{\partial v_{r}}{\partial
x_{1}}\right)^{2}$ $\displaystyle=$ $\displaystyle 2\pi$
$\displaystyle\int_{C_{r}}\left(\frac{\partial v_{r}}{\partial
x_{i}}\right)^{2}$ $\displaystyle=$ $\displaystyle 0,\quad i=2,3$
$\displaystyle\int_{R_{r}\setminus C_{r}}\left(\frac{\partial v_{r}}{\partial
x_{i}}\right)^{2}$ $\displaystyle\leq$ $\displaystyle 2\pi r(2+r),\quad
i=1,2,3,$

which yields

$\lim_{r\to 0}\|v_{r}\|_{\mathcal{L}}=\sqrt{2\pi}.$

The rest of the proof is analogous to the last part of the proof of [2, Lemma
2.1]. Denoting $u_{r}=(\lambda\mathcal{I}-\mathcal{L}^{-1}\mathcal{A})v_{r}$,
we get

$\|u_{r}\|_{\mathcal{L}}^{2}=\langle{\mathcal{L}}u_{r},u_{r}\rangle=\langle(\lambda{\mathcal{L}}-{\mathcal{A}})v_{r},u_{r}\rangle=\int_{\Omega}\nabla
u_{r}\cdot(\lambda I-K)\nabla v_{r}\leq\left(\int_{\Omega}\|(\lambda
I-K)\nabla v_{r}\|^{2}\right)^{\frac{1}{2}}\|u_{r}\|_{\mathcal{L}}.$

Then

$\displaystyle\|u_{r}\|_{\mathcal{L}}^{2}$ $\displaystyle\leq$
$\displaystyle\int_{\Omega}(\lambda-\kappa_{1})^{2}\left(\frac{\partial
v_{r}}{\partial
x_{1}}\right)^{2}+\sum_{i=2}^{3}\int_{\Omega}(\lambda-\kappa_{i})^{2}\left(\frac{\partial
v_{r}}{\partial x_{i}}\right)^{2}$ $\displaystyle\leq$
$\displaystyle(2\pi+2\pi r(2+r))\,\sup_{\bm{x}\in
R_{r}}|\kappa_{1}(\bm{x}^{0})-\kappa_{1}(\bm{x})|^{2}+2\pi
r(2+r)\sum_{i=2}^{3}\sup_{\bm{x}\in
R_{r}}|\kappa_{1}(\bm{x}^{0})-\kappa_{i}(\bm{x})|^{2}.$

From continuity of $K$ in $\bm{x}^{0}$ we get

$\lim_{r\to 0}\|u_{r}\|_{\mathcal{L}}=0,$

which concludes the proof of (2). ∎

######  Lemma 2.3.

Let $\kappa_{i}$, $i=1,2,3$, be constant on an open subdomain
$S\subset\Omega$. Assume

$\sup_{\bm{x}\in\Omega}\kappa_{j}(\bm{x})<\inf_{\bm{x}\in\Omega}\kappa_{m}(\bm{x}),$

for some pair $j,m\in\\{1,2,3\\}$, $j\neq m$. Then

$[\sup_{\bm{x}\in\Omega}\kappa_{j}(\bm{x}),\inf_{\bm{x}\in\Omega}\kappa_{m}(\bm{x})]\subset{\rm
sp}(\mathcal{L}^{-1}\mathcal{A}).$

###### Proof.

Let $j=1$ and $m=2$ without any loss of generality. Denote
$k_{1}=\kappa_{1}(\bm{x})$ and $k_{2}=\kappa_{2}(\bm{x})$, $\bm{x}\in S$, the
values of $\kappa_{1}$ and $\kappa_{2}$ in $S$. Note that

$[\sup_{\bm{x}\in\Omega}\kappa_{1}(\bm{x}),\inf_{\bm{x}\in\Omega}\kappa_{2}(\bm{x})]\subset[k_{1},k_{2}].$

Let $\lambda\in(k_{1},k_{2})$ and choose some point $\bm{x}^{0}\in S$. Denote

$S_{h}=(x_{1}^{0},x_{1}^{0}+h\sqrt{\lambda-
k_{1}})\times(x_{2}^{0},x_{2}^{0}+h\sqrt{k_{2}-\lambda})\times(x_{3}^{0},x_{3}^{0}+h)$

where $h\in(0,1)$ is sufficiently small that $S_{h}\subset S$. Then for any
$n\in{\mathbb{Z}}$ the tensor product function

$\phi(\bm{x})=\sin\left(\frac{n\pi(x_{1}-x_{1}^{0})}{h\sqrt{\lambda-
k_{1}}}\right)\sin\left(\frac{n\pi(x_{2}-x_{2}^{0})}{h\sqrt{k_{2}-\lambda}}\right)$

fulfills

$(\lambda-k_{1})\frac{\partial^{2}\phi(\bm{x})}{\partial
x_{1}^{2}}=-\frac{n^{2}\pi^{2}}{h^{2}}\phi(\bm{x}),\quad(k_{2}-\lambda)\frac{\partial^{2}\phi(\bm{x})}{\partial
x_{2}^{2}}=-\frac{n^{2}\pi^{2}}{h^{2}}\phi(\bm{x}),\quad(\lambda-
k_{3})\frac{\partial^{2}\phi(\bm{x})}{\partial x_{3}^{2}}=0$

in $S_{h}$ and the boundary condition $\phi(\bm{x})=0$ on $\partial S_{h}$.
Thus the function $v\in H_{0}^{1}(\Omega)$ defined as

$\displaystyle v(\bm{x})$ $\displaystyle=$
$\displaystyle\phi(\bm{x}),\quad\bm{x}\in S_{h}$ $\displaystyle=$
$\displaystyle 0,\quad\bm{x}\notin S_{h}$

solves the weak form of the generalized eigenvalue problem (1). ∎

######  Lemma 2.4.

Assume that $K$ is continuous at least at a single point in $\Omega$. Let

$\sup_{\bm{x}\in\Omega}\kappa_{j}(\bm{x})<\inf_{\bm{x}\in\Omega}\kappa_{m}(\bm{x}),$

for some pair $j,m\in\\{1,2,3\\}$, $j\neq m$. Then

$[\sup_{\bm{x}\in\Omega}\kappa_{j}(\bm{x}),\inf_{\bm{x}\in\Omega}\kappa_{m}(\bm{x})]\subset{\rm
sp}(\mathcal{L}^{-1}\mathcal{A}).$

###### Proof.

The proof is analogous to the proof of [2, Lemm 2.3]. ∎

######  Lemma 2.5.

Assume that $K$ is continuous in the closure $\overline{\Omega}$. Then

${\rm sp}(\mathcal{L}^{-1}\mathcal{A})\subset{\rm
Conv}(\cup_{i=1}^{3}\kappa_{i}(\overline{\Omega})).$

###### Proof.

The statement trivially follows from

$\min_{i=1,2,3}\inf_{\bm{x}\in\overline{\Omega}}\kappa_{i}(\bm{x})\int_{\Omega}\nabla\phi\cdot\nabla\phi\;\leq\;\int_{\Omega}\nabla\phi\cdot
K\nabla\phi\;\leq\;\max_{i=1,2,3}\sup_{\bm{x}\in\overline{\Omega}}\kappa_{i}(\bm{x})\int_{\Omega}\nabla\phi\cdot\nabla\phi$

for $\phi\in H_{0}^{1}(\Omega)$. Full details can be found in the proof of [2,
Lemm 2.4]. ∎

##  3 Discussion

We introduce the proof of the main results of [2] for 3D problems. The main
contribution is Lemma 2.2, where a certain construction of a set of functions
$v_{r}$ is presented. The construction can be naturally generalized to higher
dimensions. The methodology can also help to derive estimates of eigenvalues
of discretized operators, and thus provide a link between preconditioned
differential operators and associated numerical linear algebra problems.
Especially, the functions $v_{r}$ can serve as approximations of
eigenfunctions of the discretized preconditioned operators.

Some questions remain open: What is the distribution of eigenvaues of the
associated preconditioned discretized operator? or What can we say about the
spectral estimates of preconditioned operators if $K$ is piecewise constant?
and How these spectra depend on a discretization basis?

Acknowledgment. The author thanks Tomáš Gergelits and Zdeněk Strakoš for
fruitful discussions and the Center of Advanced Applied Sciences, the European
Regional Development Fund (project No. CZ.02.1.01/0.0/0.0/16_019/0000778) for
a partial support.

## References

  * [1] T. Gergelits, K.-A. Mardal, B. F. Nielsen, Z. Strakoš, Laplacian preconditioning of elliptic PDEs: Localization of the eigenvalues of the discretized operator. SIAM Journal on Numerical Analysis, 57, 2019, pp. 1369–1394.
  * [2] T. Gergelits, B. F. Nielsen, Z. Strakoš, Generalized spectrum of second order differential operators. SIAM Journal on Numerical Analysis, 58, 2020, pp. 2193–2211.
  * [3] M. Ladecký, I. Pultarová, J. Zeman, Guaranteed two-sided bounds on all eigenvalues of preconditioned diffusion and elasticity problems solved by the finite element method. Applications of Mathematics, 66(1), 2020, pp. 21–42.
  * [4] B. F. Nielsen, A. Tveito, W. Hackbusch, Preconditioning by inverting the Laplacian. IMA Journal of Numerical Analysis, 29, 2009, pp. 24–42.
  * [5] I. Pultarová, M. Ladecký, Two-sided guaranteed bounds to individual eigenvalues of preconditioned finite element and finite difference problems. Numerical Linear Algebra with Applications, 28(5), 2021, e2382.


11institutetext:  GESIS – Leibniz Institute for the Social Sciences, Cologne,
Germany 11email<EMAIL_ADDRESS>22institutetext: Heinrich-
Heine-University Düsseldorf, Germany
22email<EMAIL_ADDRESS>
# Enhancing Software-Related Information Extraction via Single-Choice Question
Answering with Large Language Models

Wolfgang Otto 11 0000-0002-9530-3631 Sharmila Upadhyaya 11 0009-0003-7142-3887
Stefan Dietze 1122 0009-0001-4364-9243

###### Abstract

This paper describes our participation in the Shared Task on Software Mentions
Disambiguation (SOMD), with a focus on improving relation extraction in
scholarly texts through generative Large Language Models (LLMs) using single-
choice question-answering. The methodology prioritises the use of in-context
learning capabilities of LLMs to extract software-related entities and their
descriptive attributes, such as distributive information. Our approach uses
Retrieval-Augmented Generation (RAG) techniques and LLMs for Named Entity
Recognition (NER) and Attributive NER to identify relationships between
extracted software entities, providing a structured solution for analysing
software citations in academic literature. The paper provides a detailed
description of our approach, demonstrating how using LLMs in a single-choice
QA paradigm can greatly enhance IE methodologies. Our participation in the
SOMD shared task highlights the importance of precise software citation
practices and showcases our system’s ability to overcome the challenges of
disambiguating and extracting relationships between software mentions. This
sets the groundwork for future research and development in this field.

###### Keywords:

Generative Large Language Models Information Extraction Named Entity
Recognition Relation Extraction Software Citation Retrieval-Augmented
Generation Single-Choice Question Answering Software Mentions Disambiguation
Task

##  1 Introduction

The evolution of information extraction (IE) in scholarly communications
necessitates the development of innovative methodologies to accurately
identify and categorise software mentions. This is a critical component for
ensuring research transparency and reproducibility. The Shared Task on
Software Mentions Disambiguation (SOMD) highlights the importance of refined
citation practices amidst inconsistent referencing of software artifacts. This
research employs generative Large Language Models (LLMs) to address the
challenges in software mention extraction and relation identification, marking
a significant step towards the sophisticated extraction of software-related
information. Our approach integrates Retrieval-Augmented Generation (RAG)
techniques with LLMs to dissect and comprehend the intricate web of software
citations and their attributive details (e.g. the developer or the version)
within scientific texts. By examining the potential of LLMs in performing IE
tasks by transforming them into single-choice question-answering, we present a
comprehensive analysis that addresses the nuances of software mention
extraction and could be applied on a broader scope of scholarly artifacts NER
and relation extraction. This paper explores the complexities of applying LLMs
to NER tasks, providing insights into the challenges and proposing a new
methodology for relation extraction that could pave the way for future
innovations in the field.

##  2 Related Work

This section explores works in Named Entity Recognition (NER) and Information
Extraction (IE), focusing on software mention extraction, scholarly artifact
NER, and the use of generative LLMs in these areas.

In software mention extraction, the Softcite dataset [2] and the SoMeSci
knowledge graph [7] are significant. Softcite offers a gold-standard dataset
for extracting software mentions from biomedical and economic research, while
SoMeSci Knowledge Graph includes software mentions in scientific articles with
relation labels such as version, developer and citations, highlighting the
need for accurate software mention extraction.

In scholarly artifact NER, Saji and Matsubara [6] introduced a method using
academic knowledge graphs to extract research resource metadata from scholarly
papers, enhancing metadata quality and repository size. Otto et al. [5]
developed the GSAP-NER corpus for extracting machine learning-related entities
from scientific publications, filling the gap in general-purpose NER models.
These words demostrate the importance of domain-specific NER tasks and
leverageing knowledge graphs for enhancing research resource repositories.

Regarding LLMs for NER, Wang [10] explores the use of text-generation models
for sequence labeling and underlying challenges, particularly in low-resource
and few-shot setups. Furthermore, Xie [11] investigates LLMs’ self-improving
capabilities for zero-shot NER.

For LLMs in relation extraction, Wadhwa [8] uses few-shot prompting and fine-
tuning with large language models achieving state-of-the-art performances in
relation extraction using LLMs, while Wan [9] introduce GPT-RE to enhancing
relation extraction accuracy through task-specific entity representations.

Lastly, Xu [12] conducts a comprehensive survey providing an overview of
generative information extraction using LLMs, categorizing works by IE
subtasks and learning paradigms, and highlighting the transformative potential
of LLMs in IE.

Overall, these studies underscore the evolving methods and significant
contributions in extracting information from scholarly texts using specialized
datasets, domain-specific approaches, and advanced generative models.

##  3 SOMD Shared Task

The Shared Task on Software Mentions Disambiguation (SOMD) aims to enhance
transparency and reproducibility in scientific research by improving software
citation practices. Participants are required to develop models that can
identify and disambiguate software mentions in scholarly texts using the
expanded Software Mentions in Science (SoMeSci) knowledge graph, with a focus
on AI and Computer Science.

Subtask 1, Software NER, involves identifying four types of software-related
entities: Application, Plugin, Operating System, and Programming Environment.
Accurately classifying software mentions is necessary to understand their role
in academic discourse.

Subtask 2, Attribute NER, aims to extract ten different attributive
information associated with software entities, such as alternative names,
abbreviations, authorship (developers), software release maintenance
(versions, extensions, release dates, licenses), and elements of the
referencing system (in-text citations, URLs, and software coreferences for
cross-sentence linkage), to a specific software mention.

Subtask 3, Relation Extraction, involves establishing relationships between
software entities and their attributes using specific relation types and
mapping each attribute to these relations. This task aids in comprehending the
interrelations of software entities within scholarly texts.

The tasks evaluation metric, is the Weighted Average Macro F1 score. This
metric adjusts the influence of each label on the final result based on the
number of test instances assigned to it, ensuring that labels with fewer
instances have a proportionally smaller impact on the overall performance
evaluation.

##  4 Using LLMs for Software Related IE-Tasks

The exploration of generative LLMs such as GPT-4 for Information Extraction
(IE) tasks related to software underscores the transformative potential these
models hold as general-purpose task solvers. The allure of leveraging LLMs in
this capacity is significant, given their ability to process and generate
human-like text across a wide range of topics and formats. However, the
challenge arises in the specificity and nuanced requirements of domain-
specific tasks, such as NER within specialized fields. It has been observed
that, despite their vast knowledge and versatility, current models like GPT-4
often fall short when tasked with domain-specific NER, primarily due to their
generalized training and lack of domain-specific tuning.

To mitigate these shortcomings and enhance the performance of LLMs in
specialized IE tasks, various in-domain learning strategies are employed.
These strategies are designed to equip the LLM with a deeper understanding of
the task at hand, essentially guiding the model towards more accurate
identification and classification of relevant text spans. Among these
strategies, optimizing the task description plays a crucial role. A well-
crafted, precise task description can significantly improve the model’s focus
and comprehension of the task’s objectives, leading to more relevant and
accurate outcomes.

Furthermore, the provision of speaking, prototypical examples serves as
another effective strategy. By presenting the model with clear, illustrative
examples that encapsulate the essence of the task, we can anchor its
understanding and enhance its ability to generalize from these examples to
new, unseen instances. This approach leverages the model’s inherent learning
capabilities, allowing it to draw parallels and apply learned concepts to the
task at hand.

Additionally, augmenting the model’s capabilities with Retrieval Augmented
Generation (RAG) introduces a powerful dimension to the information extraction
process. RAG combines the generative prowess of LLMs with the specificity and
relevance of retrieved documents, enabling the model to access a broader
context and detailed examples that are pertinent to the task. This strategy is
particularly advantageous in domain-specific applications, where the relevance
and accuracy of the information extracted are paramount.

In our approach, we capitalize on the last strategy, RAG, to maximize the
utility of training sets for each specific task. The retrieval component of
this strategy entails identifying instances within the training set that are
similar to the test instance and can provide valuable insights for the
identification and classification of relevant information. This method not
only enhances the model’s performance by providing it with task-relevant data
but also ensures that the information extracted is of high relevance and
quality, tailored to the specific demands of the domain-specific NER task.
Through these tailored strategies, we aim to bridge the gap between the broad
capabilities of LLMs and the precise requirements of domain-specific
information extraction tasks, paving the way for more effective and efficient
utilization of generative language models in specialized domains.

###  4.1 Challenges in Applying LLMs to NER Tasks

The integration of generative LLMs into NER tasks introduces a set of unique
challenges that can significantly impact the performance and reliability of
extraction outcomes. One of the most prevalent issues in generative NER
approaches is the phenomenon of hallucination[10], where the model generates
entities not present in the test instance. This can result from the model
misinterpreting the provided examples as part of the text from which entities
should be extracted, leading to inaccuracies and inconsistencies in the
results. n Furthermore, matching problems during the location of mention
positions present considerable challenges, particularly in the context of
span-based evaluation systems. These systems evaluate the accuracy of entity
extraction based on the precise span of text identified as an entity.
Discrepancies in the extracted span—whether through corrected spellings or
variations in representation—can complicate the matching process. For example,
the extraction of “jquery” as “jQuery” illustrates a common scenario where the
prevalent spelling of a software library may differ from its mention in the
text, yet both are correct. This variability necessitates sophisticated
matching strategies to ensure accuracy in evaluation.

The situation is further complicated by texts containing multiple mentions of
a single entity. The challenge lies in determining whether each mention can be
accurately classified as an entity, alongside dealing with the ambiguity of
different entities that are written in the same way. Overlapping or nested
entity mentions that do not align with the ground truth data introduce
additional layers of complexity, requiring nuanced approaches to entity
recognition and classification. Our proposed baseline solution, filtering out
non-matching entities and employing rule-based decisions for handling multiple
matches have been adopted. However, the disadvantage of this method is that it
is based on simplistic heuristics.

Future work may explore advanced solutions like using LLMs for precise entity
matching, offering potential improvements for NER challenges. While this paper
does not delve into these complex methods, it highlights the importance of
ongoing research to further refine and improve LLMs for more accurate and
efficient entity extraction.

###  4.2 Sample Retrieval for RAG on various IE-Tasks

Retrieval-Augmented Generation (RAG) significantly bolsters the capabilities
of LLMs in IE tasks by effectively utilizing both unstructured and structured
contexts [3, 4]. This dual approach is essential in IE for achieving precise
extractions, yet the selection of optimal samples for the generative process
poses a substantial challenge. Addressing this involves two primary
strategies: utilizing sentence embeddings to find contextually similar textual
content for the LLM, and identifying analogous entities to uncover beneficial
training sentences. These methods enable the LLM to discern structural and
semantic patterns for more accurate text extractions. A crucial obstacle is
accurately identifying target entities within test instances, for which we
employ a pre-trained Language Model (PLM) tailored to our NER task. This PLM
is instrumental in both spotting potential entity candidates and facilitating
entity similarity searches, leveraging last hidden state embeddings from
training examples to locate matching entities within the dataset.

Our methodology extends to evaluating various retrieval techniques and their
impact on the LLM’s efficiency, particularly within a Few-Shot learning
framework. We explore different methods, including random illustrative
samples, text similarity-based RAG, and entity-based sentence retrieval, to
provide the LLM with contextually relevant examples, thereby optimizing the
software entity extraction process from scientific texts. This exploration
aims to identify the most effective strategies for utilizing LLMs in domain-
specific software entity extraction tasks. By overcoming challenges related to
resource demand, execution time, and precision in entity identification, our
approach aims to enhance the accuracy and efficiency of NER processes,
contributing valuable insights and methodologies to the field of computational
linguistics and information extraction.

A special retrieval method is used for the RE Task. Because for this task, the
entities for each test instance are given, we easily could list all possible
relations (compare Table 3). We could use them to find similar relations in
the train set. If we find more then one, we decide for the one, with highest
sentence similarity.

For the RE Task, a special retrieval method is employed. As entities for each
test instance are provided, all possible relations can be listed based on
Table 3. These relations can then be used to find relations of the same type,
and with the same domain and range in the train set. If multiple similar
relations are found, the one with the highest sentence similarity is selected.

###  4.3 Extraction of Software Entities

The extraction of software entities from scientific texts represents a
specialized challenge within the realm of NER, targeting the identification of
software entities across four distinct categories ranging from applications to
operating systems. Furthermore, this task extends beyond mere identification,
seeking to understand the intent behind each mention of software
entities—whether it pertains to creation, usage, deposition, or mere citation.

We address this challenging task using LLMs, despite their high demands on
resources and time, especially when processing extensive publications with
sparse relevant text. Our approach includes a pipeline strategy that
prioritizes selecting relevant text passages for LLM analysis, improving
efficiency by filtering out unrelated content. This method’s success depends
on the selection accuracy, directly impacting recall. However, the trade-off
for reduced computational costs justifies the potential minor decrease in
recall. Our performance optimization employs a hybrid method, combining a
fine-tuned NER model for sentence selection with LLMs for information
extraction. This approach faces limits, notably when sentences crucial for
analysis are missed in the selection phase, capping the LLM extraction phase’s
accuracy as indicated by an initial sentence classification task recall of
0.882 (0.884 F1). This establishes a theoretical limit on extraction accuracy
due to potential false negatives, illustrating a balance between efficiency
and the precision constraints of LLMs in detailed text analysis.

###  4.4 Extraction of Software Attributes

Following the identification of software entities within scientific texts, a
further nuanced aspect of NER and IE tasks emerges in the extraction of
associated software attributes. These attributes encompass a wide array of
specific details, the version, developer, citations, URLs, release dates,
abbreviations, licenses, extensions, software co-references, and alternative
names. We used a similar approach as for subtask 1, and utilised train sample
retrieval to augment the task description in a few shot setup. For each
sample, including those derived from few-shot learning, the process entails
presenting the sentence containing the software entity(ies) and then
predicting a JSON list of identified entities along with their respective
attribute types.

###  4.5 Relation Extraction as Single-Choice Question Answering Task

In the field of Natural Language Processing (NLP), the extraction of relations
between entities within a text corpus poses significant challenges. This study
proposes a novel approach by conceptualizing the task of relations extraction
as a single-choice question-answering (QA) activity. This method entails
generating a comprehensive list of all possible entities within a sentence,
drawing from the existing entities and their relationships as delineated in
the training dataset. Each potential pair of entities is then evaluated to
ascertain if a specific relation, such as “version_of”, appropriately links
them. For instance, considering the relationship “version_of”, a sentence may
be formulated as “8 is the version of SPSS”, representing a possible relation
between the version number and the software entity.

For every sentence in the dataset, this process yields a set of single-choice
questions, each positing a potential relationship between entities. These
questions are then prompted to a LLM for answering. The LLM’s task is to
select the most plausible relation from among the given options, thereby
facilitating the extraction of accurate entity relations from the text.
However, this approach is not without its challenges. A primary source of
error stems from instances where multiple relations could plausibly link a
pair of entities, leading to ambiguity and complicating the single-choice
question-answering framework. Despite the challenges, we demonstrate that
treating relation extraction as a single-choice QA task provides a structured
and innovative approach to extracting valuable insights from complex textual
data.

##  5 Experiments

###  5.1 Models

Fine-tuned Model In our experimental setup, the initial phase focused on fine-
tuning a language model specifically for Subtask 1, which involved NER.
Following the methodology similar to Schindler [7], we employed the SciBERT
model [1], given its pre-training on scientific corpus, making it apt for NER
fine-tuning within scholarly texts. The fine-tuning process entailed rigorous
parameter optimization, including adjustments to the batch size, learning
rate, and the relative share of negative samples—sentences that do not contain
any annotations. This optimization utilized a 90/10 train/evaluation data
split from the available training dataset. Subsequent to parameter tuning, we
conducted a final training run with a modified data split of 95/5
train/evaluation to maximize the training data’s utility.
Generative Large Language Models For a comparative analysis, our study
integrated LLMs, specifically examining the performance differences between
GPT-3.5-fast and GPT-4-fast models accessed via the OpenAI API. To ensure
deterministic outputs for comparison, we set the temperature parameter to
zero, eliminating randomness in the model’s response generation.

###  5.2 Prompting

Software NER The prompting strategy for Software NER involved providing a
concise task description, which included specifying the task as NER and
intention classification, alongside an enumeration of target labels. We also
highlighted the domain specificity of the texts (i.e., scientific
publications) and requested the output in JSON format, delineating separate
labels for entity type and intention. Sample sentences from the training set,
along with their corresponding JSON output, were included as illustrative
examples. This setup varied in the number and order of displayed examples
based on the retrieval method, exploring the impact of these factors on model
performance through separate experiments.

Attributive NER For Attributive NER, the prompt construction mirrored the
approach taken in Subtask 1 but incorporated more detailed rules, emphasizing
the necessity for attributes to relate directly to software entities. Known
software entities were additionally provided as input, even for test
instances, adhering to the oracle setup defined in the shared task guidelines.
This method aimed to refine the model’s ability to extract and classify
attributive information accurately.

Relation Extraction The approach to Relation Extraction reimagined the task as
a series of single-choice question-answering challenges. Each potential
relation, given the software and attributive entities within a sentence, was
listed, with claims formulated for each possible relationship (e.g., "IBM is
the developer of Windows"). The model was tasked with identifying the veracity
of each claim through a single-choice question format, where all claims were
enumerated and solutions provided in a batch format for each example sentence.
This setup culminated in presenting the test instance alongside its single-
choice questions, expecting the model to deliver decisions on the relational
claims.

###  5.3 Train Sample Retrieval for Few-Shot Generation

Our experimental framework explored various methods for test sample retrieval
to ascertain the most effective approach in enhancing model performance. These
methods included the use of random illustrative samples to represent every
possible signature, retrieval based on entity similarity, and retrieval based
on sentence similarity. Each method was evaluated against a baseline to
determine its impact on the accuracy and efficiency of the information
extraction tasks at hand.

###  5.4 Relation Extraction Baseline

In the development of our baseline for relation extraction, we established a
robust heuristic framework derived from an analysis of potential relations
indicated within the text. Our strategy was guided by two principal rules
aimed at simplifying the decision-making process for identifying accurate
relationships between entities. Firstly, we limited our consideration to
relations that necessitate the presence of at least one related entity,
deliberately excluding optional inter-software entity relations such as
"specification_of" and "PlugIn_of." Given the infrequent occurrence of these
cases within the training dataset, we anticipated only a minimal impact on the
overall performance metric, specifically the weighted mean macro F1 score for
the subtask. Secondly, for all remaining relation types, our approach favored
selecting the closest possible entity positioned to the left of the focal
software entity as the most likely relation partner. This heuristic was not
only straightforward but proved to be highly effective, aligning our baseline
performance with that of the top contenders in the shared task. This
methodology highlights the potential of leveraging simple, rule-based
strategies to achieve competitive results in complex relation extraction
challenges.

##  6 Results

Table 1: Results of Different Models and Retrieval Methods on Subtask 1 Paradigm | Retrieval | F1 | Model | parameter
---|---|---|---|---
Finetuned | - | 0.599 | SciBERT | -
Prompt | Random | 0.483 | GPT 3-5 | Random k=7
Prompt | Random | 0.525 | GPT 3-5 | Random all entity types shown
Prompt | Sim. sentences | 0.647 | GPT 3.5 | topn=10
Prompt | Sim. entities | 0.624 | GPT 3.5 | topn=7 x n entities
Prompt | Random | 0.574 | GPT 4 | Random all entity types shown
Prompt | Sim. sentences | 0.677 | GPT 4 | topn=10
Prompt | Sim. entities | 0.679 | GPT 4 | topn=7 x n entities

Our analysis in Subtask 1 (Software NER) shows a varied performance landscape
across different models and retrieval methods (Table 1). The finetuned
baseline, which uses SciBERT, achieved a solid foundation with a 59.9% F1
score. However, LLMs that use random samples without fine-tuning showed a
decrease in performance, with the highest F1 score reaching only 57.4%.
A closer examination of retrieval-based models indicates that LLMs perform
better. The highest F1 score of 67.9% was achieved by sentence similarity
retrieval models, while entity retrieval showed the best performance at 67.7%
F1. The transition from GPT 3.5 to GPT 4 models resulted in a significant
improvement of approximately 3–5%, although it required around three times
more computation time. Notably, our best models were able to perform within a
mere 3% below the theoretical maximum by utilizing SciBERT for sentence
selection, leveraging oracle positive sentences.
In Subtask 2 (Attributive NER), our methodology showed a significant
improvement of +10% in F1 performance compared to our competitors,
demonstrating the effectiveness of our approach in a low data regime (Table
2). For Subtask 3 (Relation Extraction), our LLM Single-Question Answering
model further improved the F1 score by 5.1%, building on the already competent
performance of the heuristic baseline and highlighting the advantage of our
method. Furthermore, it has been demonstrated that using 7-10 samples is the
most effective strategy, as it optimises the balance between input complexity
and model performance.
This analysis highlights the potential of utilising advanced LLM techniques
and carefully selected retrieval methods to significantly improve the accuracy
and efficiency of NER tasks in specialised domains.

Table 2: SOMD Performance Rankings (Weighted Average Macro) Task | # | User | F1 | Precision | Recall | Task | # | User | F1 | Precision | Recall
---|---|---|---|---|---|---|---|---|---|---|---
1 | 1 | phinx | .740 | .761 | .750 | 2 | 1 | _ours_ | _.838_ | _.835_ | _.847_
1 | 2 | david-s477 | .692 | .739 | .711 | 2 | 2 | phinx | .743 | .745 | .748
1 | 3 | ThuyNT03 | .678 | .729 | .649 | 3 | 1 | _ours_ | _.916_ | _.911_ | _.924_
1 | 4 | _ours_ | _.652_ | _.679_ | _.664_ | 3 | 2 | phinx | .897 | .900 | .897
1 | 5 | vampire | .648 | .682 | .637 | 3 | - | _baseline_ | _.864_ | _.857_ | _.875_
|  |  |  |  |  | 3 | - | _necessary_ | _.562_ | _.933_ | _.415_

##  7 Conclusion

Our research on enhancing Relation Extraction (RE) with LLMs through Single-
Choice QA has introduced a novel intersection of methodologies aimed at
improving the precision of information extraction in the context of scientific
texts. By integrating Retrieval-Augmented Generation (RAG) with LLMs and
adopting a methodical approach to fine-tuning and leveraging large language
models such as SciBERT and use the model to support GPT variants, we have
demonstrated the capability of LLMs to navigate the complexities inherent in
the extraction of software entities and their attributes.

The exploration of different retrieval strategies—ranging using entity and
sentence similarity underscores our commitment to refining the inputs for
generative models, ensuring that they are fed the most relevant and
contextually appropriate data. This meticulous preparation has allowed us to
significantly boost the performance of LLMs in recognising nuanced
distinctions among software-related entities and accurately extracting
relation types within scholarly articles. Our experiments have not only
highlighted the efficacy of LLMs in addressing domain-specific tasks with a
limited set of examples but also revealed the inherent challenges, such as the
difficulties in matching entity mentions accurately. Despite these hurdles,
our single-choice QA approach for RE lead to a strong heuristic baseline for
relation extraction and show how altering the task simplifies the problem.

The outcomes of our research indicate a promising direction for future work in
leveraging LLMs for NER and RE tasks. The development of our system for
participation in the SOMD shared task has illustrated the potential of a
single-choice QA approach to relation extraction, offering a structured and
scalable method for extracting meaningful insights from textual data. Our
findings contribute to the growing body of knowledge on the application of
generative models in the field of computational linguistics, paving the way
for more sophisticated and efficient methodologies in information extraction
from scientific literature.

## Acknowledgements

We thank the anonymous reviewers for their constructive feedback. This work
has been partially funded by the Deutsche Forschungsgemeinschaft (DFG, German
Research Foundation) as part of the Projects BERD@NFDI (grant number
460037581) as well as NFDI4DS (grant number 460234259).

##  Appendix 0.A Dataset Overview

Each subtask in the SOMD Shared Task is evaluated independently. Subtask 2’s
attributive NER uses the identified software entities from Subtask 1 as
inputs, and the annotations from the first two subtasks are combined for
Subtask 3’s Relation Extraction. The dataset is hierarchically structured,
with each subtask’s test set forming a subset of the previous one. This lead
to data leakage in Subtask 1 and to a lesser extent in Subtask 2. This design
ensures that tasks are interconnected, but it requires careful interpretation
of results, particularly with respect to error propagation. Although this
design creates data leakage, it ensures the interconnectedness and cumulative
nature of the challenges. The initial presentation of the Corpus is as the
SoMeSci knowledge graph [7]. It is a comprehensive dataset consisting of 1,367
documents and containing 399,942 triples that represent 47,524 sentences. The
dataset includes 2,728 software entities, totaling 7,237 labeled entities. The
dataset includes positive and negative examples, i.e. sentences without
software mentions, which improves the accuracy of the model. The presence of
duplicate sentences, headings, and varying sentence lengths increases the
complexity of the text and makes extraction tasks from scientific texts more
challenging. For possible domains and ranges of the relations int the train
set compare Table 3

Table 3: Object Properties, Domain, and Range Classes with Inner Software Relations. Range Class | Application | Operating | PlugIn | Programming
---|---|---|---|---
$\Rightarrow$ Object Property |  | System |  | Environment
Abbreviation $\Rightarrow$ Abbreviation_of |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$
Developer $\Rightarrow$ Developer_of |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$
Release $\Rightarrow$ Release_of |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$
Version $\Rightarrow$ Version_of |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$
Citation $\Rightarrow$ Citation_of |  $\checkmark$ |  $\times$ |  $\checkmark$ |  $\checkmark$
PlugIn $\Rightarrow$ PlugIn_of |  $\checkmark$ |  $\times$ |  $\checkmark$ |  $\checkmark$
URL $\Rightarrow$ URL_of |  $\checkmark$ |  $\times$ |  $\checkmark$ |  $\checkmark$
Extension $\Rightarrow$ Extension_of |  $\checkmark$ |  $\checkmark$ |  $\times$ |  $\times$
License $\Rightarrow$ License_of |  $\checkmark$ |  $\times$ |  $\checkmark$ |  $\times$
AlternativeName $\Rightarrow$ AlternativeName_of |  $\checkmark$ |  $\times$ |  $\times$ |  $\times$
Application $\Rightarrow$ PlugIn_of $\Rightarrow$ Specification_of |  $\checkmark$ |  $\times$ |  $\times$ |  $\times$
ProgrammingEnvironment $\Rightarrow$ Specification_of |  $\times$ |  $\times$ |  $\times$ |  $\checkmark$

##  Appendix 0.B Similarity Search Examples

###  0.B.1 Search By Entity Similarity

Table 4: Retrieval example based on entity similarity for RAG. The first row is the query entity. “sim” reflects the cosine similarity of the enities.  entity | label | sim | split | sentence
---|---|---|---|---
PhosphOrtholog | Application | 1.00 | test | To this end , we have developed an
| Creation |  |  | automated web - based tool ,
|  |  |  | PhosphOrtholog , which allows batch
|  |  |  | processing and mapping of large species -
|  |  |  | specific PTM datasets to compare overlap
|  |  |  | at a site - specific level .
SNPdetector | Application | 0.93 | train | We developed a software tool ,
| Creation |  |  | SNPdetector , for automated
|  |  |  | identification of SNPs and mutations in
|  |  |  | fluorescence - based resequencing reads .
ESPRIT - Forest | Application | 0.92 | train | In this paper we developed a new
| Creation |  |  | algorithm called ESPRIT - Forest for
|  |  |  | parallel hierarchical clustering of sequences .

###  0.B.2 Search By Sentence Similarity

Table 5: Retrieval example based on sentence similarity for RAG. The first row is the query entity. “sim” reflects the cosine similarity of the sentences. No information about the annotated entities are given for retrieval. The example sentence is the same as in Table 4 split | sim | text
---|---|---
test | 1.00 | To this end , we have developed an automated web - based tool ,
|  | PhosphOrtholog , which allows batch processing and mapping of large
|  | species - specific PTM datasets to compare overlap at a site -
|  | specific level .
train | 0.99 | Here we present a tool , Podbat ( Positioning database and analysis
|  | tool ) , that incorporates data from various sources and allows
|  | detailed dissection of the entire range of chromatin modifications
|  | simultaneously .
train | 0.99 | We designed and developed a new method , MSACompro , to
|  | synergistically incorporate predicted secondary structure ,
|  | relative solvent accessibility , and residue - residue contact
|  | information into the currently most accurate posterior probability
|  | \- based MSA methods to improve the accuracy of mult…
train | 0.99 | In this paper , we present a lossless compression tool , MAFCO ,
|  | specifically designed to compress MAF ( Multiple Alignment Format )
|  | files .

##  Appendix 0.C Prompting Examples

Figure 1: Software NER Few-Shot prompt (n=2). The shown sample is the same as
in the similarity examples in Table 4 and 5.

⬇

1# Task: Exctract entities in a closed NER setting and classify their
intention.

2## Entity Types (closed setting):

3The entity types in this closed setting are: [’Application’,
’ProgrammingEnvironment’, ’SoftwarePackageOrPlugin’, ’OperatingSystem’,
’SoftwareCoreference’]

4Do not include version information for the entities.

5if no entity of the given entity types in the context return an empty list

6

7## Intention Classification:

8For each entity mention classify the intention of the usage in the context.

9Use only and exact one intention per entity.

10The intention classes are: [’Usage’, ’Deposition’, ’Creation’, ’Mention’]

11

12## The Contexts:

13The contexts to extract the entities are about software and from scientific
publications.

14

15## Output Format:

16Return the result in a json list of objects.

17

18# Examples:

19## Context:

20"""We␣developed␣a␣software␣tool␣,␣SNPdetector␣,␣for␣automated␣identification␣of␣SNPs␣and␣mutations␣in␣fluorescence␣-␣based␣resequencing␣reads␣."""

21## Detected Named Entities with Intention:

22[ {

23 "text": "SNPdetector",

24 "label": "Application",

25 "intention": "Creation" } ]

26## Context:

27"""In␣this␣paper␣we␣developed␣a␣new␣algorithm␣called␣ESPRIT␣-␣Forest␣for␣parallel␣hierarchical␣clustering␣of␣sequences␣."""

28## Detected Named Entities with Intention:

29[ {

30 "text": "ESPRIT␣-␣Forest",

31 "label": "Application",

32 "intention": "Creation" } ]

33## Context:

34"""To␣this␣end␣,␣we␣have␣developed␣an␣automated␣web␣-␣based␣tool␣,␣PhosphOrtholog␣,␣which␣allows␣batch␣processing␣and␣mapping␣of␣large␣species␣-␣specific␣PTM␣datasets␣to␣compare␣overlap␣at␣a␣site␣-␣specific␣level␣."""

35## Detected Named Entities with Intention:

Figure 2: Attibutive NER Few-Shot prompt (n=2).

⬇

1# Task:

2## Task Description

3Exctract attributive text spans about software related entities in a closed
attibutive NER setting.

4

5## The Contexts:

6The contexts to extract the attributive text spans are about software and
from scientific publications.

7

8## The Software Related Entities:

9The software related entities are given as a list of json object with mention
and entity type.

10Do not annotate any related software entities.

11

12## Attribute Types (closed setting):

13The attribute types to identify in this closed setting are:

14 [’Version’, ’Developer’, ’Citation’, ’URL’, ’Release’, ’Abbreviation’,
’License’, ’Extension’, ’AlternativeName’]

15Keep the "Version" note separate, without overlap with other entities or
attribute spans.

16No span overlaps are allowed inbetween attributive spans of to the given
software related entites.

17Note that the attribute types must be associated with a software-related
entity of one of the following types: [’Application’,
’ProgrammingEnvironment’, ’PlugIn’, ’OperatingSystem’, ’SoftwareCoreference’].

18To help you, you will receive all software-related entities as json for each
context.

19

20## Output Format:

21Return the result in a json list of objects.

22

23# Examples:

24## Context:

25"""LAMINA␣uses␣the␣Java␣Advanced␣Imaging␣(␣JAI␣)␣package␣http://java.sun.com/javase/technologies/desktop/media/jai/␣to␣support␣common␣image␣file␣formats␣,␣which␣is␣bundled␣with␣the␣installation␣and␣hence␣no␣additional␣installation␣should␣be␣required␣."""

26### Detected Software Related Named Entities:

27[{

28 "text": "LAMINA",

29 "label": "Application"

30 }, {

31 "text": "Java␣Advanced␣Imaging",

32 "label": "Application"}]

33

34### Detected Attributive Text Spans:

35[{

36 "text": "JAI",

37 "label": "Abbreviation"

38 }, {

39 "text": "http://java.sun.com/javase/technologies/desktop/media/jai/",

40 "label": "URL"}]

Figure 3: Single-Choice QA prompt for Relation Annotation (n=2).

⬇

1# Answer the following single-choice questions on sentences from scientific
publications on the subject of software.

2 * Please note that there is exactly one answer to each question.

3## Sentence:

4"""The␣web␣service␣[44]␣available␣at␣http://www.fragit.org␣enables␣users␣to␣upload␣their␣structure␣,␣fragment␣it␣and␣download␣the␣resulting␣input␣file␣to␣GAMESS␣."""

5### Question(s):

6Which of the statements about the Citation ’[44]’ is true?

7idx_0: ’[44]’ is the citation of ’web␣service’.

8idx_1: ’[44]’ is the citation of ’GAMESS’.

9

10Which of the statements about the URL ’http://www.fragit.org’ is true?

11idx_2: ’http://www.fragit.org’ is the url of ’web␣service’.

12idx_3: ’http://www.fragit.org’ is the url of ’GAMESS’.

13

14### Answers to the questions about the sentence(s):

15idx_0: True

16idx_1: False

17idx_2: True

18idx_3: False

19

20## Sentence:

21"""ASPASIA␣,␣released␣under␣the␣Artistic␣License␣(␣2.0␣)␣,␣can␣be␣downloaded␣from␣http://www.york.ac.uk/ycil/software␣."""

22### Question(s):

23Which of the statements about the Version ’2.0’ is true?

24idx_4: ’2.0’ is the version of ’ASPASIA’.

25idx_5: ’2.0’ is the version of ’Artistic␣License’.

26

27Which of the statements about the URL ’http://www.york.ac.uk/ycil/software’
is true?

28idx_6: ’http://www.york.ac.uk/ycil/software’ is the url of ’ASPASIA’.

29idx_7: ’http://www.york.ac.uk/ycil/software’ is the url of
’Artistic␣License’.

30

31### Answers to the questions about the sentence(s):

32idx_4: False

33idx_5: True

34idx_6: True

35idx_7: False

36

37## Sentence:

38"""The␣web␣-␣tool␣can␣be␣accessed␣from␣http://www.phosphortholog.com/␣using␣any␣typical␣web␣browser␣(␣except␣Internet␣Explorer␣)␣."""

39

40### Question(s):

41Which of the statements about the URL ’http://www.phosphortholog.com/’ is
true?

42idx_8: ’http://www.phosphortholog.com/’ is the url of ’web␣-␣tool’.

43idx_9: ’http://www.phosphortholog.com/’ is the url of ’Internet␣Explorer’.

44

45### Answers to the questions about the sentence(s):

## References

  * [1] Beltagy, I., Lo, K., Cohan, A.: Scibert: A pretrained language model for scientific text. In: Conference on Empirical Methods in Natural Language Processing (2019). https://doi.org/10.18653/v1/d19-1371
  * [2] Du, C.F., Cohoon, J., Lopez, P., Howison, J.: Softcite dataset: A dataset of software mentions in biomedical and economic research publications. Journal of the Association for Information Science and Technology 72, 870 – 884 (2021). https://doi.org/10.1002/asi.24454
  * [3] Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., Wang, H.: Retrieval-augmented generation for large language models: A survey (2024). https://doi.org/10.48550/arXiv.2312.10997
  * [4] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.t., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval-augmented generation for knowledge-intensive nlp tasks. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 9459–9474. Curran Associates, Inc. (2020)
  * [5] Otto, W., Zloch, M., Gan, L., Karmakar, S., Dietze, S.: GSAP-NER: A novel task, corpus, and baseline for scholarly entity extraction focused on machine learning models and datasets. In: Bouamor, H., Pino, J., Bali, K. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2023. pp. 8166–8176. Association for Computational Linguistics, Singapore (Dec 2023). https://doi.org/10.18653/v1/2023.findings-emnlp.548, https://aclanthology.org/2023.findings-emnlp.548
  * [6] Saji, A., Matsubara, S.: Extracting information about research resources from scholarly papers. In: International Conference on Asian Digital Libraries. pp. 440–480 (2022). https://doi.org/10.1007/978-3-031-21756-2_35
  * [7] Schindler, D., Bensmann, F., Dietze, S., Krüger, F.: Somesci- a 5 star open data gold standard knowledge graph of software mentions in scientific articles. Proceedings of the 30th ACM International Conference on Information & Knowledge Management pp. 4574–4583 (2021). https://doi.org/10.1145/3459637.3482017
  * [8] Wadhwa, S., Amir, S., Wallace, B.: Revisiting relation extraction in the era of large language models. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 15566–15589. Association for Computational Linguistics, Toronto, Canada (Jul 2023). https://doi.org/10.18653/v1/2023.acl-long.868, https://aclanthology.org/2023.acl-long.868
  * [9] Wan, Z., Cheng, F., Mao, Z., Liu, Q., Song, H., Li, J., Kurohashi, S.: GPT-RE: In-context learning for relation extraction using large language models. In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 3534–3547. Association for Computational Linguistics, Singapore (Dec 2023). https://doi.org/10.18653/v1/2023.emnlp-main.214, https://aclanthology.org/2023.emnlp-main.214
  * [10] Wang, S., Sun, X., Li, X., Ouyang, R., Wu, F., Zhang, T., Li, J., Wang, G.: Gpt-ner: Named entity recognition via large language models (2023). https://doi.org/10.48550/arXiv.2304.10428
  * [11] Xie, T., Li, Q., Zhang, Y., Liu, Z., Wang, H.: Self-improving for zero-shot named entity recognition with large language models (2024). https://doi.org/10.48550/arxiv.2311.08921
  * [12] Xu, D., Chen, W., Peng, W., Zhang, C., Xu, T., Zhao, X., Wu, X., Zheng, Y., Chen, E.: Large language models for generative information extraction: A survey. ArXiv (2023). https://doi.org/10.48550/arxiv.2312.17617


#  Gopher: Categorical probabilistic forecasting with graph structure via
local continuous-time dynamics

Ke Alexander Wang
Stanford University
<EMAIL_ADDRESS>
&Danielle Maddix
Amazon Research
<EMAIL_ADDRESS>
&Yuyang Wang
Amazon Research
<EMAIL_ADDRESS>
Work done as an intern at Amazon Research

###### Abstract

We consider the problem of probabilistic forecasting over categories with
graph structure, where the dynamics at a vertex depends on its local
connectivity structure. We present Gopher, a method that combines the
inductive bias of graph neural networks with neural ODEs to capture the
intrinsic local continuous-time dynamics of our probabilistic forecasts. We
study the benefits of these two inductive biases by comparing against baseline
models that help disentangle the benefits of each. We find that capturing the
graph structure is crucial for accurate in-domain probabilistic predictions
and more sample efficient models. Surprisingly, our experiments demonstrate
that the continuous time evolution inductive bias brings little to no benefit
despite reflecting the true probability dynamics.

##  1 Introduction

In categorical probabilistic forecasting, we seek to predict a discrete
probability distribution $\mathbf{p}(t)$ at some instantaneous time $t$, based
on observed time-stamped data [10]. Consider the example of forecasting the
most likely locations of the next earthquake over a finite set of locations at
$t$, given the history of earthquake times and locations. We can view
locations as vertices $V$ on a graph $G=(V,E)$ with edges $E$ that represent
adjacency. Specifically, the probability of an earthquake at node $v\in V$ in
the near future is mostly influenced by the probability of earthquakes at
nodes within its neighborhood. This type of graphical structure also appears
in other problems, including traffic forecasting [28], information diffusion
in social networks [1], epidemic diffusion [24, 13], urban conflict patterns
[16], and is an example of a marked temporal point process [6].

In this paper, we consider categorical probabilistic forecasts where there is
a graphical structure to inform us of the local dynamics governing
$\mathbf{p}(t)$ over time. We formalize the intuition that each component of
the probability vector $\mathbf{p}(t)\in\mathbb{R}^{|V|}$ obeys local dynamics
using the differential equation

$\frac{\mathrm{d}{}p_{v}}{\mathrm{d}{}t}=g\left(p_{v},\\{p_{u}\mid u\in
N(v)\\},t\right),$ (1)

which we use to inform our model’s inductive bias. Here, $g$ governs the local
dynamics, $\mathcal{N}(v)\subseteq V$ denotes the set of neighboring nodes of
$v$, and $p_{v}$ denotes the probability at node $v$. To capture the
equivariant local dynamics of our forecast $\mathbf{p}(t)$, we propose Gopher,
a model that learns a neural ODE [4] with graph neural network (GNN) [26]
dynamics.

Our method Gopher introduces two inductive biases to aid with probabilistic
forecasting over graph-structured categories by 1) utilizing graph structure
explicitly and 2) introducing temporal evolution through a neural ODE. To
disentangle the benefits of these two biases, we introduce two baseline
models, ablating each bias. We find that utilizing the known graph structure
results is key, and results in 10x improvements in accuracy and sample
efficiency. On the other hand, explicitly modelling the temporal dynamics
surprisingly results in little benefits.

##  2 Gopher: Forecasting with temporal dynamics and graph structure

Let $G=(V,E)$ be a graph, and let $t_{i}\in\mathbb{R}_{0}^{+}$ denote the
timestamp of an event at node $v_{i}\in V$. Given $G$ and an irregularly
sampled dataset $\mathcal{D}=\\{(t_{i},v_{i})\\}_{i=1}^{N}$, we want to learn
the probability vector $\mathbf{p}(t)\in\mathbb{R}^{|V|}$ of each $v\in V$ at
any time $t$. We wish to model the dynamics of $\mathbf{p}(t)$ such that the
change in the probability at node $v$ depends only on the neighborhood
$\mathcal{N}(v)$ around $v$, as described in Equation 1. However, directly
parameterizing $g$ from Equation 1 with a neural ODE can violate conservation
of probability: $\mathbf{1}{}^{\top}\mathbf{p}(t)=1$.

Instead of explicitly enforcing the sum constraint into our neural ODE, we
model the dynamics in a continuous-time embedding space from which we derive
the dynamics $\mathrm{d}{}\mathbf{p}/\mathrm{d}{}t$. Specifically, let
$\mathbf{Z}_{0}\in\mathbb{R}^{|V|\times D}$, where $D$ denotes the embedding
space dimension. We use $\mathbf{z}_{0,i}$ to denote row $i$ of
$\mathbf{Z}_{0}$ at initial time $t_{0}$, corresponding to the embedding of
node $v_{i}$. We then model the dynamics of the continuous-time embeddings
$\mathbf{Z}(t)\in\mathbb{R}^{|V|\times D}$ via

$\displaystyle\frac{\mathrm{d}{}\mathbf{Z}}{\mathrm{d}{}t}$
$\displaystyle=g(\mathbf{Z},G,t)\quad\text{subject to}\
\quad\mathbf{Z}(t_{0})=\mathbf{Z}_{0},$ (2)

where $g$ is the learned graph neural network (GNN) dynamics. To map
$\mathbf{Z}(t)$ to a probability space while preserving equivariance, we learn
a shared projection $\pi:\mathbb{R}^{D}\to\mathbb{R}$ such that

$\displaystyle\mathbf{p}(t)=\operatorname{Softmax}\left(\pi(\mathbf{z}(t)_{1}),\ldots,\pi(\mathbf{z}(t)_{|V|})\right).$
(3)

Provided that $g$ and $\pi$ are differentiable, which can be done using smooth
activation functions, our model then implicitly models the local temporal
dynamic of our problem in Equation 1. Finally, we train our model Gopher by
maximizing the log likelihood $\sum_{i=1}^{N}\log p_{v_{i}}(t_{i})$ with
respect to the parameters of $\pi$, $g$, and the initial condition
$\mathbf{Z}_{0}$.

#### Incorporating node attributes.

In some cases $G$ may have node attributes $\\{a_{v}\\}_{v\in V}$ for each
node $v\in V$ that affect the interaction dynamics, such as the geographical
coordinates of each node in a spatial graph or the demographics of a user in a
social network. Node attributes can be easily incorporated by letting the
initial node embeddings be a learned function of the attributes,
$\mathbf{Z}_{0}[v]=\psi_{v}(a_{v})$, and optimizing with respect to the
parameters of $\\{\psi_{v}\\}$.

#### Related works.

Our paper lies at the intersection of probabilistic forecasting, neural ODEs,
and graph neural networks (GNNS), and can be seen as the discrete analogue of
continuous normalizing flows [4, 11, 5] on manifolds [17, 18]. Probabilistic
forecasting seeks to predict a full distribution at each time step [14, 10],
with contemporary methods often relying on deep probabilistic models [22, 25,
21, 20]. A direct application of categorical probabilistic forecasts is marked
temporal point processes, which learn the rate of an event type $v$ at time
$t$, summarized by the conditional intensity function
$\lambda(t,v)=\lambda(t)\cdot p_{v}(t)$ [6]. The inductive bias of a learnable
ODE with GNN dynamics has also been explored in the context of other problems,
including graph generation [7], node classification [19, 2], multi-particle
trajectory prediction [19], learning partial differential equations [15], and
knowledge graph forecasting [12].

##  3 Results: I Can’t Believe Temporal Dynamics Don’t Matter!

#### Synthetic datasets.

Figure 1: (Top) Advection on a cyclic graph set chosen so that the probability
mass is transported more strongly in the counter-clockwise direction. (Bottom)
Advection on a random geometric graph with randomly chosen edge weights.
(Both) Colors are shown in log-scale to make dynamics more visually-apparent.
Light-gray coloring corresponds to $\log(1/|V|)$ which is the steady-state
probability mass for each node.

We apply our method to model the mark component of a marked temporal point
process (TPP) occurring on the nodes of a graph such that $p_{v}(t)$ is the
probability of an event occurring on vertex $v$ at time $t$. We create a
synthetic dataset where events occur over time on a directed graph $G$, with
node probabilities that obey graph advection as an example of local dynamics
[3]. Graph advection conserves the total probability by ensuring
$\mathbf{1}{}^{\top}\mathrm{d}{}\mathbf{p}/\mathrm{d}{}t=0$. We represent the
graph $G$ by the weighted adjacency matrix $A$, where $A_{uv}>0$ for $(u,v)\in
E$. We sample sequences of events over time $[0,T]$ from a homogeneous Poisson
process with constant temporal intensity $\lambda(t)=\lambda=2.5$ and temporal
node probability $\mathbf{p}(t)\in\mathbb{R}^{|V|}$ governed by the graph
advection equation [3]

$\frac{\mathrm{d}{}\mathbf{p}}{\mathrm{d}{}t}=-L_{\text{out}}(A){}^{\top}\mathbf{p}\iff\frac{\mathrm{d}{}p_{v}}{\mathrm{d}{}t}=\sum_{v:\
(v,u)\in E}A_{vu}p_{v}-\sum_{v:\ (u,v)\in E}A_{uv}p_{u}.$ (4)

Here, $L_{\text{out}}(A):=D_{\text{out}}(A)-A$ denotes the out-degree graph
Laplacian, and $D_{\text{out}}(G)$ denotes the diagonal out-degree matrix with
$D_{\text{out}}(G)_{ii}=\sum_{j}A_{ij}$.

We create two graphs structures for our synthetic datasets, a ring graph and a
random geometric graph, and visualize their advection on the graph over time
in Figure 1; see Appendix D for more details on their construction. We also
visualize the advection dynamics of each component of $\mathbf{p}(t)$ for the
ring graph in Figure 9 of Appendix D. We use $T=5$ seconds for the ring graph
dataset and $T=1$ second for the geometric graph dataset. Since the timestamps
are sampled from a Poisson process and _not_ equidistantly spaced over
$[0,T]$, the continuous time aspect of the problem is clearly evident in the
dataset.

#### Evaluating each inductive bias of Gopher.

Figure 2: KL divergence between the true probabilities $\mathbf{p}(t)$ and the
predicted probabilities $\mathbf{q}(t)$ trained on 1024 sampled sequences.
Gray is the training set time interval $[0,T]$ and red is the extrapolation
region $[T,2T]$ beyond the training set.

We evaluate the accuracy and sample-efficiency improvements from incorporating
graph structure and modelling temporal dynamics in Gopher. To disentangle the
effects of these two inductive biases, we compare our model to two baseline
models. The first is a two-layer MLP that acts on node-embeddings concatenated
with time, which has none of the above inductive biases. The second is a
single-layer GNN that also acts on node-embeddings concatenated with time. The
GNN incorporates the explicit graph structure, but does not incorporate
dynamical systems structure. We refer to the models as NaiveMLP and NaiveGNN
respectively. In our experiments, Gopher learns $g$ using a Graph Isomorphism
Network (GIN) layer [27] parameterized by a two-layer MLP; we use another two-
layer MLP for the projection $\pi$. NaiveGNN uses the same GIN architecture
and projection except that it does not learn a differential equation. Finally,
NaiveMLP replaces the GIN layer with a two layer MLP. See Appendix D for
further details on our experiment hyperparameters.

Figure 2 shows the KL divergence betweeen the ground truth $\mathbf{p}(t)$ and
the learned predictions over time for the ring graph. We summarize the KL
divergence over $[0,T]$ in Figure 3 by the geometric mean since the error
varies over multiple overs of magnitudes over time [9]. In both figures, we
show the 95% confidence intervals over 3 seeds. For both datasets, there is
10x difference in accuracy between the graph structured models and NaiveMLP,
indicating that utilizing the graph structure is greatly beneficial. Though
NaiveGNN does not explicitly model the local temporal dynamics of the
datasets, it performs nearly identically to our model Gopher in fitting
$\mathbf{p}(t)$ over the training interval $[0,T]$. In principle, Gopher has
the best chance of extrapolating to the $[T,2T]$ time period not seen during
training since Gopher explicitly models the local dynamics. However, Gopher’s
poor extrapolation ability suggests that its learned dynamics do not actually
reflect the true dynamics. Indeed, in Figure 7 of Appendix C we show that
although Gopher can fit the training data well, it is brittle to edge
deletions, further indicating Gopher does not learn the true dynamics.

Figure 3: Sample complexity of each model on the geometric graph dataset
(left) and cyclic graph dataset (right). The error is measured in terms of the
geometric mean of the KL divergence over the evaluation time period $[0,T]$.
Prediction of uniform probability corresponds to a geometric mean KL
divergence of $0.72$ for the geometric graph and $0.15$ for the cyclic graph.

#### Real-world dataset.

Figure 4: Learned $\mathbf{p}(t)$ over a subset of the counties fit using the
COVID-19 dataset preprocessed by Chen et al. [5]. Only the graph-based models
are able to capture the variations over time. See Appendix A for the empirical
distribution of $\mathbf{p}(t)$.

We use data released publicly by the New York Times [23] on daily COVID-19
cases in New Jersey state to construct a real-world categorical probabilistic
forecasting dataset, following the preprocessing script of Chen et al. [5]. We
aggregate the cases by county and form a graph with 21 nodes where each node
is a county and each edge is a county border. Using the train/test split from
Chen et al. [5], we obtain per event log likelihoods with 1-standard-
deviations of $-2.766\pm 0.003$ for NaiveMLP, $-2.768\pm 0.003$ for NaiveGNN,
and $-2.767\pm 0.000$ for Gopher over 3 seeds. However, these likelihoods are
not representative of the model differences since we find a large distribution
shift between the train and test distribution shown in Figure 5 of Appendix A.
This distribution shift causes the models to perform equally poorly on the
test set. In actuality, NaiveMLP completely fails to capture variations in
$\mathbf{p}(t)$ over time, as shown in Figure 4.

##  4 Discussion

Although the inductive biases of Gopher, directly reflect properties of
categorical forecasting with local continuous-time dynamics, our experiments
find that, surprisingly, explicitly modelling the temporal dynamics does not
improve performance. Most of the performance gains of Gopher come from
incorporating a graph structure, which can be done with a simple baseline
model like NaiveGNN. The failure of Gopher can be attributed to the fact that
the learned dynamics in the embedding space do not accurately reflect the
ground truth dynamics in probability space.

## References

  * Bakshy et al. [2012] Eytan Bakshy, Itamar Rosenn, Cameron Marlow, and Lada Adamic.  The role of social networks in information diffusion.  In _Proceedings of the 21st International Conference on World Wide Web_ , WWW ’12, pages 519–528, New York, NY, USA, April 2012. Association for Computing Machinery.  ISBN 978-1-4503-1229-5.  doi: 10.1145/2187836.2187907.
  * Chamberlain et al. [2021] Ben Chamberlain, James Rowbottom, Maria I. Gorinova, Michael Bronstein, Stefan Webb, and Emanuele Rossi.  GRAND: Graph Neural Diffusion.  In _International Conference on Machine Learning_ , pages 1407–1418. PMLR, July 2021.
  * Chapman and Mesbahi [2011] Airlie Chapman and Mehran Mesbahi.  Advection on graphs.  _IEEE Conference on Decision and Control and European Control Confereence (CDC-ECC)_ , 50:1461–1466, December 2011.
  * Chen et al. [2018] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.  Neural Ordinary Differential Equations.  In _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc., 2018.
  * Chen et al. [2020] Ricky T. Q. Chen, Brandon Amos, and Maximilian Nickel.  Neural Spatio-Temporal Point Processes.  In _International Conference on Learning Representations_ , September 2020.
  * Daley and Vere-Jones [2003] D. J. Daley and D. Vere-Jones.  _An Introduction to the Theory of Point Processes: Volume I: Elementary Theory and Methods_.  Probability and Its Applications, An Introduction to the Theory of Point Processes. Springer-Verlag, New York, second edition, 2003.  ISBN 978-0-387-95541-4.  doi: 10.1007/b97277.
  * Deng et al. [2019] Zhiwei Deng, Megha Nawhal, Lili Meng, and Greg Mori.  Continuous Graph Flow.  _arXiv:1908.02436 [cs, stat]_ , September 2019.
  * Dupont et al. [2019] Emilien Dupont, Arnaud Doucet, and Yee Whye Teh.  Augmented Neural ODEs.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.
  * Finzi et al. [2020] Marc Finzi, Ke Alexander Wang, and Andrew G. Wilson.  Simplifying Hamiltonian and Lagrangian Neural Networks via Explicit Constraints.  _Advances in Neural Information Processing Systems_ , 33:13880–13889, 2020.
  * Gneiting and Katzfuss [2014] Tilmann Gneiting and Matthias Katzfuss.  Probabilistic Forecasting.  _Annual Review of Statistics and Its Application_ , 1(1):125–151, 2014.  doi: 10.1146/annurev-statistics-062713-085831.
  * Grathwohl et al. [2018] Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud.  FFJORD: Free-Form Continuous Dynamics for Scalable Reversible Generative Models.  In _International Conference on Learning Representations_ , September 2018.
  * Han et al. [2021] Zhen Han, Zifeng Ding, Yunpu Ma, Yujia Gu, and Volker Tresp.  Temporal Knowledge Graph Forecasting with Neural ODE.  _arXiv:2101.05151 [cs]_ , August 2021.
  * Huang et al. [2010] Wenzhang Huang, Maoan Han, and Kaiyu Liu.  Dynamics of an SIS reaction-diffusion epidemic model for disease transmission.  _Mathematical Biosciences & Engineering_, 7(1):51, 2010.  doi: 10.3934/mbe.2010.7.51.
  * Hyndman and Athanasopoulos [2018] Robin John Hyndman and George Athanasopoulos.  _Forecasting: Principles and Practice_.  OTexts, Australia, 2nd edition, 2018.
  * Iakovlev et al. [2020] Valerii Iakovlev, Markus Heinonen, and Harri Lähdesmäki.  Learning continuous-time PDEs from sparse data with graph neural networks.  In _International Conference on Learning Representations_ , September 2020.
  * Linderman and Adams [2014] Scott Linderman and Ryan Adams.  Discovering Latent Network Structure in Point Process Data.  In _International Conference on Machine Learning_ , pages 1413–1421. PMLR, June 2014.
  * Lou et al. [2020] Aaron Lou, Derek Lim, Isay Katsman, Leo Huang, Qingxuan Jiang, Ser Nam Lim, and Christopher M De Sa.  Neural manifold ordinary differential equations.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 17548–17558. Curran Associates, Inc., 2020.
  * Mathieu and Nickel [2020] Emile Mathieu and Maximilian Nickel.  Riemannian Continuous Normalizing Flows.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 2503–2515. Curran Associates, Inc., 2020.
  * Poli et al. [2021] Michael Poli, Stefano Massaroli, Clayton M. Rabideau, Junyoung Park, Atsushi Yamashita, Hajime Asama, and Jinkyoo Park.  Continuous-Depth Neural Models for Dynamic Graph Prediction.  _arXiv:2106.11581 [cs, stat]_ , June 2021.
  * Rangapuram et al. [2018] Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski.  Deep State Space Models for Time Series Forecasting.  In _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc., 2018.
  * Rasul et al. [2020] Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs M. Bergmann, and Roland Vollgraf.  Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows.  In _International Conference on Learning Representations_ , September 2020.
  * Salinas et al. [2020] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski.  DeepAR: Probabilistic forecasting with autoregressive recurrent networks.  _International Journal of Forecasting_ , 36(3):1181–1191, July 2020.  ISSN 0169-2070.  doi: 10.1016/j.ijforecast.2019.07.001.
  * Times [2021] The New York Times.  Coronavirus (Covid-19) Data in the United States, 2021.  URL https://github.com/nytimes/covid-19-data.
  * Wang et al. [2021] Rui Wang, Danielle Maddix, Christos Faloutsos, Yuyang Wang, and Rose Yu.  Bridging physics-based and data-driven modeling for learning dynamical systems.  In Ali Jadbabaie, John Lygeros, George J. Pappas, Pablo A. Parrilo, Benjamin Recht, Claire J. Tomlin, and Melanie N. Zeilinger, editors, _Proceedings of the 3rd Conference on Learning for Dynamics and Control_ , volume 144 of _Proceedings of Machine Learning Research_ , pages 385–398. PMLR, 07 – 08 June 2021.  URL https://proceedings.mlr.press/v144/wang21a.html.
  * Wang et al. [2019] Yuyang Wang, Alex Smola, Danielle Maddix, Jan Gasthaus, Dean Foster, and Tim Januschowski.  Deep Factors for Forecasting.  In _International Conference on Machine Learning_ , pages 6607–6617. PMLR, May 2019.
  * Wu et al. [2021] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu.  A Comprehensive Survey on Graph Neural Networks.  _IEEE Transactions on Neural Networks and Learning Systems_ , 32(1):4–24, January 2021.  ISSN 2162-237X, 2162-2388.  doi: 10.1109/TNNLS.2020.2978386.
  * Xu et al. [2018] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka.  How Powerful are Graph Neural Networks?  In _International Conference on Learning Representations_ , September 2018.
  * Yu et al. [2018] Bing Yu, Haoteng Yin, and Zhanxing Zhu.  Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting.  In _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ , IJCAI’18, pages 3634–3640, Stockholm, Sweden, July 2018. AAAI Press.  ISBN 978-0-9992411-2-7.

##  Appendix A Distribution shift in our real-world dataset

Figure 5: Distribution shift between the train and test distribution of the
New Jersey COVID-19 cases created by Chen et al. [5] when binned by the 21
counties. For each of the two distributions, the height of each violin plots
is normalized by the total count of observations in that split, i.e. size of
training set or size of test set. For most vertices, there are fewer COVID
cases later in the 7 day interval in the test set than in the training set.

##  Appendix B Learned forecasts on COVID-19 dataset

Figure 6: A copy of Figure 4 for easier comparison to the empirical
distribution in Figure 5

##  Appendix C Model robustness

Figure 7: (Top) The learned and ground truth $\mathbf{p}(t)$ for the ring
graph dataset. (Bottom) The predicted $\mathbf{p}(t)$ after we remove all
edges from the ring graph. Notice that the ground truth of a completely
disconnected graph is to have $\mathbf{p}(t)=\mathbf{p}(0)$ for all $t$.
However, all of the models fail completely on this new disconnected graph,
suggesting that they do not learn the true dependence of $\mathbf{p}(t)$ on
the graph structure.

##  Appendix D Implementation details

Figure 8: Architecture of Gopher.

We use 2 layer MLPs with 64 hidden units per layer whenever we use a MLP. We
use Swish activations to ensure smoothness of our dynamics. We also use an
augmented neural ODE [8], using 16 dimensions as augmented dimensions out of
the 64 hidden dimensions.

###  D.1 Model architectures.

####  Gopher.

We parameterize the dynamics $g$ from Equation 2 using one graph isomorophism
network layer parameterized by a MLP. We also use a MLP to model the
projection $\pi$.

####  NaiveGNN.

We use the same architecture as Gopher, namely a GIN layer followed by a
projection $\pi$. However, instead of using the model to parameterize ODE
dynamics, we directly input the node embeddings concatenated with time $t$
through the GNN.

####  NaiveMLP.

We replace the GIN layer of NaiveGNN with a MLP, keeping all else the same.

###  D.2 Training procedure and dataset details.

To maximize hardware parallelism, we parallelize our neural ODE computation
across sequence timesteps and across sequences using the time-
reparameterization trick outlined in Chen et al. [5]. For the synthetic
datasets, we use the AdamW optimizer with 0.01 learning rate and batch size 64
for 30 epochs. For the COVID-19 dataset, we use a $3\times 10^{-4}$ learning
rate and batch size 4 for 15 epochs. Here, each batch consists of multiple
sequences drawn from the training period $[0,T]$. We use $T=5$ for the ring
graph, $T=1$ for the geometric graph, and $T=8$ for the New Jersey counties
graph. We generate the ring graph dataset by using hand-set coefficients for
the edge weights $A_{uv}$ to allow for counter-clockwise transport. We
generate the geometric graph dataset by generating a random geometric graph
via the `networkx` python package and drawing a random sample of
$\\{A_{uv}\\}$.

Figure 9: The dynamics of each component of $\mathbf{p}(t)$ on the cyclic
graph. Each component corresponds to the probability of a vertex on the graph.