# A pathwise parameterisation for stochastic transport

Oana Lang Wei Pan

###### Abstract

In this work we set the stage for a new probabilistic pathwise approach to
effectively calibrate a general class of stochastic nonlinear fluid dynamics
models. We focus on a 2D Euler SALT equation, showing that the driving
stochastic parameter can be calibrated in an optimal way to match a set of
given data. Moreover, we show that this model is robust with respect to the
stochastic parameters.

##  1 Introduction

A fundamental challenge in observational sciences, such as weather forecasting
and climate change predictions, is the modelling of uncertainty due, for
example, to unknown or neglected physical effects, and incomplete information
in both the data and the formulation of the theoretical models for prediction.
Various dynamical parameterisation approaches have been proposed to tackle
this challenge, see e.g. [7], [4], [12], [5], [1]. Of particular interest are
the recently developed Data Driven models, that accommodate uncertainty by
predicting both the expected future measurement values and their
uncertainties, based on input from measurements and statistical analysis of
the initial data. To effectively incorporate uncertainty in the data driven
approach, such predictions are made in a probabilistic sense. Additionally, a
data assimilation procedure is used to take into account the time integrated
information obtained from the data being observed along the solution path
during the forecast interval as “in flight corrections”.

In the geoscience community, data assimilation (DA) refers to a set of
methodologies designed to efficiently combine past knowledge of a geophysical
system (in the form of a numerical model) with new information about that
system (in the form of observations). DA is a central component of Numerical
Weather Prediction where it is used to improve forecasting by adjusting the
model parameters and reducing the uncertainties. To achieve this, a stochastic
feedback loop between the model and the observation may be introduced: the
assimilation of more data during the prediction interval will then decrease
the uncertainty of the forecasts based on the initial data, by selecting the
more likely paths as more observational data is collected. This is the basis
of the so-called ensemble data assimilation which uses a set of model
trajectories that are intermittently updated according to data.

A key step for ensuring the successful application of the combined stochastic
parameterisation and data assimilation procedure, is the “correct” calibration
of stochastic model parameters. For Stochastic Advection by Lie Transport
(SALT) and Location Uncertainty (LU) models, current numerical methods for
calibration, see [4], [1], [5], have largely been inspired by the physical
interpretation of the models derivations. More specifically on the assumption
that the flow map is decoupled into a slow scale mean part and a fast scale
fluctuating part. In the references mentioned before, it was shown that these
methods are effective and led to successful combination of data driven models
and state of the art data assimilation techniques.

In this work, we wish to investigate the feasibility and viability of
probabilistic pathwise approach for calibration. Our general aim is to explore
such ideas for a wide class of nonlinear stochastic transport models. This
will be very useful in data assimilation problems, as in real world
applications the signal is usually observed through discrete observations, but
no results of this type for SALT or LU models have been obtained before.
Currently, Lagrangian particle trajectories are simulated starting from each
point on both the physical grid and its refined version, then the differences
between the particle positions are used to calibrate the noise. This is
computationally expensive and not fully justified from a theoretical
perspective. In the same spirit as [3] but with a more complicated noise term
and without any smoothing effects of a Laplacian, we propose an approach which
uses high-frequency in time and low-frequency in space observations of a
single path of the solution, to rigorously infer properties of the stochastic
parameters. The knowledge of the noise is crucial for determining the
behaviour of the solution and for assessing to what degree the solution of the
coarse resolution SPDE deviates from the solution of the fine resolution PDE
in the model reduction procedure, so an optimal calibration of the noise
parameters is relevant from both a theoretical and an applied perspective.

In this work we look at stochastic calibration for the two-dimensional
incompressible Euler equation in vorticity form. This stochastic equation
models the local rotation of a fluid flow in the presence of spatial
uncertainties and it has been derived from fundamental principles in [7]. This
equation is a key ingredient in modelling phenomena in oceanography and in
order to ensure that it efficiently encodes the small-scale variablity in the
upper part of the ocean, one needs to specify the stochastic parameters based
on real observations. One of main issues in parameter estimation using real
data is the fact that the model parameters do not map to observations in a
unique way (model identifiability problem, see e.g. [2]). For this reason, we
believe that a probabilistic approach is much more suitable.

The 2D Euler equation in the form derived in [7] and studied in [4], [5] and
[9] reads:

$d{\omega}_{t}+u_{t}\cdot\nabla{\omega}_{t}dt+\displaystyle\sum_{i=1}^{\infty}\xi_{i}\cdot\nabla{\omega}_{t}\circ
dW_{t}=0$ (1)

where $u=(u^{1},u^{2})$ is the fluid velocity, $\omega=curl\
u=\partial_{2}u_{1}-\partial_{1}u_{2}$ is the vorticity, $(\xi_{i})_{i}$ are
divergence-free time-independent vector fields such that

$\displaystyle\sum_{i=1}^{\infty}\|\xi_{i}\|_{k+1,\infty}^{2}<\infty$ (2)

and $(W^{i})_{i\in\mathbb{N}}$ is a sequence of independent Brownian motions.
Global well-posedness for equation (1) has been studied in [9] and the
numerical and data assimilation perspective has been studied in [4] and [5].
In [9] the authors have proven that equation (1) admits a uniques pathwise
solution which lives in the Sobolev space $\mathcal{W}^{k,2}(\mathbb{T}^{2})\
(k\geq 2)$ when $\omega_{0}\in\mathcal{W}^{k,2}(\mathbb{T}^{2})$ and can be
extended to $L^{\infty}(\mathbb{T}^{2})$ when $\omega_{0}\in
L^{\infty}(\mathbb{T}^{2})$.

In this paper we consider the following SPDE on the two-dimensional torus
$\mathbb{T}^{2}=\mathbb{R}^{2}/\mathbb{Z}^{2}$, driven by a 1-dimensional
Brownian motion $W$:

$d{\omega}_{t}+u_{t}\cdot\nabla{\omega}_{t}dt+\xi\cdot\nabla{\omega}_{t}\circ
dW_{t}=0$ (3)

where $u$ and $\omega$ are as above and $\circ$ denotes Stratonovich
integration. We impose the following condition on the stochastic parameter
$\xi$, in the same spirit as (2):

$\|\xi\|_{k+2,\infty}^{2}<\infty$ (4)

with $k\geq 2$. This condition ensures that for any
$f\in\mathcal{W}^{2,2}(\mathbb{T}^{2})\cap\mathcal{W}^{2,\infty}(\mathbb{T}^{2})$,

$\|\xi\cdot\nabla f\|_{2}^{2}\leq C\|f\|_{1,2}^{2}\ \ \ \ \ \ \
\|\xi\cdot\nabla(\xi\cdot\nabla f)\|_{2}^{2}\leq C\|f\|_{2,2}^{2}$ (5)
$\|\xi\cdot\nabla f\|_{\infty}^{2}\leq C\|f\|_{1,\infty}^{2}\ \ \ \ \ \ \
\|\xi\cdot\nabla(\xi\cdot\nabla f)\|_{\infty}^{2}\leq C\|f\|_{2,\infty}^{2}.$
(6)

######  Remark 1.

We can view the stochastic part as a space-time noise $(\xi,W)$ where the
spatial component is given by $\xi$ and the time component is a standard
Brownian motion. This perspective is many times useful in numerical
applications where $(\xi\circ dW_{t})\cdot\nabla$ is implemented as a random
operator applied to the solution $\omega$.

The problem of parameter estimation, known also as statistical inference, is
technically challenging for such (infinite-dimensional) SPDEs driven by
transport noise, as most methods used in the literature benefit from a
diagonalizable structure of the underlying space-covariance matrices. This
structure is specific for additive noise and therefore it does not apply in
our case. Also, most results are obtained for stochastic variations of the
heat equation, which contain a smoothing Laplace operator (see for instance
[3]). Our model does not contain a Laplacian a priori, and therefore we cannot
exploit the properties of a heat kernel. These makes the analysis much harder.

Contributions of the paper
In this work, we focus on equation (3) from two perspectives:

  * •

First, we show that the driving stochastic parameter $\xi$ can be calibrated
in an optimal way to match a set of high-frequency in time given data. This is
done using a forced and damped version of the equation and a parametric form
of the stream function and the corresponding stochastic parameter which is
implemented using an orthonormal basis. Our technique can be explicitly
applied to calibrate the 2D Euler model using real oceanic data and we intend
to do this in coming work.

  * •

Second, we show that the original 2D Euler model is robust with respect to the
stochastic parameters $\xi$ in the sense that if we consider two couples
$(\omega^{1},\xi^{1})$ and $(\omega^{2},\xi^{2})$ which solve equation (3),
then the $L^{2}$ distance between $\omega^{1}$ and $\omega^{2}$ can be
controlled using the initial conditions and the difference between $\xi^{1}$
and $\xi^{2}$ only (see Section 4). This is important in applications as it
shows that if we consider approximate values for $\xi$, the corresponding
model solution remains close to the true solution.

Structure of the paper
In Section 2 below we present the problem formulation. In Section 3 we
introduce the methodology. In Section 4 we prove the robustness of the
original model and in Section 5 we present the numerical results.

##  2 Problem formulation

Let $(\Omega,\mathcal{F},(\mathcal{F}_{t})_{t\geq 0},\mathbb{P})$ be a
filtered probability space and $W$ a one-dimensional Brownian motion adapted
to the complete and right-continuous filtration $(\mathcal{F}_{t})_{t\geq 0}$.

We assume we are given a finite sequence of _high frequency_ in time vorticity
fields, that are denoted by $\omega^{*}_{t_{i}}(x)$, $i=1,\dots,N$, and are
adapted to $(\mathcal{F}_{t})_{t\geq 0}$. We take the view that the
$\omega^{*}_{t_{i}}$’s are the given observation _data_. We further assume
that $\omega^{*}_{t_{i}}\in\mathcal{W}^{k,2}({\mathbb{T}^{2}})$.

Writing $\omega_{\xi}$ to denote solutions to the model (3) for a given vector
field $\xi$, the generic problem we are interested in is to find a $\xi$ so
that solutions to (3) matches the data as best as possible, i.e.

$\displaystyle\arg\min_{\xi}\|\omega^{*}-\omega_{\xi}\|$ (7)

for some suitable norm.

The dimension of the observations currently coincides with the number of
sources of noise, that is we have a determined system. However, in practice
this is not always a realistic assumption and in future work we will look at
underdetermined or overcomplete systems i.e. when the number of noise sources
is larger than the dimension of the observation operator.

In general, the infinite dimensional optimisation problem (7) may be too hard
to solve in practice. We thus make concrete the form of $\xi$. Let
$({\mathfrak{e}}_{j})_{j\in\mathbb{N}}$ be an orthonormal basis in
$L^{2}(\mathbb{T}^{2})$. We assume the following parametric form for the
stream function of $\xi$, which is henceforth denoted by $\zeta$,

$\zeta(x)=\sum_{j=1}^{\infty}\alpha_{j}{\mathfrak{e}}_{j},$ (8)

where $\alpha_{j}$ are reals. Then

$\xi(x)=\nabla^{\perp}\zeta(x)=\sum_{j=1}^{\infty}\alpha_{j}\nabla^{\perp}{\mathfrak{e}}_{j}(x)$
(9)

and the optimisation problem (7) then reduces to finding the coefficients
$\alpha_{j}$.

##  3 Methodology

We will first introduce a couple of known results for vorticity equations, for
further discussions on the topic see [10] or [11]. The link between the
vorticity and the velocity vector field in equation (3) is uniquely
established using the Biot-Savart operator $K$:

${\bf
u}(x)=(K\star\omega)(x)=\displaystyle\int_{\mathbb{T}^{2}}K(x-y)\omega(y)dy$
(10)

with

$K(x)=\nabla^{\perp}G(x)=\displaystyle\sum_{k\in\mathbb{Z}^{2}\setminus\\{0\\}}\frac{ik^{\perp}}{\|k\|^{2}}e^{ik\cdot
x}$ (11)

where $G$ is the Green function of the operator $-\Delta$ on $\mathbb{T}^{2}$

$G(x)=\displaystyle\sum_{k\in\mathbb{Z}^{2}\setminus\\{0\\}}\frac{e^{ik\cdot
x}}{\|k\|^{2}}$

and $k=(k_{1},k_{2})$, $k^{\perp}=(k_{2},-k_{1})$. It is known that, for any
$k\geq 0$, there exists a constant $C_{k,2}$, independent of $u$ such that

$\|{\bf u}\|_{k+1,2}\leq C_{k,2}\|\omega\|_{k,2}.$

If $\psi:\mathbb{T}^{2}\times[0,\infty)\rightarrow\mathbb{R}$ is a solution
for $\Delta\psi=-\omega$ then ${\bf u}=\nabla^{\perp}\psi$ solves
$\omega=\rm{curl}\ {\bf u}$, so ${\bf u}=-\nabla^{\perp}\Delta^{-1}\omega$.
The reconstruction of ${\bf u}$ from $\omega$ is ensured by the
incompressibility condition $\nabla\cdot{\bf u}=0$ and a periodic,
distributional solution of $\Delta\psi=-\omega$ is given by

$\psi(x)=(G\star\omega)(x).$

From (3) we have

$\omega_{t}(x)=\omega_{0}(x)-\displaystyle\int_{0}^{t}B_{s}(x;\omega)\
ds-\displaystyle\int_{0}^{t}\displaystyle\sum_{j}\alpha_{j}\nabla^{\perp}{\mathfrak{e}}_{j}(x)\cdot\nabla\omega_{t}(x)dW_{t}$
(12)

in which for simplicity of notation, we defined $B_{s}(x;\omega):={\bf
u}_{s}(x)\cdot\nabla\omega_{s}(x)$. Combining (12) with the Biot-Savart law
(10) we obtain

$\displaystyle{\bf u}_{t}(x)$ $\displaystyle={\bf
u}_{0}(x)-\int_{0}^{t}\int_{\mathbb{T}^{2}}K(x-y)B_{s}(y;\omega)dyds-\int_{0}^{t}\int_{\mathbb{T}^{2}}K(x-y)\xi(y)\cdot\nabla\omega_{s}(y)dy\circ
dW_{s}$ (13)

Now consider the “kinetic energy”

$e_{t}:=\frac{1}{2}\int_{\mathbb{T}^{2}}|{\bf u}_{t}|^{2}dx$ (14)

which is not conserved in the SALT case. Using Itô’s lemma, we obtain

$\displaystyle e_{t}-e_{0}$ $\displaystyle=-\int_{0}^{t}\langle{\bf
u}_{s},K\star(B_{s}-\frac{1}{2}\xi\cdot\nabla(\xi\cdot\nabla\omega_{s}))\rangle
ds$ (15) $\displaystyle\qquad-\int_{0}^{t}\langle{\bf
u}_{s},K\star(\xi\cdot\nabla\omega_{s})\rangle\ dW_{s}$

where $\langle.,.\rangle$ is the standard $L^{2}({\mathbb{T}^{2}})$ pairing.

For a stochastic process $X_{t}$ defined on a filtered probability space, its
_quadratic variation_ is defined by

$[X]_{t}:=\lim_{\max_{j}\Delta t_{j}\rightarrow
0}\sum_{i=1}^{n}|X_{t_{i}}-X_{t_{i-1}}|^{2},$ (16)

where $t_{0}=0<t_{1}<\dots<t_{n}=t$ is a partition of the interval $[0,t]$,
$\Delta t_{i}:=|t_{i}-t_{i-1}|$, and the convergence holds in probability (see
e.g. [8]). In our case, for the semimartingale $e_{t}$, we have

t $\displaystyle=\int_{0}^{t}\langle{\bf
u}_{s},K\star(\xi\cdot\nabla\omega_{s})\rangle^{2}\ ds.$ (17)

Substituting in the parametric form for $\xi$, we obtain the following
quadratic form

t
$\displaystyle=\sum_{i,j=1}^{\infty}\alpha_{i}\alpha_{j}\int_{0}^{t}\langle{\bf
u}_{s},K\star(\nabla^{\perp}{\mathfrak{e}}_{j}\cdot\nabla\omega_{s})\rangle\langle{\bf
u}_{s},K\star(\nabla^{\perp}{\mathfrak{e}}_{i}\cdot\nabla\omega_{s})\rangle\
ds.$ (18)

Due to global existence and uniqueness of solutions to (3), $[e]_{t}$ exists
globally $\mathbb{P}$-almost surely. Thus the right hand side of (18) can be
arbitrarily well approximated by its truncation for all $t$ i.e. for a given
$\epsilon>0$, there exists $M_{\epsilon}$ such that

$\left|[e]_{t}-\sum_{i,j=1}^{M_{\epsilon}}\alpha_{i}\alpha_{j}\int_{0}^{t}\langle{\bf
u}_{s},K\star(\nabla^{\perp}{\mathfrak{e}}_{j}\cdot\nabla\omega_{s})\rangle\langle{\bf
u}_{s},K\star(\nabla^{\perp}{\mathfrak{e}}_{i}\cdot\nabla\omega_{s})\rangle\
ds\right|<\epsilon.$ (19)

Additionally, from the computational perspective, for any fixed
$M_{\epsilon}$, the linear map

$\mathbf{A}_{ij}:=\int_{0}^{t}\langle{\bf
u}_{s},K\star(\nabla^{\perp}{\mathfrak{e}}_{j}\cdot\nabla\omega_{s})\rangle\langle{\bf
u}_{s},K\star(\nabla^{\perp}{\mathfrak{e}}_{i}\cdot\nabla\omega_{s})\rangle\
ds$ (20)

that defines the truncated quadratic form is symmetric and positive
definite111$[e]_{t}$ is strictly positive., and thus can be diagonalised by a
unitary linear map. Doing so, we obtain the following linear problem

$[e]_{t}=\sum_{j=1}^{M_{\epsilon}}\tilde{\alpha}_{j}^{2}\lambda_{j}+\epsilon^{\prime},$
(21)

where $\epsilon^{\prime}$ denotes the truncation error of (18), $\lambda_{j}$
are the eigenvalues of the associated linear map, and $\tilde{\alpha}_{j}$’s
are the original $\alpha$ values which get rescaled by the unitary matrix from
the diagonalisation.

We can estimate $[e]_{t}$ using the high frequency in time data $\omega^{*}$
and (16), assuming the discrete sum converges fast enough,

$[e]_{t}\approx\hat{[e]}_{t,N}:=\frac{1}{4}\sum_{i=1}^{N}(\int_{\mathbb{T}^{2}}|\nabla^{\perp}\Delta^{-1}\omega^{*}_{t_{i}}|^{2}-|\nabla^{\perp}\Delta^{-1}\omega^{*}_{t_{i-1}}|^{2}dx)^{2}.$
(22)

The estimate $\hat{[e]}_{t,N}$ could then be used in (21) to get an estimate
for the $\tilde{\alpha}$. One could then recover the original $\alpha$’s by
applying the unitary linear map that’s associated with the diagonalisation of
$\mathbf{A}_{ij}$.

######  Remark 2.

The calculations shown in this section could also be directly applied to the
vorticity equation (12) with little modification. In the numerics section of
this work, this is what we did. The linear system for estimation, however,
would would also depend on the space variable. Recall that the system (3)
conserves spatial integrals of $\omega_{t}$ (it is a Casimir of the system,
see [4]), the quadratic variation of the spatially integrated $\omega$ would
be trivial.

##  4 Robustness

######  Theorem 3.

Let $\omega^{1},\omega^{2}$ be two solutions of the 2D Euler equation (3) and
$\xi^{1},\xi^{2}$ the corresponding stochastic parameters for each of these
two solutions. More precisely, $(\omega^{\ell},\xi^{\ell})$ for $\ell=1,2$
solves

$d{\omega}_{t}^{\ell}+u_{t}^{\ell}\cdot\nabla{\omega}_{t}^{\ell}dt+\xi^{\ell}\cdot\nabla{\omega}_{t}^{\ell}dW_{t}=\frac{1}{2}\xi^{\ell}\cdot\nabla\left(\xi^{\ell}\cdot\nabla\omega^{\ell}\right).$
(23)

Then for any $p\geq 2$ there exist non-negative constants $C=C(p,T)$,
$C_{1,p},C_{2,p}$, such that

$\mathbb{E}\left[e^{-\gamma(T)}\displaystyle\sup_{t\in[0,T]}\|\omega_{t}^{1}-\omega_{t}^{2}\|_{2}^{p}\right]\leq
C\left(\|\omega_{0}^{1}-\omega_{0}^{2}\|_{2}^{p}+\|\xi^{1}-\xi^{2}\|_{2}^{p}\right)$
(24)

where

$\gamma(T):=C_{1,p}\displaystyle\int_{0}^{T}\|\omega_{s}^{1}\|_{k,2}^{p}ds+C_{2,p}T^{p}.$

###### Proof of Theorem 3..

Let
$\bar{\omega}:=\omega^{1}-\omega^{2},\bar{u}=u^{1}-u^{2},\bar{\xi}=\xi^{1}-\xi^{2}$.
Then $\bar{\omega}$ satisfies

$d{\bar{\omega}}_{t}+(\bar{u}_{t}\cdot\nabla\omega_{t}^{1}+u_{t}^{2}\cdot\nabla{\bar{\omega}_{t}})dt+\left(\xi^{1}\cdot\nabla\omega_{t}^{1}-\xi^{2}\cdot\nabla\omega_{t}^{2}\right)dW_{t}=\frac{1}{2}\left(\xi^{1}\cdot\nabla(\xi^{1}\cdot\nabla\omega_{t}^{1})-\xi^{2}\cdot\nabla(\xi^{2}\cdot\nabla\omega_{t}^{2})\right)dt.$

By the Itô formula:

$\displaystyle d\|\bar{\omega}_{t}\|_{2}^{2}=$
$\displaystyle-2\langle\bar{\omega}_{t},\xi^{1}\cdot\nabla\omega_{t}^{1}-\xi^{2}\cdot\nabla\omega_{t}^{2}\rangle
dW_{t}-2\langle\bar{\omega}_{t},\bar{u}_{t}\cdot\nabla\omega_{t}^{1}+u_{t}^{2}\cdot\nabla{\bar{\omega}_{t}}\rangle
dt$
$\displaystyle+\left(\langle\bar{\omega}_{t},\xi^{1}\cdot\nabla(\xi^{1}\cdot\nabla\omega_{t}^{1})-\xi^{2}\cdot\nabla(\xi^{2}\cdot\nabla\omega_{t}^{2})\rangle+\langle\xi^{1}\cdot\nabla\omega_{t}^{1}-\xi^{2}\cdot\nabla\omega_{t}^{2},\xi^{1}\cdot\nabla\omega_{t}^{1}-\xi^{2}\cdot\nabla\omega_{t}^{2}\rangle\right)dt.$

The difference of the nonlinear terms is analysed explicitly in [9] pp. 9:

$\langle\bar{\omega}_{t},\bar{u}_{t}\cdot\nabla\omega_{t}^{1}\rangle\leq\|\bar{\omega}_{t}\|_{2}\|\bar{u}_{t}\|_{4}\|\nabla\omega_{t}^{1}\|_{4}\leq
C\|\bar{\omega}_{t}\|_{2}^{2}\|\omega_{t}^{1}\|_{k,2}.$

We used here that $\|\nabla\omega_{t}^{1}\|_{4}\leq C\|\omega_{t}^{1}\|_{k,2}$
and $\|\bar{u}_{t}\|_{4}\leq C\|\bar{u}_{t}\|_{1,2}\leq
C\|\bar{\omega}_{t}\|_{2}$. Also, since $u^{2}$ is divergence-free,
$\langle\bar{\omega}_{t},u_{t}^{2}\cdot\nabla{\bar{\omega}_{t}}\rangle=-\frac{1}{2}\displaystyle\int_{\mathbb{T}^{2}}(\nabla\cdot
u_{t}^{2})(\bar{\omega}_{t})^{2}dx=0$. We estimate the difference terms which
include $\xi^{1}$ and $\xi^{2}$ in Lemma 5 below. Note here that the term
$\langle\bar{\omega}_{t},\xi^{2}\cdot\nabla\left(\xi^{2}\cdot\nabla\bar{\omega}_{t}\right)\rangle$
is negative. To estimate the stochastic term let

$D_{t}:=\displaystyle\int_{0}^{t}\langle\bar{\omega}_{s},\xi^{1}\cdot\nabla\omega_{s}^{1}-\xi^{2}\cdot\nabla\omega_{s}^{2}\rangle
dW_{s}$

and

$m_{t}:=\|\bar{\omega}_{t}\|_{2}^{2}\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
Z:=\|\bar{\xi}\|_{2}^{2}.$

By the Burkholder-Davis-Gundy inequality, for arbitrary $p\geq 2$ there exists
a constant $C_{p}$ such that222In this proof $C,C_{p}$ are generic constants
which may differ from line to line and from term to term.

$\mathbb{E}\left[\displaystyle\sup_{s\in[0,t]}|D_{s}|^{p}\right]\leq
C_{p}\mathbb{E}\left[[D]_{t}^{p/2}\right]$

where $[D]_{t}$ is the quadratic variation of the martingale $D_{t}$. Note
that $[D]_{t}$ can be controlled using the control for $Q$ in Lemma 5 so we
have

$\displaystyle\mathbb{E}\left[[D]_{t}^{p/2}\right]\leq
C_{p}\int_{0}^{t}\mathbb{E}\left[\sup_{r\in[0,s]}\left(m_{r}^{p}+\tilde{Z}^{p}\right)\right]ds$

where $\tilde{Z}:=C\|\omega_{t}^{1}\|_{k,2}^{2}Z$ with $k\geq 3$. We then
have, for $t\in[0,T]$:

$m_{t}\leq
m_{0}+\displaystyle\int_{0}^{t}\psi(s)ds-2D_{t}+\displaystyle\int_{0}^{t}\phi(s)m_{s}ds$

where

$\phi(t):=C\|\omega_{t}^{1}\|_{k,2}^{2}+\tilde{C}$

and

$\psi(t):=(C\|\omega_{t}^{1}\|_{k,2}^{2}+1)Z.$

Then

$\displaystyle
d\left(e^{-\displaystyle\int_{0}^{t}\phi(r)^{p}dr}m_{t}^{p}\right)$
$\displaystyle=e^{-\displaystyle\int_{0}^{t}\phi(r)^{p}dr}\left(dm_{t}^{p}-m_{t}^{p}\phi(t)^{p}dt\right)$
$\displaystyle\leq
e^{-\displaystyle\int_{0}^{t}\phi(r)^{p}dr}\left(\psi(t)^{p}dt-2(dD_{t})^{p}\right)$

and

$\displaystyle\mathbb{E}\left[e^{-\displaystyle\int_{0}^{t}\phi(r)^{p}dr}\sup_{s\in[0,t]}m_{s}^{p}\right]$
$\displaystyle\leq
C_{p}\left(m_{0}^{p}+\mathbb{E}\left[\displaystyle\int_{0}^{t}e^{-\displaystyle\int_{0}^{s}\phi(r)^{p}dr}\displaystyle\sup_{r\in[0,s]}\psi(r)^{p}ds\right]+2\mathbb{E}\left[e^{-\displaystyle\int_{0}^{t}\phi(r)^{p}dr}\displaystyle\sup_{s\in[0,t]}|D_{s}|^{p}\right]\right)$
$\displaystyle\leq
C_{p}\left(m_{0}^{p}+\mathbb{E}\left[\displaystyle\int_{0}^{t}e^{-\displaystyle\int_{0}^{s}\phi(r)^{p}dr}\displaystyle\sup_{r\in[0,s]}\psi(r)^{p}ds\right]\right)$
$\displaystyle\leq
C_{p}\left(m_{0}^{p}+\mathbb{E}\left[\displaystyle\int_{0}^{t}\displaystyle\sup_{r\in[0,s]}\psi(r)^{p}ds\right]\right)$

since $D_{t}$ is a martingale so its expectation is zero. Therefore

$\displaystyle\mathbb{E}\left[e^{-\gamma(t)}\sup_{s\in[0,t]}m_{s}^{p}\right]\leq
C_{p}\left(m_{0}^{p}+\mathbb{E}\left[\displaystyle\int_{0}^{t}\displaystyle\sup_{r\in[0,s]}\psi(r)^{p}ds\right]\right)$

where $\gamma(t):=\displaystyle\int_{0}^{t}\phi(s)^{p}ds$. This gives

$\mathbb{E}\left[e^{-\gamma(T)}\displaystyle\sup_{s\in[0,T]}\|\omega_{s}^{1}-\omega_{s}^{2}\|_{2}^{p}\right]\leq
C_{p,T}\left(\|\omega_{0}^{1}-\omega_{0}^{2}\|_{0}^{p}+\|\xi^{1}-\xi^{2}\|_{2}^{p}\right)$

for any $p\geq 2$, with $\gamma$ as above. We used here the fact that the 2D
Euler equation (3) has a unique global solution in
$\mathcal{W}^{k,2}(\mathbb{T}^{2})$ for $k\geq 2$, as proven in [9]. Moreover
$\bar{\xi}$ is deterministic and time-independent so

$\displaystyle\mathbb{E}\left[\int_{0}^{T}\psi(s)ds\right]$
$\displaystyle=\mathbb{E}\left[\int_{0}^{T}(C\|\omega_{s}^{1}\|_{k,2}^{2}+1)\|\bar{\xi}\|_{2}^{2}ds\right]$
$\displaystyle\leq\|\bar{\xi}\|_{2}^{2}\left(CT\mathbb{E}\left[\displaystyle\sup_{s\in[0,T]}\|\omega_{s}^{1}\|_{k,2}^{2}\right]+T\right)$
$\displaystyle\leq\tilde{C}_{T}\|\bar{\xi}\|_{2}^{2}.$

∎

######  Remark 4.

Note that we have shown above a result which is similar to an application of
the Growall Lemma 8 in 6. However, since we work with random variables, we
made this step explicitly, for clarity.

######  Lemma 5.

Let $(\omega_{t}^{1},\xi^{1})$ and $(\omega_{t}^{2},\xi^{2})$ be two solutions
of the 2D Euler equation with
$\bar{\omega}_{t}:=\omega_{t}^{1}-\omega_{t}^{2}$ and
$\bar{\xi}:=\xi^{1}-\xi^{2}.$ Then there exists constants $C$333$C$ differs
from line to line and from term to term depending on the Sobolev embedding we
use. such that the following estimates hold:

$Q:=|\langle\bar{\omega}_{t},\xi^{1}\cdot\nabla\omega_{t}^{1}-\xi^{2}\cdot\nabla\omega_{t}^{2}\rangle|\leq
C\|\bar{\omega}_{t}\|_{2}^{2}+C\|\omega_{t}^{1}\|_{k,2}^{2}\|\bar{\xi}\|_{2}^{2}.$
(25a)
$A:=\langle\xi^{1}\cdot\nabla\omega_{t}^{1}-\xi^{2}\cdot\nabla\omega_{t}^{2},\xi^{1}\cdot\nabla\omega_{t}^{1}-\xi^{2}\cdot\nabla\omega_{t}^{2}\rangle\leq
C\|\bar{\omega}_{t}\|_{2}^{2}+C\|\omega_{t}^{1}\|_{k,2}^{2}\|\bar{\xi}\|_{2}^{2}$
(25b) $|B|\leq
C\|\omega_{t}^{1}\|_{k,2}^{2}\|\bar{\omega}_{t}\|_{2}^{2}+\|\bar{\xi}\|_{2}^{2}$
(26)

where

$B:=\langle\omega_{t}^{1}-\omega_{t}^{2},\xi^{1}\cdot\nabla\left(\xi^{1}\cdot\nabla\omega_{t}^{1}\right)-\xi^{2}\cdot\nabla\left(\xi^{2}\cdot\nabla\omega_{t}^{2}\right)\rangle.$

and $k\geq 4$.

###### Proof.

For the difference terms which include $\xi^{1}$ and $\xi^{2}$ we use that

$\displaystyle\xi^{1}\cdot\nabla\omega_{t}^{1}-\xi^{2}\cdot\nabla\omega_{t}^{2}=\bar{\xi}\cdot\nabla\omega_{t}^{1}+\xi^{2}\cdot\nabla\bar{\omega}_{t}.$

We have

$\displaystyle Q$
$\displaystyle=|\langle\bar{\omega}_{t},\xi^{1}\cdot\nabla\omega_{t}^{1}-\xi^{2}\cdot\nabla\omega_{t}^{2}\rangle|$
$\displaystyle\leq|\langle\omega_{t}^{1}-\omega_{t}^{2},(\xi^{1}-\xi^{2})\cdot\nabla\omega_{t}^{1}\rangle|+|\langle\omega_{t}^{1}-\omega_{t}^{2},\xi^{2}\cdot\nabla(\omega_{t}^{1}-\omega_{t}^{2})\rangle|$
$\displaystyle\leq\frac{1}{2}\|\omega_{t}^{1}-\omega_{t}^{2}\|_{2}^{2}+\frac{1}{2}\|\nabla\omega_{t}^{1}\|_{\infty}^{2}\|\xi^{1}-\xi^{2}\|_{2}^{2}$
$\displaystyle\leq\frac{1}{2}\|\bar{\omega}_{t}\|_{2}^{2}+\frac{C}{2}\|\omega_{t}^{1}\|_{k,2}^{2}\|\bar{\xi}\|_{2}^{2}$

with $k\geq 3$, since the second scalar product is zero due to the fact that
$\div\xi^{2}=0.$ Also

$\displaystyle A$
$\displaystyle=\langle\xi^{1}\cdot\nabla\omega_{t}^{1}-\xi^{2}\cdot\nabla\omega_{t}^{2},\xi^{1}\cdot\nabla\omega_{t}^{1}-\xi^{2}\cdot\nabla\omega_{t}^{2}\rangle=\|\xi^{1}\cdot\nabla\omega_{t}^{1}-\xi^{2}\cdot\nabla\omega_{t}^{2}\|_{2}^{2}$
$\displaystyle\leq\|(\xi^{1}-\xi^{2})\cdot\nabla\omega_{t}^{1}\|_{2}^{2}+\|\xi^{2}\cdot\nabla(\omega_{t}^{1}-\omega_{t}^{2})\|_{2}^{2}$
$\displaystyle\leq\|\xi^{1}-\xi^{2}\|_{2}^{2}\|\nabla\omega_{t}^{1}\|_{\infty}^{2}+C\|\omega_{t}^{1}-\omega_{t}^{2}\|_{2}^{2}$
$\displaystyle\leq
C\|\omega_{t}^{1}\|_{k,2}^{2}\|\bar{\xi}\|_{2}^{2}+C\|\bar{\omega}_{t}\|_{2}^{2}$

where $k\geq 3$. For the higher order term we have

$\displaystyle B$
$\displaystyle=\langle\omega_{t}^{1}-\omega_{t}^{2},\xi^{1}\cdot\nabla\left(\xi^{1}\cdot\nabla\omega_{t}^{1}\right)-\xi^{2}\cdot\nabla\left(\xi^{2}\cdot\nabla\omega_{t}^{2}\right)\rangle$
$\displaystyle=\langle\omega_{t}^{1}-\omega_{t}^{2},(\xi^{1}-\xi^{2})\cdot\nabla(\xi^{1}\cdot\nabla\omega_{t}^{1})\rangle$
$\displaystyle+\langle\omega_{t}^{1}-\omega_{t}^{2},\xi^{2}\cdot\nabla\left((\xi^{1}-\xi^{2})\cdot\nabla\omega_{t}^{1}\right)\rangle$
$\displaystyle+\langle\omega_{t}^{1}-\omega_{t}^{2},\xi^{2}\cdot\nabla\left(\xi^{2}\cdot\nabla(\omega_{t}^{1}-\omega_{t}^{2})\right)\rangle$
$\displaystyle=:a+b+c.$

Note that $c$ is negative:

$\displaystyle\langle\omega_{t}^{1}-\omega_{t}^{2},\xi^{2}\cdot\nabla\left(\xi^{2}\cdot\nabla(\omega_{t}^{1}-\omega_{t}^{2})\right)\rangle$
$\displaystyle=-\langle\xi^{2}\cdot\nabla(\omega_{t}^{1}-\omega_{t}^{2}),\xi^{2}\cdot\nabla(\omega_{t}^{1}-\omega_{t}^{2})\rangle$
$\displaystyle=-\|\xi^{2}\cdot\nabla(\omega_{t}^{1}-\omega_{t}^{2})\|_{2}^{2}$
$\displaystyle\leq 0$

so $|B|\leq|a|+|b|$. We estimate $|a|$ as follows:

$\displaystyle|\langle\omega_{t}^{1}-\omega_{t}^{2},(\xi^{1}-\xi^{2})\cdot\nabla(\xi^{1}\cdot\nabla\omega_{t}^{1})\rangle|$
$\displaystyle\leq\frac{1}{2}\|\nabla(\xi^{1}\cdot\nabla\omega_{t}^{1})\|_{\infty}^{2}\|\omega_{t}^{1}-\omega_{t}^{2}\|_{2}^{2}+\frac{1}{2}\|\xi^{1}-\xi^{2}\|_{2}^{2}$
$\displaystyle\leq\frac{C}{2}\|\omega_{t}^{1}\|_{2,\infty}^{2}\|\omega_{t}^{1}-\omega_{t}^{2}\|_{2}^{2}+\frac{1}{2}\|\xi^{1}-\xi^{2}\|_{2}^{2}$
$\displaystyle\leq\frac{C}{2}\|\omega_{t}^{1}\|_{k,2}^{2}\|\bar{\omega}_{t}\|_{2}^{2}+\frac{1}{2}\|\bar{\xi}\|_{2}^{2}$

with $k\geq 4$. Likewise, we estimate $|b|$:

$\displaystyle|\langle\omega_{t}^{1}-\omega_{t}^{2},\xi^{2}\cdot\nabla\left((\xi^{1}-\xi^{2})\cdot\nabla\omega_{t}^{1}\right)\rangle|$
$\displaystyle=|\langle\xi^{2}\cdot\nabla(\omega_{t}^{1}-\omega_{t}^{2}),(\xi^{1}-\xi^{2})\cdot\nabla\omega_{t}^{1}\rangle|$
$\displaystyle\leq\frac{1}{2}\|\nabla\omega_{t}^{1}\|_{\infty}^{2}\|\xi^{2}\cdot\nabla(\omega_{t}^{1}-\omega_{t}^{2})\|_{2}^{2}+\frac{1}{2}\|\xi^{1}-\xi^{2}\|_{2}^{2}$
$\displaystyle\leq\frac{C}{2}\|\nabla\omega_{t}^{1}\|_{\infty}^{2}\|\omega_{t}^{1}-\omega_{t}^{2}\|_{2}^{2}+\frac{1}{2}\|\xi^{1}-\xi^{2}\|_{2}^{2}$
$\displaystyle\leq\frac{C}{2}\|\omega_{t}^{1}\|_{k,2}^{2}\|\bar{\omega}_{t}\|_{2}^{2}+\frac{1}{2}\|\bar{\xi}\|_{2}^{2}$

with $k\geq 3$. ∎

##  5 Numerical results

We implemented the main equation (3) with added forcing and damping, on a unit
square domain with doubly periodic boundary conditions,

$d{\omega}_{t}+{\bf
u}_{t}\cdot\nabla{\omega}_{t}dt+\xi\cdot\nabla{\omega}_{t}\circ
dW_{t}=(Q-r\omega_{t})dt$ (27)

where we chose $r=0.001$ and $Q(x)=0.01(\cos(8\pi y)+\sin(8\pi x))$. We
considered a $\xi$ whose parametric form with respect to the Fourier basis
consists of only one $\alpha$. The stream function of our chosen $\xi$ is
given by

$\zeta(x,y)=\alpha\left(\cos(k_{1}2\pi x)\cos(k_{2}2\pi y)-\sin(k_{1}2\pi
x)\sin(k_{2}2\pi y)\right).$ (28)

Note that

$\zeta=\frac{\alpha}{2}(e^{i2\pi k\cdot x}+e^{-i2\pi k\cdot x}),$ (29)

and

$\xi=i\alpha\pi(e^{i2\pi k\cdot x}-e^{-i2\pi k\cdot x})k^{\perp}.$ (30)

To discretise (27), we followed the methods documented in [4] – a mixed Finite
Element method was used for the spatial derivatives, and an explicit strong
stability preserving Runge-Kutta scheme of order 3 was used for the time
derivative. We added the forcing and damping terms to balance out the energy
dissipation caused by discretisation. This helped with maintaining the
statistical homogeneity of the numerical solution, once it has reached a spun-
up state from some set initial state. Our choice for the set initial state was

$\displaystyle\omega(0,x,y)$ $\displaystyle=\sin(8\pi x)\sin(8\pi
y)+0.4\cos(6\pi x)\cos(6\pi y)$ (31) $\displaystyle\quad+0.3\cos(10\pi
x)\cos(4\pi y)+0.02\sin(2\pi y)+0.02\sin(2\pi x).$

Spatially, we chose a grid of size $64\times 64$ cells. We first spun-up the
system until it reached a statistical equilibrium state. This statistical
equilibrium state was then set as the initial condition for our experiment.
Figure 1 shows a snapshot of the obtained initial condition. Over the spin-up
phase, we used a smaller $\alpha=0.000001$ value and $k^{\intercal}=(2,4)$.

The time horizon for the experiment data was chosen to be the unit interval,
i.e. we generated data $\omega^{*}(t_{i},x)$ for
$0=t_{0}<t_{1}<\dots<t_{N}=1$. See Figure 1 for snapshots of
$\omega^{*}(0,{\bf x})$ and $\omega^{*}(1,{\bf x})$. When generating the data,
we used the larger value of $\alpha=0.001$. This was so to avoid any possible
numerical issues444When $\alpha$ is small, $\alpha^{2}$ is close to machine
precision. when we attempted to recover $\alpha$ from data.

Figure 1: Snapshots of the numerical solution $\omega(t,x)$ to (27) at times
$t=0$ (left), and $t=1$ (right).

For these concrete experiment parameter choices, we chose to follow Remark 2,
and directly worked with the quadratic variation of $\omega$.

Assuming we know in-advance the exact Fourier wavenumber $k$, the linear
system for estimation reduces to

$\displaystyle\hat{[\omega]}_{t,N}({\bf
x}):=\displaystyle\sum_{i=1}^{N}(\omega_{t_{i}}({\bf x})-\omega_{t_{i-1}}({\bf
x}))^{2}\approx\alpha^{2}4\pi^{2}\ B(t,k,{\bf
x}){\mathfrak{e}}^{\prime}_{k}({\bf x})$ (32)

where

$B(t,k,{\bf x}):=\displaystyle\int_{0}^{t}(k^{\perp}\cdot\nabla\omega_{s}({\bf
x}))^{2}ds$ (33)

and

${\mathfrak{e}}^{\prime}_{k}(x,y):=\left(\cos(k_{1}2\pi x)\sin(k_{2}2\pi
y)+\sin(k_{1}2\pi x)\cos(k_{2}2\pi y)\right)^{2}.$ (34)

Thus our estimate for $\alpha$ is given by

$\hat{\alpha}_{N}^{2}=\frac{1}{4\pi^{2}}\frac{\int_{\mathbb{T}^{2}}\hat{[\omega]}_{t,N}({\bf
x})d{\bf x}}{\int_{\mathbb{T}^{2}}B(t,k,{\bf
x}){\mathfrak{e}}^{\prime}_{k}({\bf x})d{\bf x}}.$ (35)

######  Remark 6.

In (35), we applied spatial averaging before dividing to avoid possible
division by zero issues, and to help with stablising estimation errors.

######  Remark 7.

The assumption that we know $k$ in advance is of course too strong from the
applications viewpoint. The aim of this experiment is to test the strength of
the pathwise approach under the assumption of ”perfect knowledge”. If we
cannot accurately recover $\alpha$ in this case, then getting a good estimate
for $\alpha$ using the pathwise approach may be too difficult or impractical
in more realistic scenarios.

Figure 2: Shown on the left is a snapshot of the estimate
$\hat{[\omega]}_{t}$, which was computed using $N=200000$ data samples. Shown
on the right is a snapshot of the basis element
$B_{t}(k,x)\left(\cos(k_{1}2\pi x)\sin(k_{2}2\pi y)+\sin(k_{1}2\pi
x)\cos(k_{2}2\pi y)\right)^{2}$, which was approximated using the same $N$
number of data samples.

Figure 2 shows snapshots of $\hat{[\omega]}_{t,N}({\bf x})$ and $B(t,k,{\bf
x}){\mathfrak{e}}^{\prime}_{k}({\bf x})$. We applied (35) for different values
of $N$. In each case, the time integral that constitutes $B(t,k,{\bf x})$ was
approximated using a simple trapezoidal rule, for which the same $N$ number of
data snapshots were used. Figure 3 shows the results for the relative error

$\mathrm{err}_{N}=\frac{|\alpha-\hat{\alpha}_{N}|}{\alpha}$ (36)

for the different values of $N$. The results show that, in the worst case of
$N=2500$, the relative error was no greater than 0.89. This translates to an
absolute error of range of $0.001\pm 0.00089$. The best case was when all
$200000$ data samples were used to estimate $\alpha$, the relative error in
that case was $0.00135$. This suggests convergence and stabilisation of the
sum for $\hat{[\omega]}_{t}$.

For future work, we aim to test the pathwise approach for cases in which we do
not know the exact selection of basis elements for $\xi$. Further, we wish to
extend and test these ideas on coarse grained PDE data and compare with the
results that were obtained in [4] using previously developed calibration
methods.

Figure 3: The plot shows the relative error $\mathrm{err}_{N}$ defined in (36)
as a function of $N$. $\mathrm{err}_{N}$ was computed for
$N=2500,5000,10000,20000,40000,50000,66667,100000,200000.$

Acknowledgments
The authors would like to thank Prof Dan Crisan for the many helpful
suggestions and constructive ideas he shared with them during the preparation
of this work. They also thank Prof Darryl Holm, Prof Bertrand Chapron, Prof
Etienne Mémin, and the whole STUOD team for many inspiring discussions they
had during the STUOD meetings.

Funding
Both authors were partially supported by the European Research Council (ERC)
under the European Union’s Horizon 2020 Research and Innovation Programme
(ERC, Grant Agreement No 856408).

##  6 Appendix

######  Lemma 8.

(Gronwall lemma) Let $\beta:[0,T]\rightarrow[0,\infty)$ be a non-negative
absolutely continuous function that satisfies for a.e. $t$

$d\beta(t)\leq\phi(t)\beta(t)dt+\psi(t)dt$

where $\phi,\psi$ are non-negative integrable functions on $[0,T]$. Then

$\beta(t)\leq
e^{\displaystyle\int_{0}^{t}\phi(s)ds}\left(\beta(0)+\displaystyle\int_{0}^{t}\psi(s)ds\right)$

for all $t\in[0,T]$.

## References

  * [1] Brecht, R., Li, L., Bauer, W., Mémin, E. (2021), Rotating shallow water flow under location uncertainty with a structure-preserving discretization, Journal of Advances in Modeling Earth Systems, 13, e2021MS002492, https://doi.org/10.1029/2021MS002492
  * [2] Browning AP, Warne DJ,Burrage K, Baker RE, Simpson MJ, Identifiability analysis for stochastic differential equation models in systems biology. J. R. Soc. Interface 17: 20200652. (2020) https://doi.org/10.1098/rsif.2020.0652
  * [3] Chong, C., High-frequency analysis of parabolic stochastic PDES, The Annals of Statistics 2020, Vol. 48, No. 2, 1143–1167.
  * [4] Cotter, C., Crisan, D., Holm, D., Pan, W., Shevchenko, I., 2019. Numerically modelling stochastic Lie transport in fluid dynamics. Multiscale Model. Simul. 17, 192–232.
  * [5] Cotter, C., Crisan, D., Holm, D., Pan, W., Shevchenko, I., 2020a. A particle filter for stochastic advection by Lie transport (SALT): A case study for the damped and forced incompressible 2D Euler equation. SIAM/ASA J. Uncertain. Quantif. 8, 1446–1492.
  * [6] Glatt-Holtz, N., Ziane, M., et al.: Strong pathwise solutions of the stochastic navier-stokes system. Advances in Differ- ential Equations 14(5/6), 567–600 (2009).
  * [7] Holm, D., Variational principles for stochastic fluid dynamics, Proc. R.Soc.A 471:20140963, 2015.
  * [8] Karatzas, I., Shreve, S.E., Brownian Motion and Stochastic Calculus, Springer, 1998, ISBN 0-387-97655-8.
  * [9] Lang, O., Crisan, D., Well-posedness for a stochastic 2D Euler equation with transport noise, Stoch PDE: Anal Comp (Jan 2022), https://doi.org/10.1007/s40072-021-00233-7.
  * [10] Majda, A., Bertozzi, A., Vorticity and Incompressible Flow, Cambridge University Press, 2007, ISBN 0 521 63948 4 paperback.
  * [11] Marchioro, C., Pulvirenti, M., Mathematical Theory of Incompressible Nonviscous Fluids, Springer-Verlag, 1994, 978-0387940441.
  * [12] Mémin, E., Fluid dynamics under location uncertainty, Journal Geophysical & Astrophysical Fluid Dynamics, Volume 108 (2014).


11institutetext: NLP Group, Pompeu Fabra University, Roc Boronat, 138,
Barcelona, Spain
11email<EMAIL_ADDRESS>22institutetext: Catalan
Institute for Research and Advanced Studies (ICREA)

# Concept Extraction Using Pointer–Generator Networks

Alexander Shvets 11 0000-0002-8370-2109 Leo Wanner 1122 0000-0002-9446-3748

###### Abstract

Concept extraction is crucial for a number of downstream applications.
However, surprisingly enough, straightforward single token/nominal
chunk–concept alignment or dictionary lookup techniques such as DBpedia
Spotlight still prevail. We propose a generic open-domain OOV-oriented
extractive model that is based on distant supervision of a pointer–generator
network leveraging bidirectional LSTMs and a copy mechanism. The model has
been trained on a large annotated corpus compiled specifically for this task
from 250K Wikipedia pages, and tested on regular pages, where the pointers to
other pages are considered as ground truth concepts. The outcome of the
experiments shows that our model significantly outperforms standard techniques
and, when used on top of DBpedia Spotlight, further improves its performance.
The experiments furthermore show that the model can be readily ported to other
datasets on which it equally achieves a state-of-the-art performance.

###### Keywords:

Open-domain discourse texts Concept extraction Pointer-generator neural
network Distant supervision

##  1 Introduction

In knowledge discovery and representation, the notion of concept is most often
used to refer to sense, i.e., ‘abstract entity’ or ‘abstract object’ in the
Fregean dichotomy of sense vs. reference [9]. In Natural Language Processing
(NLP), the task of Concept Extraction (CE) deals with the identification of
the language side of the concept coin, i.e., Frege’s reference. Halliday [15]
offers a syntactic interpretation of reference. In his terminology, it is a
“classifying nominal group”. For instance, renewable energy or nuclear energy
are classifying nominal groups: they denote a class (or type) of energy,
while, e.g., cheap energy or affordable energy are not: they do not typify,
but rather qualify energy (and are thus “qualifying nominal groups”).

CE is crucial for a number of downstream applications, including, e.g.,
language understanding, ontology population, semantic search, and question
answering; it is also the key to entity linking [21]. In generic open domain
subject-neutral discourse across different (potentially unrelated) subjects,
indexing the longest possible nominal chunks and their head words located in
sequences of tokens between specified “break words” [32] and special
dictionary lookups such as DBpedia Spotlight [5] and WAT [26] are very common
techniques. They generally reach outstanding precision, but low recall due to
constant evolvement of the language vocabulary. Advanced deep learning models
that already dominate CE in specialized closed domain discourse on one or a
limited range of related subjects, e.g., biomedical discourse [13, 31], and
that are also standard in keyphrase extraction [24, 2] are an alternative.
However, such models need a tremendous amount of labeled data for training.

We present an operational CE model that utilizes pointer–generator networks
[28] and bidirectional long short-term memory (LSTM) units [11] to retrieve
concepts from general discourse textual material.111We adopt Halliday’s notion
of classifying nominal group as definition of a concept. Furthermore, since
for a generic, domain-independent concept extraction model we need a
sufficiently large training corpus that covers a vast variety of topics and no
such annotated corpora are available, we opt for distant supervision to create
a sufficiently large and diverse dataset. Distant supervision consists in
automatic labeling of potentially useful data by an easy-to-handle (not
necessarily accurate) algorithm to obtain an annotation which is likely to be
noisy but, at the same time, to contain enough information to train a robust
model [25]. Two labeling schemes are considered. Experiments carried out on a
dataset of 250K+ Wikipedia pages show that copies of our model trained
differently and joined in an ensemble significantly outperform standard
techniques and, when used on top of DBpedia Spotlight, further improve its
performance by nearly 10%.

##  2 Related work

In this section, we focus on the review of generic discourse CE; for a
comprehensive review of the large body of work on specialized discourse CE,
and, in particular, on biomedical CE; see, e.g., [14]. We also do not discuss
recent advances in keyphrase extraction [2] because their applicability to
generic concept extraction is limited due to specificity of the task.

The traditional CE techniques interpret any single and multiple token nominal
chunk as a concept [32] or do a dictionary lookup, as, e.g., DBpedia Spotlight
[5], which matches and links identified nominal chunks with DBpedia entries
(6.6M entities, 13 billion RDF
triples)222https://wiki.dbpedia.org/develop/datasets/dbpedia-version-2016-10,
based on the Apache OpenNLP333https://opennlp.apache.org/ models for phrase
chunking and named entity recognition (NER). Given the large coverage of
DBpedia, the performance of DBpedia Spotlight is rather competitive. However,
obviously, the presence of an entry cannot always be ensured. Consider, e.g.,
a paper title “Natural language understanding with Bloom embeddings,
convolutional neural networks and incremental parsing”, where DBpedia
Spotlight does not detect “Bloom embeddings” or “incremental parsing”, as
there are no such entries in DBpedia.

As DBpedia Spotlight, AIDA [33] relies on an RDF repository, YAGO2. WAT and
its predecessor TagMe [26] use a repository of possible spots made of wiki-
anchors, titles, and redirect pages. Both TagMe and WAT rely on statistical
attributes called link probability and commonness; WAT draws furthermore on a
set of statistics to prune a set of mentions using an SVM classifier. Wikifier
[3] focuses on relation extraction, relying on a NER, which uses gazetteers
extracted from Wikipedia and simple regular expressions to combine several
mentions into a single one. All of them are used for state-of-the-art entity
linking and (potentially nested) entity mention detection and typing [34, 16].
FRED [10] also focuses on extraction of relations between entities, with
frames [8] as the underlying theoretical constructs. Unlike Wikifier and FRED,
e.g., OLLIE [23] does not rely on any precompiled repository. It outperforms
its strong predecessors REVERB [7] in relation extraction by expanding the set
of possible relations and including contextual information from the sentence
from which the relations are extracted.

A number of works focus on the recognition of named entities, which are the
most prominent concept type. NERs work at a sentence level and aim at labeling
all occurred instances. Among them, Lample et al. [19] provide a state-of-the-
art NER model that avoids traditional heavy use of hand-crafted features and
domain-specific knowledge. The model is based on bidirectional LSTMs and
Conditional Random Fields (CRFs) that rely on two sources of information on
words: character-based word representations learned from an annotated corpus
and unsupervised word representations learned from unannotated corpora.
Another promising approach to NER is fine-tuning of a language representation
model such as, e.g., BERT [6]. The pre-trained BERT model can be fine-tuned
with just one additional output layer to create state-of-the-art models for a
wide range of tasks, including NER, without substantial task-specific
architecture modifications.

##  3 Description of the model

We implement a deep learning model and a large-scale annotation scheme for the
distant supervision to cope autonomously with dictionary-independent generic
CE and to a possible extent complement present lookup-based approaches to
increase their recall. In addition, we would like our model to perform
decently on pure NER tasks with a small gap to models specifically tuned for
the NER datasets. The model follows the well-established tendency in
information extraction adopted for NER and extractive summarization and
envisage CE as an attention-based sequence-to-sequence learning problem.

Figure 1: The neural architecture for concept extraction

###  3.1 Overview of the model

As a basis of our model, we use the pointer–generator network proposed in [28]
that aids creation of summaries with accurate reproduction of information. In
each generation step $t$, the pointer allows for copying words $w_{i}$ from
the source sequence to the target sequence using distribution of attention
layer $a^{t}$, while the generator samples tokens from the learned vocabulary
distribution ${P}_{vocab}$, conditioned by a context vector $h_{t}^{*}$
produced by the same attention layer which is built based on hidden states
$h_{i}$ of an encoder and states $s_{t}$ of a decoder (in each case, a
bidirectional LSTM [11]). In addition, coverage mechanism is applied to modify
$a^{t}$ using a coverage vector $c^{t}$ to avoid undesirable repetitions in
the output sequence. Specifically, to produce a word $w$, the above-mentioned
distributions are combined into a single final probability distribution being
weighted using the generation probability ${p}_{gen}$$\in$[0,1]:

$P(w)={p}_{gen}{P}_{vocab}(w)+(1-{p}_{gen}){\sum}_{i:w_{i}=w}{a_{i}^{t}}$ (1)

where ${P}_{vocab}(w)$ is the vocabulary distribution, which is zero if $w$ is
an out-of-vocabulary (OOV) word; $a^{t}$ is the attention distribution;
$w_{i}$ \- tokens of the input sequence; ${\sum}_{i:w_{i}=w}a_{i}^{t}$ is zero
if $w$ does not appear in the source sequence. According to [28], individual
vectors, distributions, and probability ${p}_{gen}$ are defined as follows:

$c^{t}={\sum}_{t^{\prime}=0}^{t-1}a^{t^{\prime}}$ (2)
$e_{i}^{t}=v^{T}tanh(W_{h}h_{i}+W_{s}s_{t}+w_{c}c_{i}^{t}+b_{attn})$ (3)
$a^{t}=softmax(e^{t})$ (4) $h_{t}^{*}={\sum}_{i}{a_{i}^{t}h_{i}}$ (5)
$P_{vocab}=softmax(V^{\prime}(V[s_{t},h_{t}^{*}]+b)+b^{\prime})$ (6)
$p_{gen}={\sigma}(w_{h*}^{T}h_{t}^{*}+w_{s}^{T}s_{t}+w_{x}^{T}x_{t}+b_{ptr})$
(7)

where $v$, $W_{h}$, $W_{s}$, $w_{c}$, $b_{attn}$, $V$, $V^{\prime}$, $b$,
$b^{\prime}$, $w_{h*}$, $w_{s}$, $w_{x}$, $b_{ptr}$ are learnable parameters,
$T$ stands for the transpose of a vector, $x_{t}$ is the decoder input, and
${\sigma}$ is the sigmoid function.

To adapt this basic model to the task of CE, we applied several modifications
to it (cf., Figure 1444We use a similar layout as in [28] for easier
comparison of our extension with the original model.): (i) following Gu et al.
[12], we use separate distributions for copy attention and general attention,
instead of one for both; (ii) experiments have shown that encoders and
decoders with several LSTM layers perform better than with a single layer,
such that we work with multiple layer LSTMs; how many is determined using a
development dataset; (iii) we adapt the forms of input and target sequences to
the specifics of the task of CE. The input is comprised of tokens and their
part-of-speech (PoS) tags (e.g., ‘The DT President NN is VBZ elected VBD by IN
a DT direct JJ vote NN’). The target sequence concatenates concepts in the
order they appear in the text and separates them by a token “*” especially
introduced to partition the output (e.g., ‘President * direct vote’).

This model is naturally applicable to the task of CE since it facilitates the
selection and transfer of subsequences of tokens (= concepts) from a given
source sequence of tokens (= text input) to the target sequence (= partitioned
sequence of concepts). The pointer mechanism implies the ability to cope with
OOV words, which is crucial for universal CE, while the generator implies the
ability to adjust vocabulary distribution for selecting the next word (which
might be a termination token “*”) based on a given context vector, which
allows us to implicitly take into account the domain specifics and linguistic
features that facilitate the task of CE. Furthermore, the updating of
vocabulary distribution adds the possibility to vanish or strengthen the copy
effect and thus learn to distinguish concepts with outer modifiers (such as,
e.g.,“hot air”, “[fully] crewed aircraft”, “reinforced group”) from multiword
concepts (such as, e.g., “hot air balloon”, “unmanned aerial vehicle”,
“reinforced concrete”).

###  3.2 Training and applying the model

For training, token sequences are taken from annotated sentences (see the
compilation of the annotated training dataset in Section 4.2 below) with a
sliding overlapping window of a fixed maximum length (see the Experiments
section), which is minimally expanded if needed in order not to deal with
incomplete concepts at the borders. The trained model is applied to unseen
sentences, which are also split into sequences of tokens with an overlapping
window of the same size, without any expansion. Finally, the corresponding
mentions in the plain text are determined since the output format does not
include offsets. In particular, following [16], we find all possible matches
for all detected concepts and then successively select non-nested concepts
from the beginning to the end of the sentence, giving priority to the longest,
in case of a multiple choice.

##  4 Datasets

In what follows, we describe the data and the procedure for their weak
annotation to create extensive training and test datasets.

###  4.1 Data

We take a snapshot of the WordNet synset-
typed555https://wordnet.princeton.edu/ Wikipedia [27], from which we use the
raw texts of the Wikipedia pages and text snippets of the links to other pages
as ground truth concepts regardless their type; cf., Figure 2666Wikipedia does
not contain self-links, therefore the concept “Grundy County” in a text from
the self-titled page is not a link.. These links often share the headings of
anchor pages, which are in most cases some real-world entities, cf., e.g.,
“Arthur Heurtley House”, “Price Tower”, etc. Sometimes, they are also lexical
variations of terms behind the link, as, e.g., the highlighted link in the
fragment “the two small coastal battleships General-Admiral Graf Apraxin and
Admiral Senyavin” leads to the page named “Coastal defence ship”.

Figure 2: Ground truth concept annotation

The manual annotation of multi-word expressions in 100 randomly selected
sentences with at least one multi-word link in each by a professional linguist
showed that at least 63% of such phrases are indeed concepts (cf., e.g.,
“punctuated equilibrium”, “chief of staff”, “2004 presidential election”). For
our work, we selected several data subsets from the collection of Wikipedia
pages: 250 K pages to be weakly, but densely annotated.777Henceforth, we refer
to the link snippet-based annotation of the pages as a sparse gold standard
annotation since it covers by far not all concepts encountered in a page. Our
distant supervision-based annotation is referred to as dense annotation since
it (supposedly) covers all concepts on a given page. As usual, distant
supervision-based annotation is also referred to as weak since it is an
automatic annotation. Out of these 250K pages, 220K are used for training and
30K for validation. In addition, we use 7K Wikipedia pages with the sparse
gold standard annotation as development set for choosing parameters of distant
supervision and selecting the best model among several models trained with
different parameters, and 7K pages with the sparse gold standard annotation as
test set.

###  4.2 Compilation of the training corpus

We automatically create a (noisy) training corpus using two various annotators
over a large unlabeled dataset: DBpedia Spotlight with the value of its
confidence coefficient that gains the highest recall and our own algorithm
that uses a number of rules and heuristics. Our labeling is based on the
sentence-wise analysis of statistical and linguistic features of sequences of
tokens. First, named entities and multiple token concepts and then single
token concepts are identified. The algorithm covers the following tasks:

1\. Application of a statistical NER model. A significant number of concepts
in Wikipedia are capitalized terms, which can be captured by statistical named
entity recognizers (NER); see the Related Work section above. Therefore, at
first, SpaCy’s state-of-the-art NER model [17] is applied with a successive
elimination of used tokens for further processing. The next steps are applied
then separately to fragments of texts located between the identified NEs.
2\. Selection of $n$-grams as fragments of NP chunks that can form part of
multiple token concepts. For this task, we formed the PoS-patterns based on
Penn Treebank
tagset888https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html,
which were inherited from the patterns for multiword expression detection
introduced in [4] and expanded here resulting in the following set: $P$ =
{N_N, J_N, V_N, N_J, J_J, V_J, N_of_N, N_of_DT_N, N_of_J, N_of_DT_J, N_of_V,
N_of_DT_V, CD_N, CD_J}, where N stands for “noun”, i.e.,
NN$\mid$NNS$\mid$NNP$\mid$NNPS, J stands for “adjective”, i.e.,
JJ$\mid$JJR$\mid$JJS, V - “verb” but limited to VBD$\mid$VBG$\mid$VN, CD -
“cardinal number”, DT - “determiner”, and “of” is an exact pronoun. Each
pattern matches an $n$-gram with two open-class lexical items and at most two
auxiliary tokens between them.
3\. Assessment of the distinctiveness of each selected $n$-gram. The
distinctiveness of the selected $n$-grams is assessed using word co-
occurrences from the Google Books Ngram Corpus [20]. Let us assume a given
$n$-gram $T_{1}A_{1}A_{2}T_{2}\in c_{k}$, where $T_{1}$ and $T_{2}$ are open
class lexical items and $A_{1}$ and $A_{2}$ are optional auxiliary tokens, and
$c_{k}$ is a set of all $n$-grams of a particular kind of pattern $p_{k}\in
P$. We use $T_{1}A_{1}A_{2}T_{2}$ as a point of a function that passes through
normalized document frequencies of a set of similar $n$-grams
$T_{1}A_{1}A_{2}T_{j}$, $j$ $\in$ {$i$ $|$ $T_{1}A_{1}A_{2}T_{i}\in c_{k}$}
arrayed in ascending order, to find a tangential angle at this point
$\alpha_{1}\in[0^{\circ};90^{\circ})$. Similarly,
$\alpha_{2}\in[0^{\circ};90^{\circ})$, is a tangential angle at the point
$T_{1}A_{1}A_{2}T_{2}$ on a curve of ordered frequencies of $n$-grams
$T_{h}A_{1}A_{2}T_{2}$, $h$ $\in$ {$i$ $|$ $T_{i}A_{1}A_{2}T_{2}\in c_{k}$}.
We leverage these angles to check how prominent an $n$-gram is, i.e., to what
extent it differs from its neighbors by overall usage. In case an $n$-gram is
located among equally prominent $n$-grams with a tangential angle close to
$0^{\circ}$, we do not consider it as a potential part of a concept since it
does not show a notable distinctiveness inherent in concepts, especially in
common idiosyncratic concepts. The thresholds $\alpha_{min_{1}}$ and
$\alpha_{min_{2}}$ ($\alpha_{min_{1}}$ $\geq$ $\alpha_{min_{2}}$) for
minimally allowed tangential angles such as $\max(\alpha_{1},\alpha_{2})$
$\geq$ $\alpha_{min_{1}}$, $\min(\alpha_{1},\alpha_{2})$ $\geq$
$\alpha_{min_{2}}$ are predermined in development experiments. We calculate
tangential angles through central difference approximation with a coarse-
grained grid:

$\alpha=\arctan(\frac{f(x+h)-f(x-h)}{2h})\cdot\frac{180}{\pi}$ (8)

where $h$ was chosen large enough ($h=50$ in general, and it is maximum
possible on the borders) for smoothing the curve to eliminate numerous abrupt
changes in document frequency with relatively low amplitude. Thus, the
approximation is intentionally carried out less accurately to result in such
values that in practice form a curve with longer monotonous sections; (cf.,
Figure 3) for an example of assessing the prominence of an $n$-gram
“prestressed concrete”, i.e., in the above notation, $T_{1}$ equals
“prestressed_ADJ”, $A_{1}$ and $A_{2}$ are omitted, and $T_{2}$ equals
“concrete_NOUN”.

Figure 3: Relation between document frequency and coarse-grained tangential angle approximation Table 1: Tangential angles of concept candidates Candidate | Angle | Wiki-term
---|---|---
reinforced_ADJ concrete_NOUN | 89.77 | YES
mixed_ADJ concrete_NOUN | 89.07 | NO
prestressed_ADJ concrete_NOUN | 88.40 | YES
pre-cast_ADJ concrete_NOUN | 83.66 | YES
first_ADJ concrete_NOUN | 33.12 | NO
original_ADJ concrete_NOUN | 16.63 | NO
massive_ADJ concrete_NOUN | 9.85 | NO
resistant_ADJ concrete_NOUN | 8.08 | NO
special_ADJ concrete_NOUN | 5.66 | NO
polymer_ADJ concrete_NOUN | 4.03 | YES
tall-wall_ADJ concrete_NOUN | 1.90 | NO
large_ADJ concrete_NOUN | 1.75 | NO
open_ADJ concrete_NOUN | 0.75 | NO
… | … | …
unusual_ADJ concrete_NOUN | OOV | NO
raised_ADJ concrete_NOUN | OOV | NO
… | … | …

Table 1 illustrates how the approximations of tangential angles differentiate
classifying nominal groups from qualifying nominal groups. The most of the
candidates with a large tangential angle have a separate article in Wikipedia
(i.e., they are likely to be concepts) while candidates with a small
tangential angle or without an entry in Google Books (OOV) do not have a
Wikipedia article in general. This shows that the chosen criterion for
differentiating the concepts is suitable for weak annotation within distant
supervision.

Grid search was applied to find the best combination of parameters
$\alpha_{min_{1}}$ and $\alpha_{min_{2}}$ from the three possible tangential
angles corresponding to the different levels of the distinctiveness of a
concept: 85∘, 60∘, and 0∘. As a result, $\alpha_{min_{1}}=60^{\circ}$ and
$\alpha_{min_{2}}=0^{\circ}$ gave the best scores on the development set and
were used for annotation of the training set.
4\. Combination of intersected highly distinctive parts as concepts. We
combine those distinctive $n$-grams that share common tokens and iteratively
drop the last token in each group if it is not a noun, in order to end up with
complete NP candidate concepts (e.g., “value of the played card” is a
potential concept corresponding to the patterns {N_of_DT_V; V_N}). Some
single-word concepts already might appear at this point.
5\. Recovery of missed single-word concepts. To enrich the set of candidate
concepts, we consider all unused nouns and numbers in a text as single-word
concept candidates.
The obtained training corpus contains moderate amount of noise: the proposed
annotation algorithm outperformes some baselines and might be used for CE by
itself (cf. setup (A) in Tables 2 and 3 with results of evaluation in the
following section).

##  5 Experiments

###  5.1 Setup of the experiments

For our experiments, we use the realization of See et al. [28]’s
pointer–generator model in the OpenNMT toolkit [18], which allows for the
adaptation of the model to the task of CE along the lines described in Section
3.1 above. Instead of the attention mechanism used in [28], we use the default
OpenNMT attention [22] since it showed to perform better. The model has
512-dimensional hidden states and 256-dimensional word embeddings shared
between encoder and decoder. We use a vocabulary of 50k words as we rely
mostly on a copying mechanism which uses dynamic vocabulary made up of words
from the current source sequence. We train using the Stochastic Gradient
Descent on a single GeForce GTX 1080 Ti GPU with a batch size of 64. We
trained the CE-adapted pointer–generator networks of two and three bi-LSTM
layers with 20K and 100K training steps on the two training datasets (obtained
using Google Books and DBpedia Spotlight, respectively; see above). Validation
and saving of checkpoint models was performed at each one-tenth of the number
of training steps.

In order to compare our extended pointer–generator model with state-of-the-art
techniques, several efficient entity extraction algorithms were chosen as
baselines: OLLIE [23], AIDA [33], AutoPhrase+ [29], DBpedia Spotlight [5], WAT
[26],999FRED [10] was not used as baseline as it is not scalable enough for
the task: its REST service has a strong limitation on a number of possible
requests per day, and it fails on processing long sentences (approximately
more than 40 tokens). and several state-of-the-art NER models, namely SpaCy
NER [17], FLAIR NER [1] and two deep learning-based NER models [19,
6]101010https://github.com/glample/tagger111111https://github.com/kyzhouhzau/BERT-
NER. AutoPhrase+ was used in combination with the StanfordCoreNLP PoS-tagger
(as it was reported to show better performance with PoS-tags) and trained
separately on its default DBLP dataset and on the above-mentioned raw
Wikipedia texts our training dataset is composed of. Its output was slightly
modified by removing auxiliary tokens from the beginning and the end of the
phrase to make it more competitive with the rest of the algorithms. OLLIE’s
and SpaCy’s outcomes were also modified the same way, which improved their
performance. DBpedia Spotlight was applied with two different values of
confidence coefficient: 0.5 (default value) and 0.1, which increases the
recall.

The performance is measured in terms of precision, recall, and $F_{1}$-score,
aiming at high recall, first of all. Since positive ground truth examples are
sparse, and there are no negative examples, we treated only the detected
concepts that partially overlapped the ground truth concepts as false
positives. Concepts that have the same spans as the ground truth concepts are
counted as true positives, and missed ground truth concepts as false
negatives. This perfectly meets our goal to detect the exact match. It also
allows us to penalize brute force high-recall algorithms that produce a large
number of nested concepts, which are of limited use in real-world
applications.

Table 2 shows the reached performance on the domain-specific datasets, and
Table 3 on the open domain set. The sign “*” stands for modifications made on
cutting some first and last words of detected concepts in order to present
them as “canonic” noun phrases, and “**” stands for removing nested concepts
when this procedure gave better scores.

Table 2: Results on the domain-specific datasets

| “Architecture” | “Terrorist groups”
---|---|---
Setup | Model | $P$ | $R$ | $F_{1}$ | $P$ | $R$ | $F_{1}$
| FLAIR (Akbik et al., 2019) | 0.79 | 0.74 | 0.76 | 0.77 | 0.66 | 0.71
| BERT NER (Delvin et al., 2019) | 0.78 | 0.74 | 0.76 | 0.78 | 0.67 | 0.72
|  AutoPhrase+${}_{DBLP}^{**}$ (Shang et al., 2018) | 0.38 | 0.44 | 0.41 | 0.31 | 0.34 | 0.33
|  AutoPhrase+${}_{WIKI}^{**}$ (Shang et al., 2018) | 0.42 | 0.52 | 0.46 | 0.36 | 0.45 | 0.40
| SpaCy NER (Honnibal and Montani, 2017) | 0.59 | 0.51 | 0.55 | 0.5 | 0.41 | 0.45
|  SpaCy NER∗ (Honnibal and Montani, 2017) | 0.71 | 0.61 | 0.66 | 0.59 | 0.49 | 0.54
| NER Tagger (Lample et al., 2016) | 0.78 | 0.71 | 0.75 | 0.76 | 0.65 | 0.7
|  WAT∗∗ (Piccinno and Ferragina, 2014) | 0.66 | 0.53 | 0.59 | 0.64 | 0.5 | 0.56
|  Spotlight0.5 (Daiber et al., 2013) | 0.85 | 0.74 | 0.79 | 0.8 | 0.7 | 0.75
|  Spotlight0.1 (Daiber et al., 2013) | 0.7 | 0.79 | 0.74 | 0.65 | 0.77 | 0.7
|  OLLIE∗ (Schmitz et al., 2012) | 0.46 | 0.2 | 0.28 | 0.41 | 0.22 | 0.28
| AIDA (Yosef et al., 2011) | 0.76 | 0.57 | 0.65 | 0.74 | 0.54 | 0.62
(A) |  DSA(60,0) | 0.63 | 0.74 | 0.68 | 0.5 | 0.64 | 0.56
(B) | ${PG}_{(3L,80K)}(DSA_{DICT})$ | 0.67 | 0.77 | 0.72 | 0.61 | 0.73 | 0.66
(C) | ${PG}_{(2L,18K)}(DSA_{(60,0)})$ | 0.7 | 0.8 | 0.75 | 0.59 | 0.72 | 0.65
(D) | (B) + (C) | 0.75 | 0.83 | 0.79 | 0.66 | 0.77 | 0.71
(E) |  (B) + (C) + Spotlight0.1 | 0.78 | 0.85 | 0.81 | 0.7 | 0.8 | 0.75
(F) |  (B) + (C) + Spotlight0.5 | 0.78 | 0.85 | 0.81 | 0.7 | 0.8 | 0.75
(G) |  (C) + Spotlight0.5 | 0.79 | 0.86 | 0.82 | 0.69 | 0.79 | 0.74

Table 3 displays the scores for two different experiment runs. In the first,
only concepts with an assigned WordNet type label in our typed Wikipedia
dataset (in their majority, named entities; cf. [27] for details of the
typification) were considered as positive examples (from about 276K nouns in
the test set, only 83K nouns, i.e., about 30%, were part of ground truth
concepts); in the second, all text snippets of the links were taken as ground
truth concepts (from about 390K nouns in the test set, 141K nouns, i.e., about
36%, were part of ground truth concepts). Setups A – H display the performance
of different variants of our model. ‘$DSA$’ stands for initial distant
supervision annotation obtained using DBpedia Spotlight, i.e., a dictionary
lookup, (‘$DSA_{DICT}$’), and with the proposed token-cooccurrence frequency-
based method (‘$DSA_{(60,0)}$’) (cf. Step 3 of the compilation of the training
corpus), where the values in parentheses correspond to $\alpha_{min_{1}}$ and
$\alpha_{min_{2}}$, which gave the best scores on the development set.
${PG}_{(2L,18K)}$ and ${PG}_{(3L,80K)}$ stand for pointer–generator networks
with parameters shown in parentheses chosen using the development set (2
layers, $18K$/$20K$ training steps and 3 layers, $80K$/$100K$ training steps
correspondingly).

Table 3: Results on a large-scale open-domain dataset

|  Only WordNet-typed concepts |  All ground truth concepts
---|---|---
Setup | Model | $P$ | $R$ | $F_{1}$ | $P$ | $R$ | $F_{1}$
| FLAIR (Akbik et al., 2019) | 0.8 | 0.74 | 0.77 | 0.79 | 0.59 | 0.67
|  AutoPhrase+${}_{DBLP}^{**}$ (Shang et al., 2018) | 0.42 | 0.45 | 0.43 | 0.4 | 0.43 | 0.41
|  AutoPhrase+${}_{WIKI}^{**}$ (Shang et al., 2018) | 0.46 | 0.52 | 0.49 | 0.43 | 0.49 | 0.46
| NER Tagger (Lample et al., 2016) | 0.78 | 0.72 | 0.75 | 0.77 | 0.58 | 0.66
|  WAT∗∗ (Piccinno and Ferragina, 2014) | 0.72 | 0.55 | 0.62 | 0.68 | 0.42 | 0.52
|  Spotlight0.1 (Daiber et al., 2013) | 0.73 | 0.76 | 0.75 | 0.69 | 0.73 | 0.71
|  OLLIE∗ (Schmitz et al., 2012) | 0.45 | 0.19 | 0.27 | 0.44 | 0.18 | 0.26
| AIDA (Yosef et al., 2011) | 0.8 | 0.6 | 0.68 | 0.77 | 0.45 | 0.57
(A) |  DSA(60,0) | 0.68 | 0.75 | 0.71 | 0.65 | 0.72 | 0.68
(B) | ${PG}_{(3L,80K)}(DSA_{DICT})$ | 0.71 | 0.74 | 0.73 | 0.68 | 0.72 | 0.7
(C) | ${PG}_{(2L,18K)}(DSA_{(60,0)})$ | 0.71 | 0.81 | 0.76 | 0.68 | 0.76 | 0.72
(D) | (B) + (C) | 0.76 | 0.84 | 0.8 | 0.72 | 0.8 | 0.76
(H) |  (C) + Spotlight0.1 | 0.78 | 0.85 | 0.81 | 0.75 | 0.81 | 0.78

To compare the performance of our model with state-of-the-art NER, we applied
it to two common public datasets for NER (CoNLL-2003 and GENIA). Table 4 shows
the results on the CoNLL-2003 dataset for two variants of our model (Setups B
and C) trained on our large training set, without any further NER adaptation,
as well as for their updated versions (Setups I, J, and K), which were fine-
tuned with the training set of the shared task $CoNLL_{T}$, contrasted with
the results of the two genuine state-of-the-art NE recognizers [19] and [6]
and DBpedia Spotlight. It should be noted that NER is a concept extraction
subtask which aims at detecting less generic concepts. Consider the following
statistics for the clarity: from about 69K nouns in the CoNLL-2003 training
set, only 31K nouns are part of NEs (e.g., S&P, BAYERISCHE VEREINSBANK, London
Newsroom, Lloyds Shipping Intelligence Service), while the remaining 38K nouns
(as in “air force”, “deposit rates”, “blue collar workers”) are not part of
NEs; as for GENIA, from about 132K nouns, only 93K form NEs (e.g., “tumor
necrosis factor”, “terminal differentiation”, “isolated polyclonal B
lymphocytes”), while the remaining 39K do not (as in “colonies”,
“interpretation”, “notion”, “circular dichroism”, “differential
accumulation”).

Table 4: Results on the CoNLL-2003 datasets

| CoNLL-2003 (test-a) | CoNLL-2003 (test-b)
---|---|---
Setup | Model | $P$ | $R$ | $F_{1}$ | $P$ | $R$ | $F_{1}$
| BERT NER (Delvin et al., 2019) | 0.95 | 0.96 | 0.95 | 0.94 | 0.94 | 0.94
| NER Tagger (Lample et al., 2016) | 0.97 | 0.97 | 0.97 | 0.97 | 0.96 | 0.96
|  Spotlight0.5 (Daiber et al., 2011) | 0.9 | 0.63 | 0.74 | 0.9 | 0.65 | 0.75
|  Spotlight0.1 (Daiber et al., 2011) | 0.77 | 0.77 | 0.77 | 0.76 | 0.77 | 0.77
(B) | ${PG}_{(3L,80K)}(DSA_{DICT})$ | 0.81 | 0.78 | 0.8 | 0.81 | 0.79 | 0.8
(C) | ${PG}_{(2L,18K)}(DSA_{(60,0)})$ | 0.82 | 0.82 | 0.82 | 0.79 | 0.81 | 0.8
(I) | $FineTune((B),CoNLL_{T})$ | 0.95 | 0.92 | 0.93 | 0.95 | 0.92 | 0.94
(J) | $FineTune((C),CoNLL_{T})$ | 0.94 | 0.91 | 0.93 | 0.96 | 0.92 | 0.94
(K) | (I) + (J) | 0.94 | 0.93 | 0.93 | 0.95 | 0.93 | 0.94

Table 5 shows the results of our models fine-tuned with GENIA along with the
results of concept identification by the recently published model
[30],121212https://github.com/ufal/acl2019_nested_ner which provides the most
promising scores on different GENIA tasks.

Table 5: Results on the GENIA dataset

| GENIA
---|---
Setup | Model | $P$ | $R$ | $F_{1}$
| seq2seq (Straková et al., 2019) | 0.86 | 0.79 | 0.82
(L) | $FineTune((B),GENIA_{T})$ | 0.85 | 0.8 | 0.82
(M) | $FineTune((C),GENIA_{T})$ | 0.84 | 0.77 | 0.81
(N) | (L) + (M) | 0.85 | 0.8 | 0.83

###  5.2 Discussion

Tables 2 and 3 show that a combination of the different variants of the
proposed pointer-generator model, which do not rely on external dictionaries
after being trained (cf. Setup D), outperforms in terms of recall and
$F_{1}$-score nearly all other models, including the dictionary lookup-based
DBpedia Spotlight, which is a hard to beat as it was applied to “known” data.
However, a combination of the pointer–generator model with DBpedia Spotlight
is even better; it outperforms DBpedia Spotlight by 10%. In other words, a
deep model combined with a DBpedia-lookup is the best solution for generic CE.
This applies to both runs displayed in Table 3, while it is to be noted that
all tested models show a lower performance in the discovery of non-named
entities. In particular, the NER models expectedly suffer a dramatic drop in
recall. As far as precision is concerned, DBpedia Spotlight on its own is
considerably better than any other proposal on the two small domain-specific
test sets, while AIDA is best on the open domain test set. This is to be
expected for dictionary lookup-based strategies. Also, as to be expected,
DBpedia Spotlight, applied with its confidence coefficient equal to 0.1,
showed significantly better recall than with the default value of 0.5,
although $F_{1}$-score was lower. The experiment on the CoNLL-2003 dataset
shows that the proposed model for generic CE performs well even without any
special adjustment ($F_{1}=0.8$ – $0.82$). It can be further fine-tuned to the
specific dataset resulting in scores comparable to state-of-the-art, even if
not designed specifically for the NER task ($F_{1}=0.93$ – $0.94$), while its
overall CE performance is better than of the targeted NER models (compare,
e.g., (B)+(C) with Lample et al. (2016)’s NER in Tables 2 and 3.

##  6 Conclusions

We presented an adaptation of the pointer–generator network model [28] to
generic open-domain concept extraction. Due to its capacity to cope with OOV
concept labels, it outperforms dictionary lookup-based CE such as DBpedia
Spotlight or AIDA in terms of recall and $F_{1}$-score. It also shows an
advantage over deep models that focus on NER only since it also covers non-
named concept categories. However, a combination of the pointer–generator
model with DBpedia Spotlight seems to be the best solution since it takes
advantage of both the neural model and the dictionary lookup. In order to
facilitate a solid evaluation of the proposed model and compare it to a series
of baselines, we utilized Wikipedia pages with text snippet links as a
sparsely concept-annotated dataset. To ensure that our model is capable of
extracting all generic concepts instead of detecting only texts of the page
links, we ignored this sparse annotation during training. Instead, we compiled
a large densely concept-annotated dataset for leveraging it within the distant
supervision using the algorithm described above. To the best of our knowledge,
no such dataset was available so far. In the future, we plan to address the
problem of multilingual concept extraction, using pre-trained multi-lingual
embeddings and compiling another large dataset that contains a higher
percentage of non-named entity concepts.

The code for running our pretrained models is available in the following
GitHub repository: https://github.com/TalnUPF/ConceptExtraction/.

##  7 Acknowledgments

The work presented in this paper has been supported by the European Commission
within its H2020 Research Programme under the grant numbers 700024, 700475,
779962, 786731, 825079, and 870930.

## References

  * [1] Akbik, A., Bergmann, T., Blythe, D., Rasul, K., Schweter, S., Vollgraf, R.: FLAIR: An easy-to-use framework for state-of-the-art NLP. In: Proc. NAACL (2019)
  * [2] Al-Zaidy, R., Caragea, C., Giles, C.L.: Bi-lstm-crf sequence labeling for keyphrase extraction from scholarly documents. In: Proc. of WWW (2019)
  * [3] Cheng, X., Roth, D.: Relational inference for wikification. In: Proc. of the EMNLP. pp. 1787–1796 (2013)
  * [4] Cordeiro, S., Ramisch, C., Villavicencio, A.: Ufrgs&lif at semeval-2016 task 10: rule-based mwe identification and predominant-supersense tagging. In: Proc. of SemEval-2016. pp. 910–917 (2016)
  * [5] Daiber, J., Jakob, M., Hokamp, C., Mendes, P.: Improving efficiency and accuracy in multilingual entity extraction. In: Proc. of the 9th International Conference on Semantic Systems (I-Semantics) (2013)
  * [6] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proc. of the NAACL-HLT. pp. 4171–4186 (2019)
  * [7] Fader, A., Soderland, S., Etzioni, O.: Identifying relations for open information extraction. In: Proc. of the EMNLP. pp. 1535–1545 (2011)
  * [8] Fillmore, C., Baker, C.: Frame semantics for text understanding. In: Proc. of the JWordNet and Other Lexical Resources Workshop at NAACL (2001)
  * [9] Frege, G.: Ueber Sinn und Bedeutung. Zeitschrift fuer Philosophie und philosophische Kritik 100, 25–50 (1892)
  * [10] Gangemi, A., Presutti, V., Reforgiato Recupero, D., Nuzzolese, A., Draicchio, F., Mongiovì, M.: Semantic web machine reading with fred. Semantic Web 8(6), 873–893 (2017)
  * [11] Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural networks 18(5-6), 602–610 (2005)
  * [12] Gu, J., Lu, Z., Li, H., Li, V.O.: Incorporating copying mechanism in sequence-to-sequence learning. In: Proc. of the ACL. pp. 1631–1640 (2016)
  * [13] Habibi, M., Weber, L., Neves, M., Wiegandt, D.L., Leser, U.: Deep learning with word embeddings improves biomedical named entity recognition. Bioinformatics 33(14), i37–i48 (2017)
  * [14] Hailu, N.G.: Investigation of Traditional and Deep Neural Sequence Models for Biomedical Concept Recognition. Ph.D. thesis, University of Colorado (2019)
  * [15] Halliday, M.: Halliday’s Introduction to Functional Grammar. Routledge, London & New York (2013)
  * [16] Hasibi, F., Balog, K., Bratsberg, S.: Entity linking in queries: Tasks and evaluation. In: Proc. International Conference on The Theory of Information Retrieval. pp. 171–180. ACM (2015)
  * [17] Honnibal, M., Montani, I.: spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. https://spacy.io/ (2017)
  * [18] Klein, G., Kim, Y., Deng, Y., Nguyen, V., Senellart, J., Rush, A.: Opennmt: Neural machine translation toolkit. In: Proc. of the 13th Conference of the AMTA. vol. 1, pp. 177–184 (2018)
  * [19] Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., Dyer, C.: Neural architectures for named entity recognition. In: Proc. NAACL-HLT (2016)
  * [20] Lin, Y., Michel, J.B., Aiden Lieberman, E., Orwant, J., Brockman, W., Petrov, S.: Syntactic annotations for the Google books NGram corpus. In: Proc. of the ACL 2012 System Demonstrations. pp. 169–174 (Jul 2012)
  * [21] Logeswaran, L., Chang, M.W., Lee, K., Toutanova, K., Devlin, J., Lee, H.: Zero-shot entity linking by reading entity descriptions. In: Proc. of the ACL. pp. 3449–3460 (Jul 2019)
  * [22] Luong, T., Pham, H., Manning, C.: Effective approaches to attention-based neural machine translation. In: Proc. of the EMNLP. pp. 1412–1421 (2015)
  * [23] Mausam, Schmitz, M., Soderland, S., Bart, R., Etzioni, O.: Open language learning for information extraction. In: Proc. of the 2012 Joint EMNLP and CoNLL Conferences. pp. 523–534 (2012)
  * [24] Meng, R., Zhao, S., Han, S., He, D., Brusilovsky, P., Chi, Y.: Deep keyphrase generation. In: Proc. ACL. pp. 582–592 (2017)
  * [25] Mintz, M., Bills, S., Snow, R., Jurafsky, D.: Distant supervision for relation extraction without labeled data. In: Proc. of the ACL. pp. 1003–1011 (2009)
  * [26] Piccinno, F., Ferragina, P.: From tagme to wat: A new entity annotator. In: Proc. of the First International Workshop on Entity Recognition and Disambiguation. pp. 55–62. ERD ’14, ACM, New York, NY, USA (2014)
  * [27] Schenkel, R., Suchanek, F., Kasneci, G.: Yawn: A semantically annotated wikipedia xml corpus. Datenbanksysteme in Business, Technologie und Web, –12. (2007)
  * [28] See, A., Liu, P.J., Manning, C.D.: Get to the point: Summarization with pointer-generator networks. In: Proc. of the ACL. pp. 1073–1083 (2017)
  * [29] Shang, J., Liu, J., Jiang, M., Ren, X., Voss, C., Han, J.: Automated phrase mining from massive text corpora. IEEE Transactions on Knowledge and Data Engineering 30(10), 1825–1837 (2018)
  * [30] Straková, J., Straka, M., Hajic, J.: Neural architectures for nested ner through linearization. In: Proceedings of the ACL. pp. 5326–5331 (2019)
  * [31] Tulkens, S., Šuster, S., Daelemans, W.: Unsupervised concept extraction from clinical text through semantic composition. Journal of Biomedical informatics 91, 103–120 (2019)
  * [32] Woods, W.A.: Conceptual indexing: A better way to organize knowledge. Technical Report SMLI, TR97-61, Sun Microsystems Laboratories (1997)
  * [33] Yosef, M., Hoffart, J., Bordino, I., Spaniol, M., Weikum, G.: Aida: An online tool for accurate disambiguation of named entities in text and tables. Proc. of the VLDB Endowment 4(12), 1450–1453 (2011)
  * [34] Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., Liu, Q.: ERNIE: Enhanced language representation with informative entities. In: Proc. ACL. pp. 1441–1451 (2019)