# Error-in-variables modelling for operator learning

Ravi G. Patel1∗, Indu Manickam1, Myoungkyu Lee2, and Mamikon Gulian1
1Sandia National Laboratories
2University of Alabama
<EMAIL_ADDRESS>
###### Abstract

Deep operator learning has emerged as a promising tool for reduced-order
modelling and PDE model discovery. Leveraging the expressive power of deep
neural networks, especially in high dimensions, such methods learn the mapping
between functional state variables. While proposed methods have assumed noise
only in the dependent variables, experimental and numerical data for operator
learning typically exhibit noise in the independent variables as well, since
both variables represent signals that are subject to measurement error. In
regression on scalar data, failure to account for noisy independent variables
can lead to biased parameter estimates. With noisy independent variables,
linear models fitted via ordinary least squares (OLS) will show attenuation
bias, wherein the slope will be underestimated. In this work, we derive an
analogue of attenuation bias for linear operator regression with white noise
in both the independent and dependent variables, showing that the norm upper
bound of the operator learned via OLS decreases with increasing noise in the
independent variable. In the nonlinear setting, we computationally demonstrate
underprediction of the action of the Burgers operator in the presence of noise
in the independent variable. We propose error-in-variables (EiV) models for
two operator regression methods, MOR-Physics and DeepONet, and demonstrate
that these new models reduce bias in the presence of noisy independent
variables for a variety of operator learning problems. Considering the Burgers
operator in 1D and 2D, we demonstrate that EiV operator learning robustly
recovers operators in high-noise regimes that defeat OLS operator learning. We
also introduce an EiV model for time-evolving PDE discovery and show that OLS
and EiV perform similarly in learning the Kuramoto-Sivashinsky evolution
operator from corrupted data, suggesting that the effect of bias in OLS
operator learning depends on the regularity of the target operator.

##  1 Introduction

Operator regression, or operator learning, has emerged as an important field
in scientific computing that focuses on the flexible and expressive discovery
of partial differential equations (PDEs) and related models. These methods fit
parameterizations of operators, i.e., mappings between functions, using
observations of the input and output of the operators. Recently, operator
regression methods have begun to incorporate techniques from machine learning
such as deep neural networks and Gaussian processes, opening the way for novel
applications in scientific computing challenges, such as data-driven model
discovery [1], surrogate models of data-to-solution maps [2, 3], and closure
models in fluid flow [4]. However, current operator regression methods based
on deep learning assume that the independent variable, i.e. the input to the
operator, is free of noise. The objective of this work is to explore the bias
in models learned from noisy independent variables and to propose a
correction, applicable to a wide variety of operator regression architectures,
based on the error-in-variables method in classical statistics.

Fitting parametrized operators to observations is a classical problem and has
been approached, e.g., with Bayesian methods [5, 6, 7], PDE-constrained
optimization [8, 9], and physics-informed Gaussian processes [10, 11]. Earlier
works leveraged significant prior knowledge of physics and fitted small
numbers of physically interpretable parameters. In contrast, recent methods
based on high-dimensional deep neural network (DNN) parametrization of
operators have established deep operator learning as a widely applicable and
domain-agnostic area of scientific computing. Deep operator learning methods
vary based on the discretizations of function spaces inherent in each method
as well as the architecture and training of the DNN. They include modal
methods [12, 13, 14, 15], graph based methods [16, 17], PCA based methods
[18], meshless methods [1], trunk-branch based methods [19, 3], and time-
stepping methods [20, 21, 14]. Such methods can be purely data-driven or
incorporate knowledge from physics [22], and can be utilized as general
operator surrogates or be specifically for PDE model discovery [13]. Similar
techniques in the computer science and statistics literature have been
described as function to function regression within the field of functional
data analysis (FDA) [23] but applied outside the PDE context. These methods
include reproducing kernel Hilbert space approaches [24, 25], additive models
[26], wavelet-based approaches [27], and neural network models [28, 29]. See
[30] for a broad overview of the topic.

The effect of noisy inputs in the training and test data for deep operator
regression methods is relatively unexplored, despite being widely studied in
the related context of adversarial examples in deep learning [31, 32]. In
single and multivariate statistics, error in the independent variables leads
to inconsistent parameter estimates without proper error modelling. In
particular, linear models fit via ordinary least squares (OLS) to scalar data
$\\{(x+\epsilon,y)\\}$ with error in the independent variable persistently
underpredict the slope as

$m^{*}=m\frac{\mathrm{var}(x)}{\mathrm{var}(x)+\mathrm{var}(\epsilon)},$ (1)

where $m^{*}$ is the predicted slope, $m$ is the true slope, $x$ is the
independent variable, and $\epsilon$ is the error in $x$ [33]. This phenomenon
is known as attenuation bias and methods such as total least squares [34], and
Deming regression [35] have been developed to counteract it. More generally,
including in nonlinear contexts, an error-in-variable (EiV) model is required
to correct for inconsistency arising from error in the independent variables.
Typically, EiV models are highly specialized to specific problems. See [36,
37] for overviews of this topic.

In contrast to single and multivariate data, where one often has more control
over the error in the independent variables, e.g. the placement of sensors,
functional data originate from signals which are generally noisy, particularly
when measured in extreme environments or at high sampling rates. This makes it
essential that operator regression methods be robust to noise in both the
input (independent) and output (dependent) variables. To the best of the
authors’ knowledge, there are currently no error-in-variable models available
for either function-to-function regression or operator regression methods.

In this work, we derive a generalization to (1) for OLS inference of linear
operators and demonstrate computationally that the action of OLS learned
operators underpredict the action of the true operators. Under the assumption
that the underlying functional data is smooth but corrupted by white noise, we
propose EiV models for the nonlinear operator regression methods, MOR-Physics
[12, 13] and DeepONet [19]. We demonstrate that these models can correct for
the bias introduced by noisy independent variables for several PDE learning
problems.

##  2 Operator regression for noisy input data

Many operator regression methods seek to infer an operator by solving an
ordinary least squares problem,

$\mathcal{L}=\underset{\hat{\mathcal{L}}}{\mathrm{argmin}}\
\underset{(u,v)}{\mathbb{E}}\left[\left\|\hat{\mathcal{L}}u-v\right\|_{V}^{2}\right],$
(2)

where $\mathcal{L}$ is potentially nonlinear, $(u,v)\in U\times V$ represent
the input and output of the target operator and the expectation $\mathbb{E}$
is over a distribution of functions that is appropriate for a given
application, including any stochastic variables meant to model measurement
noise. The specification of the Banach spaces $U$ and $V$ and the distribution
inherent to the expectation (2) of input functions is a major theoretical
challenge. The distribution weights a subset within the function space $U$
where the learned operator is expected to provide an accurate surrogate of the
target operator, as well as measurement noise. In practice, it assumed that a
finite sample of training functions are available which represent this
theoretical distribution, and that these functions are discretized in some way
so that they can be represented as finite-dimensional vectors,

$\mathbf{u}\in\mathbb{R}^{d_{1}},\quad\mathbf{v}\in\mathbb{R}^{d_{2}},$ (3)

It is also assumed that the operator $\mathcal{L}$ is discretized in a
consistent way,

$\mathcal{L}:\mathbb{R}^{d_{1}}\rightarrow\mathbb{R}^{d_{2}},$ (4)

and that $\|\cdot\|_{V}=\|\cdot\|_{\ell^{2}(\mathbb{R}^{d_{2}})}$ for ease of
computation.

We consider the case when the observations $(u,v)$ originate from pairs
$(\hat{u},\hat{v})$ corrupted by independent white noise. With the
discretization specified by (3) and (4), and the analogous discretization of
$(\hat{u},\hat{v})$, we assume that we are given $N$ observations such that

$\displaystyle\begin{split}\mathbf{u}^{i}&=\hat{\mathbf{u}}^{i}+\bm{\epsilon}_{u}^{i},\quad\mathbf{\mathbf{v}}^{i}=\hat{\mathbf{v}}^{i}+\bm{\epsilon}_{v}^{i},\quad
i=1,2,...,N,\\\
\bm{\epsilon}_{\mathbf{u}}^{i}&\sim\mathcal{N}(\mathbf{0},\sigma^{2}_{u}I),\quad\bm{\epsilon}_{\mathbf{v}}^{i}\sim\mathcal{N}(\mathbf{0},\sigma^{2}_{v}I),\end{split}$
(5)

where $\mathcal{N}(\bm{\mu},\Sigma)$ is the multivariate normal distribution
with mean $\bm{\mu}$ and covariance $\Sigma$. We seek an operator
$\mathcal{L}$ such that

$\mathcal{L}(\hat{\mathbf{u}})\approx\hat{\mathbf{v}}$ (6)

for pairs $(\hat{\mathbf{u}},\hat{\mathbf{v}})$ pairs of function in a given
sample set (the test set). The discretized ordinary least squares problem from
(2) for operator regression using this data is then to minimize the loss

$J=\frac{1}{N}\sum_{i=1}^{N}\|\mathcal{L}\mathbf{u}^{i}-\mathbf{v}^{i}\|^{2}_{\ell^{2}(\mathbb{R}^{d_{2}})}.$
(7)

We stress that the objective is to infer the mapping from noiseless input to
noiseless output, having access only to noisy observations of input and output
data for training.

###  2.1 Bias in operators inferred from Ordinary Least-Squares (OLS) operator
regression

The presence of noise given by (5) while minimizing a loss function as in (2)
leads to systematic bias in the prediction of $\mathcal{L}$, in a way that is
analogous to the attenuation bias described previously. We generate pairs of
noisy functions,

$(u^{i},v^{i})=(\hat{u}^{i}+\epsilon_{u}^{i},\partial_{x}\hat{u}^{i^{2}}+\epsilon_{v}^{i}),$
(8)

where the $u^{i}$ and $v^{i}$ are generated as described in Section 6, and
attempt to infer the Burgers operator $\mathcal{L}u=\partial_{x}u^{2}$ using
the MOR-Physics operator regression method, described in Section 4.1. Figure 1
demonstrates the bias in the learned operator by plotting the action on a test
function; the predicted output function exhibits $\ell_{2}$-norm smaller than
the true output function. This is typical of the output on test functions,
suggesting that the operator norm of $\mathcal{L}$ itself is attenuated, i.e.,
that the OLS learned operator “underpredicts” the action of the true Burgers
operator. In Table 1, we compute the maximum and average norms for the action
of the OLS learned and true Burgers operator over a set of 1000 test functions
in the unit ball of $\ell^{2}$; note that the former approximates the operator
norm of the learned $\mathcal{L}$ over the sample space. These statistics
strongly suggest that the operator norm is biased to zero. The hyperparameters
used in this study are available in Appendix B.

| OLS operator | True operator
---|---|---
$\max||\mathcal{L}u||_{\ell_{2}(\mathbb{R}^{d_{2}})}$ | 4.42 | 12.45
$\mathrm{average}||\mathcal{L}u||_{\ell_{2}(\mathbb{R}^{d_{2}})}$ | 2.40 | 7.75
Table 1: Operator norm statistics for true Burgers operator and OLS inferred
operator. Computed over 1000 samples with
$||u||_{\ell_{2}(\mathbb{R}^{d_{1}})}=1$. Figure 1: Operator learned from
noisy independent variables with OLS underpredicts the action of the true
operator. (Top left) sample of training $u$ () and underlying smooth function
$\hat{u}$ (). (Bottom left) sample of training $v$ () and underlying smooth
function $\hat{v}=\partial_{x}^{2}\hat{u}$ (). (Top right) smooth test
function. (Bottom right) action of true Burgers operator on smooth test
function () and action of OLS learned operator ().

The following theorem shows that this can be expected for linear operators. It
extends the classical result on attenuation bias to the case of a linear
operator learned by OLS. We obtain the theorem for an idealized case of
infinitely many finite-dimensional samples. That is, we assume a
discretization of the form (3), but for infinitely many pairs, consistent with
the implicit distribution of samples. Then, rather than the fully discrete
loss (7), the theorem is stated for a loss of the form

$\overline{J}={\mathbb{E}}\left[\left\|\hat{\mathcal{L}}\mathbf{u}-\mathbf{v}\right\|^{2}_{\ell^{2}(\mathbb{R}^{d_{2}})}\right],\quad\mathbf{u}\in\mathbb{R}^{d_{1}},\mathbf{v}\in\mathbb{R}^{d_{2}}.$
(9)

The fully discrete loss given in (7) converges to $\overline{J}$ as the sample
size increases. The following theorem is proven in Appendix A.

###### Theorem 1

The minimizer to (9) among linear $\hat{\mathcal{L}}$ is

$\mathcal{L}=\mathbb{E}\left[\hat{\mathbf{v}}\hat{\mathbf{u}}^{\top}\right]\left(\mathbb{E}\left[\hat{\mathbf{u}}\hat{\mathbf{u}}^{\top}\right]+\sigma^{2}_{u}I\right)^{-1}.$
(10)

The norm upperbound for this operator is

$\|\mathcal{L}\|\leq\frac{\|\mathbb{E}\left[\hat{\mathbf{v}}\hat{\mathbf{u}}^{\top}\right]\|}{\|\mathbb{E}\left[\hat{\mathbf{u}}\hat{\mathbf{u}}^{\top}\right]+\sigma^{2}_{u}I\|}.$
(11)

This norm upperbound decreases with increasing levels of noise, $\sigma_{u}$,
suggesting a zero bias for the OLS inferred operator with large noise in $u$.
Theorem 1 should be compared to the attenuation bias result for classical
regression (1). The significance of this result is that while zero-bias noise
in the dependent variable $v$ can be compensated for by a large sample size,
the presence of seemingly innocuous zero-bias noise in the independent
variable $u$ persists in biasing the optimal solution of the ordinary least
squares problem (9) itself. The numerical results shown in Figure 1 and Table
1 above suggests similar behavior for nonlinear operator regression as well.
This necessitates the error-in-variables model we propose below. In
particular, ad-hoc denoising that does not address the limitations of OLS is
not guaranteed to remove this bias, in contrast to our error-in-variables
approach we discuss below.

##  3 Error-in-variables (EiV) model

Instead of applying OLS to infer the operator, we model the error for both $u$
and $v$ with the joint probability density function (PDF),

$\begin{bmatrix}{\tilde{u}^{i}}-{u}^{i}\\\
{v^{i}}-\mathcal{L}\tilde{u}^{i}\end{bmatrix}\sim\mathcal{N}\left(\mathbf{0},\begin{bmatrix}\sigma^{2}_{u}I&\\\
&\sigma^{2}_{v}I\end{bmatrix}\right)$ (12)

where $\tilde{u}$ is a “denoised” version of ${u}$. $\mathcal{L}$ along with
$\tilde{u},\sigma_{u},\sigma_{v}$ can be recovered via maximum likelihood
estimation (MLE),

$\mathcal{L},\tilde{u},\sigma_{u},\sigma_{v}=\underset{\mathcal{L},\tilde{u},\sigma_{u},\sigma_{v}}{\mathrm{argmax}}\prod_{i}p\left(\begin{bmatrix}\tilde{u}^{i}-{u}^{i}\\\
v^{i}-\mathcal{L}\tilde{u}^{i}\end{bmatrix}\right),$ (13)

where $p(\cdot)$ refers to the probability density of the indicated random
variable. Real-world functional data is often smooth. Therefore, we simplify
recovery of $\tilde{u}$ via a low-pass filter, $\tilde{u}=\mathcal{G}u$, and
optimize over $\mathcal{G}$ instead of $\tilde{u}$. We select for
$\mathcal{G}$, the smooth spectral filter,

$\mathcal{G}(u)=\mathcal{F}^{-1}\mathrm{erfc}(a(\kappa-\kappa_{c}))\mathcal{F}u$
(14)

where $a$ and $\kappa_{c}$ are the filter bandwidth and cutoff wavenumber,
respectively. The optimization over $\mathcal{G}$ in (13) is replaced with
optimization over the two filter parameters.

In our numerical studies, we have observed that priors on the filter
parameters can improve robustness. For certain problems we place a Beta
distribution prior on $\kappa_{c}$,

$\displaystyle\kappa_{c}/\beta_{\kappa_{c}}\sim\mathrm{Beta}(1+\varepsilon,1+\varepsilon).$
(15)

where $\beta_{\kappa_{c}}$ is a hyperparameter that specify the maximum
allowable $\kappa_{c}$. With this prior, we perform maximum a posteriori (MAP)
estimation to obtain $\mathcal{L}$,

$\mathcal{L},\kappa_{c},a,\sigma_{u},\sigma_{v}=\underset{\mathcal{L},\kappa_{c},a,\sigma_{u},\sigma_{v}}{\mathrm{argmax}}\prod_{i}p\left(\begin{bmatrix}\mathcal{G}u^{i}-{u}^{i}\\\
v^{i}-\mathcal{L}\mathcal{G}{u}^{i}\end{bmatrix}\right)p(\kappa_{c}/\beta_{\kappa_{c}}).$
(16)

For small $\varepsilon$, $\mathrm{Beta}$ is a smooth approximation to the
uniform distribution, and therefore MAP is amenable to gradient descent based
algorithms. This prior forces our EiV model to select a smaller $\kappa_{c}$
so that $\mathcal{G}u^{i}$ is smooth. For our numerical studies, we use
$\varepsilon=0.01$.

##  4 Operator regression methods

In this section, we summarize the two operator regression methods examined in
our experiments.

###  4.1 MOR-Physics

The MOR-Physics operator regression method uses the following operator
parameterization,

$\mathcal{L}(u)=\sum_{i}^{N_{o}}\mathcal{F}^{-1}g_{i}(\kappa)\mathcal{F}h_{i}(u),$
(17)

where $\mathcal{F}$ is the Fourier transform, $g_{i}$ and $h_{i}$ are neural
networks, $\kappa$ is the wavenumber, and $N_{o}$ is a hyperparameter. Here,
$h_{i}(u)$ is applied pointwise, such that, $h_{i}(u(x))=(h_{i}\circ u)(x)$.
The optimization over $\mathcal{L}$ in (13) is replaced here with optimization
over the neural network parameters. For all numerical studies, we select the
network width=$5$, depth=$5$, and activation function as elu for both $h_{i}$
and $g_{i}$.

###  4.2 DeepONet

The DeepONet operator regression method includes a branch network, $b$, that
takes as input $u$ and a trunk network, $t$, that takes as input $x$, a single
grid point on the function $v$. In this work, we use the unstacked variant of
the DeepONet with the following operator parameterization to approximate $v$
at a single grid point $x$,

$\mathcal{L}(u,x)=\sum_{k=1}^{p}b_{k}(u)t_{k}(x)+b_{0},$ (18)

where $p$ is the network width and $b_{0}$ is an additional bias term.

For all computational studies involving DeepONet, we use fully connected
networks with tanh activation functions for both the trunk and branch. We
chose depths of 3 and 2 for the trunk and branch, respectively and 70 for the
network width, $p$.

##  5 Operator regression with error-in-variables model for time-evolving
systems

In addition to the inference problem discussed in the previous sections, we
are also interested in inferring time-evolving PDE’s from noise corrupted
solutions. Given a solution, $u(x,t)=\hat{u}(x,t)+\epsilon(x,t)$, we seek to
recover a PDE in the following form,

$\displaystyle\partial_{t}u$ $\displaystyle=\mathcal{L}(u),\quad x\in\Omega$
(19) $\displaystyle u(x,0)$ $\displaystyle=u_{0}(x)$ (20)
$\displaystyle\mathcal{B}u$ $\displaystyle=0,\quad x\in\partial\Omega$ (21)

We discretize this PDE in time with forward Euler, obtaining,

$\displaystyle u_{n+1}=u_{n}+\Delta
t\mathcal{L}(u_{n})=\mathcal{P}(u_{n})\quad n=1,2,\ldots,\hat{N}_{t},$ (22)

This temporal discretization results in a ResNet-type architecture which has
been effectively used to parametrize and discover governing evolution
equations [38, 21, 12, 14]. We still parameterize $\mathcal{L}$ with either
MOR-Physics or DeepONet. However, we can now ideally model the error in $u$
over all time steps with the following joint PDF,

$\begin{bmatrix}\mathcal{G}u_{0}-u_{0}\\\ \mathcal{P}\mathcal{G}u_{0}-u_{1}\\\
\vdots\\\ \mathcal{P}^{\hat{N}_{t}}\mathcal{G}u_{0}-u_{n}\\\
\end{bmatrix}\sim\mathcal{N}\left(\mathbf{0},\sigma_{u}^{2}I\right).$ (23)

Due to the computational complexity of evaluating probability densities from
the above distribution, we instead consider the following marginalized
distribution as a simplification,

$\begin{bmatrix}\mathcal{G}u_{0}-u_{0}\\\
\mathcal{P}^{N_{t}}\mathcal{G}u_{0}-u_{N_{t}}\end{bmatrix}\sim\mathcal{N}\left(\mathbf{0},\sigma^{2}_{u}I\right),$
(24)

where $N_{t}$ is a hyperparameter indicating a maximum timestep.

##  6 Results

In the following sections, we compare OLS and EiV operator regression using
both MOR-Physics and DeepONet. We perform all test cases on the periodic
domain, $\Omega=[0,L]^{d}$, where $d$ is the spatial dimension of the problem.
For each test, we generate samples of smooth functions, $\hat{\mathbf{u}}^{i}$
by applying a spectral low pass filter to samples of white noise. For time
independent problems, we then apply the a priori known operator to
$\hat{\mathbf{u}}^{i}$ and obtain $\hat{\mathbf{v}}^{i}$.

We next corrupt the data by adding white noise and obtain $\mathbf{u}^{i}$ and
$\mathbf{v}^{i}$. To standardize the amount of noise added to
$\hat{\mathbf{u}}$ and $\hat{\mathbf{v}}$, we tune the standard deviation of
the white noise to target specific signal-to-noise ratios (SNR’s),

$\mathrm{SNR}=10\log_{10}\left(\frac{\mathrm{RMS}(\mathbf{u})}{\sigma^{2}_{u}}\right)^{2},$
(25)

and likewise for $\mathbf{v}$. For all test cases, we target the same SNR for
both $\mathbf{u}$ and $\mathbf{v}$. With this noisy data, we attempt to infer
the operator or PDE. Finally we evaluate our inference by generating noiseless
test data, $u_{\mathrm{test}}$ using the same method discussed above, and
comparing the action of the inferred operator to the action of the true
operator on $u_{\mathrm{test}}$.

For time evolving problems, we numerically integrate the a priori known PDE,
using $\hat{\mathbf{u}}^{i}_{0}=\hat{\mathbf{u}}^{i}$ as the initial
condition, obtaining $\hat{\mathbf{u}}^{i}_{n}$ for all $n$ timesteps. We next
corrupt this data with white noise as discussed above, obtaining
$\mathbf{u}_{n}$, from which we attempt to infer an evolution PDE. We evaluate
our inference by generating a noiseless initial condition
$\mathbf{u}_{0}=\mathbf{u}_{\mathrm{test}}$, evolving the inferred PDE, and
comparing to the evolution of the true PDE using the same
$\mathbf{u}_{\mathrm{test}}$ as an initial condition.

To perform the optimization in (13) and (16), we use the stochastic gradient
based optimizer, ADAM [39]. Batch sizes, learning rates, sample sizes, and
other hyperparameters for all numerical studies are listed in Appendix B.

###  6.1 Learning operators with the MOR-Physics EiV model

In these sections, we compare the OLS and EiV models using the MOR-Physics
operator parameterization discussed in Section 4.1.

####  6.1.1 Recovering the Burgers operator

Figure 2:  EiV model improves recovery of true Burgers operator in the
presence of noisy independent variables. (Left) Underlying smooth function
$\hat{u}$ () and training $u$ for increasing $\mathrm{SNR}=8$ (),
$\mathrm{SNR}=4$ (), $\mathrm{SNR}=0$ (), and $\mathrm{SNR}=-4$ (). (Right)
Action of true Burgers operator () on noiseless test
$\mathbf{u}_{\mathrm{test}}$ and action of learned operators from data with
decreasing SNR for OLS (Top right) and EiV (Bottom right).

We consider the learning problem discussed in Figure 1 where we attempt to
recover the Burgers operator, $\mathcal{L}u=\partial_{x}u^{2}$. In Figure 2,
we apply OLS and EiV inferred error models to this dataset, recover operators,
compute the action of these operators on a noiseless test function,
$u_{\mathrm{test}}$, and compare to the action of the true operator,
$v_{\mathrm{test}}$. We observe that the OLS model recovered operators only
reproduce the action of the true Burgers operator for the highest SNR
examined, while underpredicting the action for lower SNR’s. In contrast, the
EiV model is able to recover operators down to $\mathrm{SNR}=4$ with high
accuracy. However, for lower SNR’s the EiV model also fails to recover a
suitable operator.

Figure 3:  Effect of cutoff wavenumber prior on filter for EiV model. (Left)
Action of MLE estimate of filters on noisy $u^{i}$ () for decreasing SNR and
corresponding noiseless $\hat{u}^{i}$ (). (Right) Action of MAP estimate of
filters ($\kappa_{c}$ prior) on $u^{i}$ with hyperparameters,
$\beta_{\kappa_{c}}=10$ (),
$\beta_{\kappa_{c}}=20(\includegraphics[height=6.0pt]{styles/C1,l,n,1.pdf})$,
$\beta_{\kappa_{c}}=40(\includegraphics[height=6.0pt]{styles/C2,l,n,1.pdf})$,
and
$\beta_{\kappa_{c}}=80(\includegraphics[height=6.0pt]{styles/C3,l,n,1.pdf})$.

On the left four subplots of Figure 3, we examine the action of the filter
$\mathcal{G}$ on samples of noisy input functions, $u^{i}$, with decreasing
SNR. For the high SNR data, the EiV model successfully identifies filter
parameters that smooth the input functions. However, for low SNR data, the EiV
model selects filters that fail to smooth the input functions, suggesting the
method finds high $\kappa_{c}$.

####  6.1.2 Improved operator learning with smoothness prior

Figure 4: Cutoff wavenumber prior improves EiV model. Action of EiV operator
on $u_{\mathrm{test}}$ learned from $\mathrm{SNR}=8$ (), $\mathrm{SNR}=4$ (),
$\mathrm{SNR}=0$ (), and $\mathrm{SNR}=-4$ () for various
$\beta_{\kappa_{c}}$. Action of true operator ().

The results in Figure 2 can be improved by adding priors on the filter
parameters as discussed in Section 3 to constrain $\kappa_{c}$ to smaller
values and enforce smoothness on $\mathcal{G}u^{i}$. In Figure 4, we plot the
action of the learned operators with priors on $\kappa_{c}$ and four different
hyperparameters, $\beta_{\kappa_{c}}$. We find, for a wide range of
$\beta_{\kappa_{c}}$, that the prior enables successful recovery of the
Burgers operator even for the lowest SNR. However for the largest
$\beta_{\kappa_{c}}=80$, our EiV model can no longer consistently recover the
Burgers operator. Notably, these results are relatively insensitive to
$\beta_{\kappa_{c}}$, so careful hyperparameter selection is not necessary.

On the right four subplots of Figure 3, we examine the action of the filter
$\mathcal{G}$ on sample noisy input functions, $u^{i}$, with decreasing SNR
and the four $\beta_{\kappa_{c}}$ hyperparameters. We find that the MAP
estimate always produces a operative filter that smooths $u^{i}$, even for the
lowest SNR.

####  6.1.3 Learning the 2D Burgers operator

Figure 5: EiV model improves recovery of 2D Burgers operator from noisy data.
(Top) Noiseless test data, $u_{\mathrm{test}}$, and action of true operator,
$v_{\mathrm{test}}$. (Bottom left) Action of OLS (Bottom left) and EiV (Bottom
right) learned operators on $u_{\mathrm{test}}$ for decreasing SNR.

In this section, we attempt to recover a 2D generalization of the Burgers
operator [40], $\mathcal{L}=\partial_{x}u^{2}+\partial_{y}u^{2}$. Figure 5
compares the OLS and EiV learned operators for decreasing SNR. For the EiV
model, we included the $\kappa_{c}$ prior with $\beta_{\kappa_{c}}=10$. As in
the 1D Burgers operator test, we find that action of the OLS learned operator
severely underpredicts the action of the true operator, especially at very low
SNR. The EiV learned operator, however, captured the action of the true
operator with SNR down to $-4$. Even with lower SNR’s, the EiV operator did
not underpredict the true operator as severely as the OLS operator.

####  6.1.4 Learning the Kuramoto–Sivashinsky equation

Figure 6: OLS and EiV models perform similarly for KS equation inference. (Top
left) Noiseless test data, $u_{\mathrm{test}}$. OLS (Bottom left) and EiV
(Bottom right) inferred operators for increasing hyperparameter, $N_{t}$. Top
right Lyapunov exponents for true equation (); OLS equation with $N_{t}=2$ (),
$N_{t}=4$ (), $N_{t}=8$ (); and EiV equation with $N_{t}=2$ (), $N_{t}=4$ (),
$N_{t}=8$ ().

In this section we attempt to recover the Kuramoto–Sivashinsky (KS) equation,

$\partial_{t}u+0.5\partial_{x}u^{2}+\partial_{x}^{2}u+\partial_{x}^{4}u=0,$
(26)

using the method outlined in Section 5. To generate the data, we solve KS
equation using Pseudospectral methods. We use Orszag’s 3/2 zero-padding
technique to eliminate aliases from the Fourier transform of $u^{2}$, which is
computed in the ($x$,$t$) domain [41]. We use the low-storage Runge-Kutta
method for temporal discretization [42] which has been widely used in the
direct numerical simulation of turbulent flows [43, 44]. It treats the
nonlinear term explicitly with third-order accuracy. Linear terms are
implicitly treated, similarly to the Crank-Nicolson method, with second-order
accuracy.

In Figure 6, we infer the equation using the OLS and EiV models for increasing
final timestep, $N_{t}$, and found that both performed similarly. For this
study, we used $\mathrm{SNR}=4$. Qualitatively, neither model was able to
capture the dynamics of the KS equation with $N_{t}=2$ and $N_{t}=4$, but both
models were able were able to capture the dynamics with $N_{t}=8$.

As a chaotic system, the KS equation can be characterized by computing
Lyapunov exponents as outlined in [45]. In Figure 6, we also compute the first
four Lyapunov exponents for the true and learned equations. As observed in the
qualitative comparison, OLS and EiV perform similarly, only successfully
recovering the true KS equation for $N_{t}=8$. This is in contrast to
regressing the Burgers operator, in which EiV outperformed OLS. We hypothesis
that is due to the difference in regularity of the target operators; the
Burgers operator, a differential operator, reduces the regularity of the input
function, while the KS evolution operator possesses smoothing properties [46].
As a result, one can expect the operator norm over most subspaces of sample
functions to be much higher for the Burgers operator than for the KS evolution
operator, rendering the OLS attenuation bias more significant in regressing
the former. We plan to explore in more detail the advantages of EiV over OLS
in time dependent PDE learning and time independent operator learning in
future work.

###  6.2 Learning the Burgers operator with DeepONet EiV model

Figure 7: EiV inference performs better than OLS inference for low SNR in
DeepONet. (Left) Relative $\ell_{2}$ error vs. SNR for OLS () and EiV ().
(Right) Action of true (), OLS (), and EiV () operators on $u_{\mathrm{test}}$
for increasing SNR. Error bars indicate 25th and 75th percentile over 100 test
samples. Figure 8: EiV inference improves with training set size in DeepONet.
(Left) Relative $\ell_{2}$ error vs. training set size for EiV (). (Right)
Action of true () and EiV () operators on $u_{\mathrm{test}}$ for increasing
training set size. Results computed for $\mathrm{SNR}=17$. Error bars indicate
25th and 75th percentile over 100 test samples.

In this section, we examine EiV operator learning for the 1D Burgers operator
using the DeepONet method outlined in Section 4.2. In Figure 7, we compute the
$\ell_{2}$ error between the actions of OLS and EiV learned operators on test
functions and the true actions. As in MOR-Physics, the EiV learned operator
robustly learns from noisy data, even at very low SNR. Finally, we examine the
effect of training set size on EiV learning in Figure 8. We find that for very
low samples, EiV fails to recover the Burgers operator to high accuracy. With
larger numbers of samples, EiV can successfully recover the operator. Future
work may focus on examining the posterior distribution of operators in the
small data regime.

##  7 Conclusion

We have demonstrated that operator regression performed by minimizing a least-
squares loss is prone to attenuation bias when the input, or independent
variable, is corrupted by white noise. This is supported both by an upper
bound on the operator norm for linear operator regression using ordinary least
squares (OLS) as well as numerical results for nonlinear operator regression
for problems such as recovering the Burgers operator from noisy observations.
We proposed an Error-in-Variables (EiV) model for operator regression that
replaces the least-squares loss with maximum likelihood estimation or maximum
a posteriori estimation with an appropriate smoothness prior. The EiV method
is applicable to a wide variety of existing operator regression methods; we
demonstrate this by combining it with the MOR-physics method of [13] and the
DeepONet method of [19]. For these methods, the EiV model significantly
improves the operator prediction fidelity for a variety of problems given
noise in the input variable, as demonstrated for the Burgers equation in one
and two dimensions. Finally, we introduce an EiV model for time evolving PDEs
and show OLS and EiV perform similarly in recovering the Kuramoto-Sivashinky
equation. This suggests that the regularity properties of the target operator
effect the severity of OLS attenuation bias in practice.

## Acknowledgements

Sandia National Laboratories is a multimission laboratory managed and operated
by National Technology and Engineering Solutions of Sandia, LLC, a wholly
owned subsidiary of Honeywell International, Inc., for the U.S. Department of
Energy’s National Nuclear Security Administration under contract DE-NA0003525.
This paper describes objective technical results and analysis. Any subjective
views or opinions that might be expressed in the paper do not necessarily
represent the views of the U.S. Department of Energy or the United States
Government. SAND Number: SAND2022-5179 C.

The work of R.G. Patel, I. Manickam, and M. Gulian are supported by the U.S.
Department of Energy, Office of Advanced Scientific Computing Research under
the Collaboratory on Mathematics and Physics-Informed Learning Machines for
Multiscale and Multiphysics Problems (PhILMs) project.

## References

  * [1] Nathaniel Trask, Ravi G. Patel, Ben J. Gross, and Paul J. Atzberger.  GMLS-Nets: A framework for learning from unstructured data.  arXiv preprint arXiv:1909.05371, 2019.
  * [2] Huaiqian You, Yue Yu, Marta D’Elia, Tian Gao, and Stewart Silling.  Nonlocal Kernel Network (NKN): a stable and resolution-independent deep neural network.  arXiv preprint arXiv:2201.02217, 2022.
  * [3] Shengze Cai, Zhicheng Wang, Lu Lu, Tamer A Zaki, and George Em Karniadakis.  DeepM&Mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks.  Journal of Computational Physics, 436:110296, 2021.
  * [4] Karthik Duraisamy, Gianluca Iaccarino, and Heng Xiao.  Turbulence modeling in the age of data.  Annual Review of Fluid Mechanics, 51:357–377, 2019.
  * [5] Guofei Pang, Paris Perdikaris, Wei Cai, and George Em Karniadakis.  Discovering variable fractional orders of advection–dispersion equations from field data using multi-fidelity Bayesian optimization.  Journal of Computational Physics, 348:694–714, 2017.
  * [6] Andrew M Stuart.  Inverse problems: a Bayesian perspective.  Acta Numerica, 19:451–559, 2010.
  * [7] Nicolas Garcia Trillos and Daniel Sanz-Alonso.  The Bayesian formulation and well-posedness of fractional elliptic inverse problems.  Inverse Problems, 33(6):065006, 2017.
  * [8] Marta D’Elia and Max Gunzburger.  Identification of the diffusion parameter in nonlocal steady diffusion problems.  Applied Mathematics & Optimization, 73(2):227–249, 2016.
  * [9] Olena Burkovska, Christian Glusa, and Marta D’Elia.  An optimization-based approach to parameter learning for fractional type nonlocal models.  Computers & Mathematics with Applications, 2021.
  * [10] Mamikon Gulian, Maziar Raissi, Paris Perdikaris, and George Karniadakis.  Machine learning of space-fractional differential equations.  SIAM Journal on Scientific Computing, 41(4):A2485–A2509, 2019.
  * [11] Maziar Raissi, Paris Perdikaris, and George Em Karniadakis.  Machine learning of linear differential equations using Gaussian processes.  Journal of Computational Physics, 348:683–693, 2017.
  * [12] Ravi G. Patel and Olivier Desjardins.  Nonlinear integro-differential operator regression with neural networks.  arXiv preprint arXiv:1810.08552, 2018.
  * [13] Ravi G Patel, Nathaniel A Trask, Mitchell A Wood, and Eric C Cyr.  A physics-informed operator regression framework for extracting data-driven continuum models.  Computer Methods in Applied Mechanics and Engineering, 373:113500, 2021.
  * [14] Tong Qin, Kailiang Wu, and Dongbin Xiu.  Data driven governing equations approximation using deep neural networks.  Journal of Computational Physics, 395:620–635, 2019.
  * [15] Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar.  Fourier neural operator for parametric partial differential equations.  In International Conference on Learning Representations, 2021.
  * [16] Anima Anandkumar, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Nikola Kovachki, Zongyi Li, Burigede Liu, and Andrew Stuart.  Neural operator: Graph kernel network for partial differential equations.  In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
  * [17] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar.  Neural operator: Graph kernel network for partial differential equations.  arXiv preprint arXiv:2003.03485, 2020.
  * [18] Kaushik Bhattacharya, Bamdad Hosseini, Nikola B Kovachki, and Andrew M Stuart.  Model reduction and neural networks for parametric PDEs.  arXiv preprint arXiv:2005.03180, 2020.
  * [19] Lu Lu, Pengzhan Jin, and George Em Karniadakis.  DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators.  arXiv preprint arXiv:1910.03193, 2019.
  * [20] Huaiqian You, Yue Yu, Nathaniel Trask, Mamikon Gulian, and Marta D’Elia.  Data-driven learning of nonlocal physics from high-fidelity synthetic data.  Computer Methods in Applied Mechanics and Engineering, 374:113553, Feb 2021.
  * [21] Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong.  PDE-net: Learning PDEs from data.  In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3208–3216. PMLR, 10–15 Jul 2018\.
  * [22] Sifan Wang, Hanwen Wang, and Paris Perdikaris.  Learning the solution operator of parametric partial differential equations with physics-informed DeepONets.  Science advances, 7(40):eabi8605, 2021.
  * [23] Bernard Walter Silverman and James O Ramsay.  Functional data analysis.  In International Encyclopedia of the Social and Behavioral Sciences. Amsterdam: Elsevier, 2001.
  * [24] Ming Yuan and T. Tony Cai.  A reproducing kernel Hilbert space approach to functional linear regression.  The Annals of Statistics, 38(6):3412 – 3444, 2010.
  * [25] Hachem Kadri, Emmanuel Duflos, Philippe Preux, Stéphane Canu, Alain Rakotomamonjy, and Julien Audiffren.  Operator-valued kernels for learning from functional response data.  Journal of Machine Learning Research, 17(20):1–54, 2016.
  * [26] Janet S. Kim, Ana-Maria Staicu, Arnab Maity, Raymond J. Carroll, and David Ruppert.  Additive function-on-function regression.  Journal of Computational and Graphical Statistics, 27(1):234–244, 2018.
  * [27] Mark J Meyer, Brent A Coull, Francesco Versace, Paul Cinciripini, and Jeffrey S Morris.  Bayesian function-on-function regression for multilevel functional data.  Biometrics, 71(3):563–574, 2015.
  * [28] Aniruddha Rajendra Rao and Matthew Reimherr.  Modern non-linear function-on-function regression.  arXiv preprint arXiv:2107.14151, 2021.
  * [29] Connie Khor Li Kou, Hwee Kuan Lee, and Teck Khim Ng.  A compact network learning model for distribution regression.  Neural Networks, 110:199–212, 2019.
  * [30] Jeffrey S. Morris.  Functional regression.  Annual Review of Statistics and Its Application, 2(1):321–359, 2015\.
  * [31] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  arXiv preprint arXiv:1312.6199, 2013.
  * [32] Yuichi Yoshida and Takeru Miyato.  Spectral norm regularization for improving the generalizability of deep learning.  arXiv preprint arXiv:1705.10941, 2017.
  * [33] Jennifer A. Hutcheon, Arnaud Chiolero, and James A. Hanley.  Random measurement error and regression dilution bias.  BMJ (Online), 340, 2010.
  * [34] Ivan Markovsky and Sabine Van Huffel.  Overview of total least-squares methods.  Signal Processing, 87(10):2283–2302, 2007.  Special Section: Total Least Squares and Errors-in-Variables Modeling.
  * [35] Kristian Linnet.  Evaluation of regression procedures for methods comparison studies.  Clinical chemistry, 39(3):424–432, 1993.
  * [36] Silvelyn Zwanzig.  Estimation in nonlinear functional error-in-variables models.  PhD thesis, Universität Hamburg, 2000.
  * [37] Xiaohong Chen, Han Hong, and Denis Nekipelov.  Nonlinear models of measurement errors.  Journal of Economic Literature, 49(4):901–937, 2011.
  * [38] Eldad Haber and Lars Ruthotto.  Stable architectures for deep neural networks.  Inverse Problems, 34(1):014004, dec 2017.
  * [39] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [40] N. A. Mohamed.  Solving one- and two-dimensional unsteady Burgers’ equation using fully implicit finite difference schemes.  Arab Journal of Basic and Applied Sciences, 26(1):254–268, 2019\.
  * [41] Steven A. Orszag.  On the elimination of aliasing in finite-difference schemes by filtering high-wavenumber components.  Journal of the Atmospheric sciences, 28:1074, 1971.
  * [42] Philippe R. Spalart, Robert D. Moser, and Michael M. Rogers.  Spectral methods for the navier-stokes equations with one infinite and two periodic directions.  Journal of Computational Physics, 96:297–324, 10 1991.
  * [43] Sergio Hoyas and Javier Jiménez.  Scaling of the velocity fluctuations in turbulent channels up to Re = 2003.  Physics of Fluids, 18:011702, 2006.
  * [44] Myoungkyu Lee and Robert D. Moser.  Direct numerical simulation of turbulent channel flow up to Re = 5200.  Journal of Fluid Mechanics, 774:395–415, 7 2015.
  * [45] Russell A. Edson, J. E. Bunder, Trent W. Mattner, and A. J. Roberts.  Lyapunov exponents of the Kuramoto–Sivashinsky PDE.  The ANZIAM Journal, 61(3):270–285, 2019.
  * [46] Pierre Collet, J-P Eckmann, Henri Epstein, and Joachim Stubbe.  Analyticity for the Kuramoto-Sivashinsky equation.  Physica D: Nonlinear Phenomena, 67(4):321–326, 1993.

##  Appendix A Proof of Theorem 1

We write the objective function in (9) as

$\displaystyle\overline{J}$
$\displaystyle=\mathbb{E}\left[(\mathcal{L}\mathbf{u}-\mathbf{v})^{\top}(\mathcal{L}\mathbf{u}-\mathbf{v})\right]$
(27)
$\displaystyle=\mathbb{E}\left[(\mathbf{u}^{\top}\mathcal{L}^{\top}-\mathbf{v}^{\top})(\mathcal{L}\mathbf{u}-\mathbf{v})\right]$
(28)
$\displaystyle=\mathbb{E}\left[\mathbf{u}^{\top}\mathcal{L}^{\top}\mathcal{L}\mathbf{u}-\mathbf{v}^{\top}\mathcal{L}\mathbf{u}-\mathbf{u}^{\top}\mathcal{L}^{\top}\mathbf{v}+\mathbf{v}^{\top}\mathbf{v}\right]$
(29)
$\displaystyle=\mathbb{E}\left[\mathbf{u}^{\top}\mathcal{L}^{\top}\mathcal{L}\mathbf{u}-2\mathbf{v}^{\top}\mathcal{L}\mathbf{u}+\mathbf{v}^{\top}\mathbf{v}\right].$
(30)

Using linearity of expectation, this can be written as a sum of three terms,
namely

$\displaystyle I$
$\displaystyle=\mathbb{E}\left[\mathbf{u}^{\top}\mathcal{L}^{\top}\mathcal{L}\mathbf{u}\right]$
(31)
$\displaystyle=\mathbb{E}\left[(\hat{\mathbf{u}}+\bm{\epsilon}_{u})^{\top}\mathcal{L}^{\top}\mathcal{L}(\hat{\mathbf{u}}+\bm{\epsilon}_{u})\right]$
(32)
$\displaystyle=\mathbb{E}\left[(\hat{\mathbf{u}}^{\top}+\bm{\epsilon}_{u}^{\top})\mathcal{L}^{\top}\mathcal{L}(\hat{\mathbf{u}}+\bm{\epsilon}_{u})\right]$
(33)
$\displaystyle=\mathbb{E}\left[\hat{\mathbf{u}}^{\top}\mathcal{L}^{\top}\mathcal{L}\hat{\mathbf{u}}+\hat{\mathbf{u}}^{\top}\mathcal{L}^{\top}\mathcal{L}\bm{\epsilon}_{u}+\bm{\epsilon}_{u}^{\top}\mathcal{L}^{\top}\mathcal{L}\hat{\mathbf{u}}+\bm{\epsilon}_{u}^{\top}\mathcal{L}^{\top}\mathcal{L}\bm{\epsilon}_{u}\right]$
(34)
$\displaystyle=\mathbb{E}\left[\hat{\mathbf{u}}^{\top}\mathcal{L}^{\top}\mathcal{L}\hat{\mathbf{u}}+2\bm{\epsilon}_{u}^{\top}\mathcal{L}^{\top}\mathcal{L}\hat{\mathbf{u}}+\bm{\epsilon}_{u}^{\top}\mathcal{L}^{\top}\mathcal{L}\bm{\epsilon}_{u}\right]$
(35)
$\displaystyle=\mathbb{E}\left[\hat{\mathbf{u}}^{\top}\mathcal{L}^{\top}\mathcal{L}\hat{\mathbf{u}}\right]+\mathbb{E}\left[\bm{\epsilon}_{u}^{\top}\mathcal{L}^{\top}\mathcal{L}\bm{\epsilon}_{u}\right];$
(36) $\displaystyle II$
$\displaystyle=-\mathbb{E}\left[\mathbf{v}^{\top}\mathcal{L}\mathbf{u}\right]$
(37)
$\displaystyle=-2\mathbb{E}\left[(\hat{\mathbf{v}}+\bm{\epsilon}_{v})^{\top}\mathcal{L}(\hat{\mathbf{u}}+\bm{\epsilon}_{u})\right]$
(38)
$\displaystyle=-2\mathbb{E}\left[\hat{\mathbf{v}}^{\top}\mathcal{L}\hat{\mathbf{u}}+\hat{\mathbf{v}}^{\top}\mathcal{L}\bm{\epsilon}_{u}+\bm{\epsilon}_{v}^{\top}\mathcal{L}\hat{\mathbf{u}}+\bm{\epsilon}_{v}^{\top}\mathcal{L}\bm{\epsilon}_{u}\right]$
(39)
$\displaystyle=-2\mathbb{E}\left[\hat{\mathbf{v}}^{\top}\mathcal{L}\hat{\mathbf{u}}\right]+\hat{\mathbf{v}}^{\top}\mathcal{L}\left[\mathbb{E}\bm{\epsilon}_{u}\right]+\left[\mathbb{E}\bm{\epsilon}_{v}^{\top}\right]\mathcal{L}\hat{\mathbf{u}}+\left[\mathbb{E}\bm{\epsilon}_{v}^{\top}\right]\mathcal{L}\left[\mathbb{E}\bm{\epsilon}_{u}\right]$
(40)
$\displaystyle=-2\mathbb{E}\left[\hat{\mathbf{v}}^{\top}\mathcal{L}\hat{\mathbf{u}}\right];$
(41)

and $III=\mathbb{E}\left[\mathbf{v}^{\top}\mathbf{v}\right].$ While $\partial
III/\partial\mathcal{L}=0$, we have

$\displaystyle\frac{\partial I}{\partial\mathcal{L}}$
$\displaystyle=\mathbb{E}\left[\frac{\partial}{\partial\mathcal{L}}\hat{\mathbf{u}}^{\top}\mathcal{L}^{\top}\mathcal{L}\hat{\mathbf{u}}\right]+\mathbb{E}\left[\frac{\partial}{\partial\mathcal{L}}\bm{\epsilon}_{u}^{\top}\mathcal{L}^{\top}\mathcal{L}\bm{\epsilon}_{u}\right]$
(42)
$\displaystyle=\mathbb{E}\left[2\mathcal{L}\hat{\mathbf{u}}\hat{\mathbf{u}}^{\top}\right]+\mathbb{E}\left[2\mathcal{L}\bm{\epsilon}_{u}\bm{\epsilon}_{u}^{\top}\right]$
(43)
$\displaystyle=2\mathcal{L}\mathbb{E}\left[\hat{\mathbf{u}}\hat{\mathbf{u}}^{\top}\right]+2\mathcal{L}\mathbb{E}\left[\bm{\epsilon}_{u}\bm{\epsilon}_{u}^{\top}\right]$
(44)
$\displaystyle=2\mathcal{L}\left(\mathbb{E}\left[\hat{\mathbf{u}}\hat{\mathbf{u}}^{\top}\right]+\sigma^{2}_{u}I\right),$
(45)

and

$\displaystyle\frac{\partial II}{\partial\mathcal{L}}$
$\displaystyle=-\mathbb{E}\left[\frac{\partial}{\partial\mathcal{L}}\hat{\mathbf{v}}^{\top}\mathcal{L}\hat{\mathbf{u}}\right]$
(46)
$\displaystyle=-2\mathbb{E}\left[\hat{\mathbf{v}}\hat{\mathbf{u}}^{\top}\right].$
(47)

Setting the derivative of the objective function equal to zero, we obtain

$\mathcal{L}\left(\mathbb{E}\left[\hat{\mathbf{u}}\hat{\mathbf{u}}^{\top}\right]+\sigma^{2}_{u}I\right)=\mathbb{E}\left[\hat{\mathbf{v}}\hat{\mathbf{u}}^{\top}\right]$
(48)

which has solution

$\mathcal{L}=\mathbb{E}\left[\hat{\mathbf{v}}\hat{\mathbf{u}}^{\top}\right]\left(\mathbb{E}\left[\hat{\mathbf{u}}\hat{\mathbf{u}}^{\top}\right]+\sigma^{2}_{u}I\right)^{-1}.$
(49)

##  Appendix B Hyperparameters

The following tables lists the hyperparameters used in our numerical examples.
The multiple learning rates indicate that we reinitialized the Adam optimizer
during training and lowered the learning rate.

| Figures 1, 2, 3, and 4 | Figure 5 | Figure 6 | Figures 7 and 8
---|---|---|---|---
Domain size, $L$ | $2\pi$ | $2\pi\times 2\pi$ | $30\pi$ | 512
Grid size | 128 | $128\times 128$ | 256 | 512
Training set size | 64 | $64$ | 128 | 1500
Batch size | 4 | $4$ | 32 | 1500
Learning rate 1 | 1E-3 | 1E-3 | 1E-3 | 1e-4
Learning rate 2 | 1E-4 | 1E-4 | 1E-3 | 5e-5
Learning rate 3 | 1E-5 | N/A | 1E-3 | 1e-5
Epochs | 400 | 200 | 1 | 1.5E6
Number of operators, $N_{o}$ | 1 | 1 | 2 | N/A
Table 2: Hyperparameters used for all computational studies.


# An Evidential Path Logic for Multi-Relational Networks

Marko A. Rodriguez
Center for Non-Linear Studies
Los Alamos National Laboratory
Los Alamos, New Mexico 87545
&Joe Geldart
Department of Computer Science
University of Durham
South Road, Durham, DH1 3LE

###### Abstract

> Multi-relational networks are used extensively to structure knowledge.
> Perhaps the most popular instance, due to the widespread adoption of the
> Semantic Web, is the Resource Description Framework (RDF). One of the
> primary purposes of a knowledge network is to reason; that is, to alter the
> topology of the network according to an algorithm that uses the existing
> topological structure as its input. There exist many such reasoning
> algorithms. With respect to the Semantic Web, the bivalent, monotonic
> reasoners of the RDF Schema (RDFS) and the Web Ontology Language (OWL) are
> the most prevalent. However, nothing prevents other forms of reasoning from
> existing in the Semantic Web. This article presents a non-bivalent, non-
> monotonic, evidential logic and reasoner that is an algebraic ring over a
> multi-relational network equipped with two binary operations that can be
> composed to execute various forms of inference. Given its multi-relational
> grounding, it is possible to use the presented evidential framework as
> another method for structuring knowledge and reasoning in the Semantic Web.
> The benefits of this framework are that it works with arbitrary, partial,
> and contradictory knowledge while, at the same time, it supports a tractable
> approximate reasoning process.

Knowledge structures are used to represent facts about the world. The most
common formal data structure to represent knowledge is the network. With
respect to symbolic knowledge representation, the multi-relational network
(also known as an edge labeled graph or semantic network) is widely used. A
multi-relational network is composed of a set of vertices and a family of edge
sets, where each edge set has a different nominal, or categorical, label.
Formally, a multi-relational network can be represented as $M=(V,\mathbb{E})$,
where $V$ is the set of vertices and
$\mathbb{E}=\\{E_{0},E_{1},\dots,E_{m}\subseteq(V\times V)\\}$ is the family
of directed edge sets. In recent years, perhaps the most popular instance of a
multi-relational data structure for knowledge representation is the Resource
Description Framework (?) of the Semantic Web initiative (?)111Other formal
models of RDF include a bipartite graph (?) and hypergraph (?)
representation.. An edge in an RDF network is called a statement, or triple,
as it is composed of a subject, predicate, and object. For example, suppose
the statement $(i,k,j)$. This statement denotes that $i$ is related to $j$ by
means of an $k$-type relationship. Given the previous definition of $M$, this
is equivalent to the directed edge $(i,j)\in E_{k}$. A particular instance of
a statement is $({\texttt{marko}},{\texttt{coauthor}},{\texttt{joe}})$. This
statement denotes that Marko has a coauthorship relationship to Joe. Languages
such as the RDF Schema (RDFS) and the Web Ontology Language (OWL) impose a set
of constructs that serve to structure knowledge in a particular manner. The
particularities of such a structure are used by an RDFS or OWL reasoner to
infer new statements that can be added to the RDF network. The statements
inferred by such reasoners are bivalent in that they are either true or false
and moreover, their truth value is monotonic as it does not change once it has
been asserted.

While RDFS and OWL are common languages in the Semantic Web, the flexibility
of RDF can easily support other knowledge structures and reasoning algorithms.
The purpose of this article is to present a non-bivalent, non-monotonic,
evidential logic and reasoner for multi-relational networks that leverages
many of the ideas from Non-Axiomatic Logic (NAL) (?) and the Non-Axiomatic
Reasoning System (NARS) (?). The philosophical foundation of an evidential
logic is that no statement is inherently true or false and that a statement
only maintains levels of evidence to support or negate its claim. The notion
of

> experience-grounded semantics [is where] the truth value of a judgment
> indicates the degree to which the judgment is supported by the system’s
> experience. Defined in this way, truth value is system-dependent and time-
> dependent. Different systems may have conflicting opinions, due to their
> different experiences. (?)

The typical metaphor in an evidential logic system is that of an agent that
perceives the world, represents its perceptions in an internal knowledge
structure, and reasons on that structure to infer new knowledge (?). Moreover,
it is assumed that this agent has limited computational resources in terms of
both space and time and thus, does not maintain an objective knowledge
structure nor does it necessarily have the ability to reason across its entire
subjective knowledge structure. In other words, the agent has only so much
information that it can store and process at any one time. This notion is
known as the Assumption of Insufficient Knowledge and Insufficient Resources
(AIKIR). Non-axiomatic logic is contrasted to axiomatic logic, where truth is
bivalent, is defined independent of the time and space requirements necessary
to derive it, can be reasoned from a finite set of premises, and where all
reasoning produces true, immutable conclusions.

The evidential logic presented in this article forms an algebraic ring over a
multi-relational network (i.e. the knowledge structure) equipped with two
binary operations (i.e. the atoms of the inferencing algorithms). Given the
logic’s multi-relational formulation, it is possible to comfortably represent
this structure in RDF and thus, on the Semantic Web. The primary contribution
of this article is the application of evidential logics to multi-relational
networks and the formulation of an algebraic evidential reasoner.

## Evidence in an Inheritance Network

With evidential logics, there does not exist an objective boolean truth value
for every question that can be asked of the world as the world is not reasoned
on directly (?). What is reasoned on is the agent’s internal knowledge
structure. For the agent, knowledge is gained as new evidence from the world
is discovered (either through direct perception or through communication) or
as knowledge is inferred given the agent’s internal reasoning system. The Non-
Axiomatic Reasoning System (NARS) is an example of an evidential reasoning
system (?). The data structure proposed for NARS version 2.2 is a directed
evidence network denoted $G=(V,E,\lambda)$, where $V$ is a set of vertices,
$E\subseteq(V\times V)$ is a set of directed “inheritance” edges, and
$\lambda:E\rightarrow\langle[0,1],[0,1]\rangle$ maps each edge to an evidence
tuple. For example, an edge is denoted

$(i,\langle w^{+},w^{-}\rangle,j),$

where $i$ is the tail of the inheritance edge, $j$ is the head of the
inheritance edge, and $\langle w^{+},w^{-}\rangle$ is the evidence tuple for
that edge. Moreover, $\langle w^{+},w^{-}\rangle\equiv\lambda(i,j)$. The
meaning of $(i,\langle w^{+},w^{-}\rangle,j)$ is that there is $w^{+}$
positive evidence supporting the claim that $i$ isA $j$ and $w^{-}$ negative
evidence that $i$ isA $j$. Another interpretation of this edge is that, upon
examination, it appears, to the agent, that $i$ has $w^{+}$ properties in
common with $j$ and $w^{-}$ properties not in common with $j$. This idea is
also expressed as $i$ being the intension of $j$ and $j$ being the extension
of $i$ (?).

In NARS, the evidence tuple of an edge is revised by means of external
perception or internal reasoning. The act of perceiving $i$ and $j$ in the
external world will return evidence of their relationship and thus, increment
$w^{+}$ or $w^{-}$ accordingly. With respect to internal reasoning, there are
four syllogisms that can be applied to the agent’s internal knowledge network:
deduction, induction, abduction (?), and exemplification. For deduction,

$(i,\langle w^{+}_{1},w^{-}_{1}\rangle,j),(j,\langle
w^{+}_{2},w^{-}_{2}\rangle,k)\rightarrow(i,\langle
w^{+}_{3},w^{-}_{3}\rangle,k).$

For induction,

$(i,\langle w^{+}_{1},w^{-}_{1}\rangle,j),(i,\langle
w^{+}_{2},w^{-}_{2}\rangle,k)\rightarrow(j,\langle
w^{+}_{3},w^{-}_{3}\rangle,k).$

For abduction,

$(i,\langle w^{+}_{1},w^{-}_{1}\rangle,j),(k,\langle
w^{+}_{2},w^{-}_{2}\rangle,j)\rightarrow(i,\langle
w^{+}_{3},w^{-}_{3}\rangle,k).$

Finally, there is a less widely used fourth syllogism known as exemplification
(?; ?). Exemplification is defined as

$(i,\langle w^{+}_{1},w^{-}_{1}\rangle,j),(j,\langle
w^{+}_{2},w^{-}_{2}\rangle,k)\rightarrow(k,\langle
w^{+}_{3},w^{-}_{3}\rangle,i).$

The values for $\langle w^{+}_{3},w^{-}_{3}\rangle$ depend upon the specific
inference rules of the the evidential reasoner. In (?), it is explicitly
stated that the rules presented are not set in stone, but rather subject to
revision themselves as more is understood about the design of evidential
systems.

The evidence tuple $\langle w^{+},w^{-}\rangle$ of an edge can be transformed
into a normalized “truth value” consisting of a new tuple $\langle
f,c\rangle\in\langle\emptyset\cup[0,1],\emptyset\cup[0,1]\rangle$. The first
component $f$ is the frequency of positive evidence and is defined as

$f=\frac{w^{+}}{w^{+}+w^{-}}.$

The second component $c$ is the confidence in the stability of $f$ as $k$-more
observations are made and is defined as

$c=\frac{w^{+}+w^{-}}{(w^{+}+w^{-}+k)}.$

The parameter $k\in\mathbb{R}_{0}^{+}$ is a user-defined, system constant. In
words, as more positive evidence accumulates relative to negative evidence,
$f$ increases towards $1$. As more total evidence accumulates relative to some
constant $k$, $c$ increases towards $1$. If there is no evidence, then the
edge has an $f$-component of $\emptyset$ which means that the relationship is
unknown. Finally, hard “truth” can be modeled with an evidence tuple of
$\langle 1,0\rangle$ with $k=0$ and thus, an $fc$-tuple of $\langle
1,1\rangle$.

The contribution of this article is to extend the aforementioned evidential
logic framework to multi-relational networks composed of both inheritance and
non-inheritance edges. Moreover, this article contributes an algebraic ring
formulation of evidential reasoning which situates the reasoner within well
understood mathematics. From this multi-relational foundation, evidential
reasoning can be comfortably executed in the RDF-rich world of the Semantic
Web.

## Evidential Reasoning using Path Expressions

A path algebra to map a multi-relational network to a single-relational form
was originally presented in (?). The motivation behind the algebra was to
provide a formal means by which the large class of single-relational network
analysis algorithms could be applied to multi-relational networks in a
meaningful way. With respect to this article, the binary operations of $+$ and
$\cdot$ are updated so as to work with evidence tuples.

A multi-relational, evidence network is defined as
$M=(V,\mathbb{E}=\\{E_{0},E_{1},\ldots,E_{m}\subseteq(V\times V)\\},\lambda)$,
where $V$ is a set of vertices, $\mathbb{E}$ is a family of edge sets, and
$\lambda:E_{k}\rightarrow\langle[0,1],[0,1]\rangle$ maps each edge in
$E_{k}:k\leq m$ to an evidence tuple. The algebraic formulation of the
presented evidential path algebra operates on an $n\times n\times m$ tensor
representation of this network (?)222The tensor representation can also be
thought of as a set of $m$ adjacency matrix “slices”, where each matrix has
its own label.. The evidence tensor $\mathcal{A}$ is defined as

$\mathcal{A}^{k}_{i,j}=\begin{cases}\lambda(i,j)&\text{if }(i,j)\in E_{k}\\\
\langle 0,0\rangle&\text{otherwise}.\end{cases}$

The two $n$ dimensions represent the vertices and the single $m$ dimension
represents the various edge labels. Thus, there is a one-to-one mapping
between a multi-relational, evidence network and an evidence tensor. The
entries of the tensor denote the amount of positive ($w^{+}$) and negative
($w^{-}$) evidence for the edge $(i,j)\in E_{k}$, where $\langle 0,0\rangle$
denotes no evidence when no such edge exists. Inference on this tensor can be
accomplished through the two binary operations

$+:\langle[0,1],[0,1]\rangle\times\langle[0,1],[0,1]\rangle\rightarrow\langle[0,1],[0,1]\rangle$

and

$\cdot:\langle[0,1],[0,1]\rangle\times\langle[0,1],[0,1]\rangle\rightarrow\langle[0,1],[0,1]\rangle.$

The function rules of these operations are

$\langle w_{1}^{+},w_{1}^{-}\rangle+\langle
w_{2}^{+},w_{2}^{-}\rangle=\langle(w_{1}^{+}+w_{2}^{+}),(w_{1}^{-}+w_{2}^{-})\rangle$

and

$\langle w_{1}^{+},w_{1}^{-}\rangle\cdot\langle
w_{2}^{+},w_{2}^{-}\rangle=\langle(w_{1}^{+}\cdot w_{2}^{+}),(w_{1}^{+}\cdot
w_{2}^{-}+w_{1}^{-}\cdot w_{2}^{+}+w_{1}^{-}\cdot w_{2}^{-})\rangle.$

These two operations form an algebraic ring over the evidence tensor (i.e.
multi-relational, evidence network). The binary operation $+$ is associative,
has an identity of $\langle 0,0\rangle$, and is commutative. The binary
operation $\cdot$ is associative with an identity of $\langle 1,0\rangle$. The
operation $+$ supports the notion that evidence (from two independent
experiences/inferences) can be summed together (?). The operation $\cdot$
supports the notion that positive evidence can be multiplied to form new
positive evidence and that conflicting and negative evidence accounts for
negative evidence.

The next subsections will formalize the various syllogisms on inheritance
edges in a multi-relational network and then the following subsection will
discuss the application of these operations to any arbitrary path through an
evidence tensor. Note that what is presented is a set of operations that will
operate on $n\times n$ matrix “slices” of the evidence tensor. In this
respect, the presented operations are “global” computations and thus, are
computationally inefficient and contrary to AIKIR. However, these operations
can be implemented as “local” computations using various methods such grammar
walks (?) in specific areas of the network because evidential reasoning does
not require a network-wide computation.

### Inferring Inheritance Evidence

Inheritance relations are handled as a special case situaion when reasoning in
a multi-relational network. Example inheritance predicates in an RDF network
are rdf:type and rdfs:subClassOf. Figure 1 presents a simple example network
that will be used to demonstrate the inference rules of deduction, induction,
abduction, and exemplification.

Figure 1:  An an inheritance network.

#### Deductive Inheritance

Deduction is defined as a two step “walk” on the inheritance component of an
evidential network. A two step walk can be computed by squaring a matrix.
Suppose a standard square $\\{0,1\\}$-matrix denoted $\mathbf{A}$, where
$\mathbf{A}_{i,j}=1$ if there is an edge between vertex $i$ and $j$, and $0$
otherwise. The $2^{\text{nd}}$ power of this matrix, as defined by ordinary
matrix multiplication, $\mathbf{A}\mathbf{A}$, will yield a new matrix where
entry $(\mathbf{A}\mathbf{A})_{i,j}$ denotes the total number of paths of
length $2$ starting from vertex $i$ and ending on vertex $j$ (?). With respect
to an evidence tensor, determining the product of two adjacency matrix
“slices”, will yield a new adjacency matrix where the entry $(i,j)$ denotes
the total amount of deductive evidence supporting $(i,j)$.

Evidential matrix multiplication is defined as ordinary matrix multiplication,
but respective of the rules of $\cdot$ and $+$. Thus,

$\left(\mathcal{A}^{k}\mathcal{A}^{k^{\prime}}\right)_{i,j}=\sum_{l\in
V}\mathcal{A}_{i,l}^{k}\cdot\mathcal{A}_{l,j}^{k^{\prime}}:k,k^{\prime}\leq
m.$

In the degenerate case where all positive evidence is $1$ and all negative
evidence is $0$, such that

$\mathcal{A}^{k}_{i,j}=\begin{cases}\langle 1,0\rangle&\text{if }(i,j)\in
E_{k}\\\ \langle 0,0\rangle&\text{otherwise},\end{cases}$

evidential matrix multiplication will set $w^{+}$ to the total number of paths
from vertex $i$ to vertex $j$ and $0$ to $w^{-}$. In this form, the evidential
path algebra yields results that are isomorphic to the original formulation of
the path algebra in (?).

In Figure 2, deduction, as defined by
$\mathcal{A}^{{\texttt{isA}}}\mathcal{A}^{{\texttt{isA}}}$, infers four new
edges. Note that the evidence tuples are not presented in order to preserve
diagram clarity.

Figure 2:  Deduction in an inheritance network.

#### Inductive Inheritance

Induction is the process of generalization given instances. In order to
compute induction in an inheritance region of an evidence tensor, it is
important to take the converse transpose of an adjacency matrix “slice”. The
operation of taking a statement like “a scholar is a writer” and reversing it
to derive the statement “a writer is a scholar” is known as taking the
converse of the statement. With respect to determining the evidence for the
converse of a statement, all positive evidence for a “scholar is a writer” is
considered positive evidence that a “writer is a scholar”. However, all
negative evidence for “a scholar is a writer” should not be considered
negative evidence for a “writer is a scholar” as there is no evidence in the
original statement for writers not being scholars. Such statement converses
can be expressed using the evidential algebra. For standard matrices, the
transpose of a matrix is defined as
$(\mathbf{A}^{\top})_{i,j}=\mathbf{A}_{j,i}$ and denotes reversing the
direction of an edge (i.e. taking the converse of a statement). However, for
evidential edges, the converse transpose of an evidential matrix is defined as

$\hat{\mathcal{A}}^{k}_{i,j}=\langle\gamma^{+}(\mathcal{A}^{k}_{j,i}),0\rangle,$

where $\gamma^{+}:\langle[0,1],[0,1]\rangle\rightarrow[0,1]$ maps an evidence
tuple to its first component (i.e. $w^{+}$ positive evidence). This operation
ensures that the converse of an evidence tuple maintains no negative evidence.

In Figure 3, induction, as defined by
$\mathcal{A}^{{\texttt{isA}}}\hat{\mathcal{A}}^{{\texttt{isA}}}$, infers two
new isA edges between journalist and scholar. As stated previous, and to
stress the point, the negative evidence for these two new evidence tuples is
$0$.

Figure 3:  Induction in an inheritance network.

#### Abductive Inheritance

Abduction is the reverse of induction. In Figure 4, abduction, as defined by
$\hat{\mathcal{A}}^{{\texttt{isA}}}\mathcal{A}^{{\texttt{isA}}}$, infers two
new isA edges between person and author. Similar to induction, a converse
transpose will generate no negative evidence for these two new evidence
tuples.

Figure 4:  Abduction in an inheritance network.

#### Exemplative Inheritance

Exemplary inheritance paths can be determined by the multiplication of two
converse transpose matrices. In Figure 5, exemplification, as defined by
$\hat{\mathcal{A}}^{{\texttt{isA}}}\hat{\mathcal{A}}^{{\texttt{isA}}}$, infers
four new evidence tuples.

Figure 5:  Exemplification in an inheritance network.

This subsection presented the syllogisms of deduction, induction, abduction,
and exemplification and their use in an inheritance region of an evidence
tensor. Note that this region may account for more than a single $m$-dimension
as many labels can have an similar meaning to isA (e.g. similarTo,
equivalentTo, implies, etc.). The next section will discuss reasoning using
arbitrary paths through a multi-relational evidence network and thus, for
those paths that may not necessarily contain isA edges.

### Inferring Non-Inheritance Evidence

A multi-relational network may be composed of various types of relationships.
Figure 6 diagrams an example multi-relational network that will be referred to
in the examples of this subsection333There is nothing that prevents the
network in Figure 1 to be merged with the network in Figure 6 (e.g. marko and
joe are both scholars). However, for diagram clarity, this is not
represented..

Figure 6:  A multi-relational knowledge network.

The network in Figure 6 is composed of a reference to this article, the
denoted authors of this article, and two citations from this article to other
articles. In a bivalent logic, these statements are true because they exist.
However, in scholarly publishing it is rare, nearly impossible, for two people
to “equally” write an article together. While ideas are shared and drafts are
written, read, and edited, the article’s final form is always a biased
reflection of the approach of some authors over others. With respect to the
statements diagrammed in Figure 6, what is the evidence for these statements?
The following descriptions are provided to expose, for each edge presented
above, how much supporting or detracting evidence there is for $i$’s $m$-type
relationship to $j$:
$({\texttt{marko}},{\texttt{wrote}},{\texttt{this\\_article}})$:

  * •

$w^{+}$: notation, writing style, diagram style, american spelling

  * •

$w^{-}$: logic, reasoning, citations, philosophy

$({\texttt{marko}},{\texttt{wrote}},{\texttt{path\\_article}})$:

  * •

$w^{+}$: notation, writing style

  * •

$w^{-}$: algebra, no diagrams

$({\texttt{joe}},{\texttt{wrote}},{\texttt{this\\_article}})$:

  * •

$w^{+}$: logic, reasoning, citations, philosophy, rings

  * •

$w^{-}:$ notation, writing style, diagram style, american spelling

$({\texttt{this\\_article}},{\texttt{cites}},{\texttt{path\\_article}})$:

  * •

$w^{+}$: (?) citation, rings

  * •

$w^{-}$: graph notation, philosophy, only a single algebra citation

$({\texttt{this\\_article}},{\texttt{cites}},{\texttt{nars\\_article}})$:

  * •

$w^{+}$: (?) citation, evidential notation, syllogisms

  * •

$w^{-}$: path expressions, semantic web, rdf, owl, rdfs

The presented positive and negative evidence “metadata” (e.g. writing style,
american spelling, citation patterns, etc.) can be represented in a multi-
relational network. From this multi-relational encoding, it is possible,
through automated means, to infer new evidence or revise existing evidence in
the network with prescribed path expressions. In other words, a region of the
network can provide further supporting and/or detracting evidence for another
region of the network. In order to demonstrate two examples of this, the
multi-relational network in Figure 6 will be used in this section to

  1. 1.

infer new independent evidence supporting the claims that marko and joe wrote
this_article according to the notion that self-citations are positive evidence
supporting authorship, and

  2. 2.

infer new evidence that marko and joe have a coauthorship edge between them.

#### Self-Citation Paths

Suppose that the article citation evidence in Figure 6 was experienced by the
agent (e.g. a repository feed) and added to its existing internal network. At
the point of insertion, it is possible to revise the evidence tuple for wrote
based upon the idea that self-citations in an article are considered evidence
for authorship of that article (?). Furthermore, in order to ensure
independent evidence, assume that the current evidence for wrote in Figure 6
was not determined using self-citation information444Refer to (?) for the
definition and importance of independent evidence in evidential logics.. Given
this scenario, the following inference rule will update all wrote evidence
according to inferred self-citation evidence:

$\mathcal{A}^{{\texttt{wrote}}}_{(t+1)}=\left(\left(c(\mathcal{A}^{{\texttt{wrote}}}_{(t)})\mathcal{A}^{{\texttt{cites}}}_{(t)}{\mathcal{A}^{{\texttt{wrote}}}_{(t)}}^{\top}\right)\circ\mathbf{I}\right)+\mathcal{A}^{{\texttt{wrote}}}_{(t)},$

where $t\in\mathbb{N}^{+}_{0}$ is the current time step,
$c(\mathcal{A}^{{\texttt{wrote}}})$ “clips” the evidence in
$\mathcal{A}^{{\texttt{wrote}}}$, $\circ$ is the entry-wise Hadamard
multiplication operation555For review, Hadamard entry-wise multiplication is
defined as
$\mathbf{A}\circ\mathbf{B}=\left[\begin{array}[]{ccc}\mathbf{A}_{1,1}\cdot\mathbf{B}_{1,1}&\cdots&\mathbf{A}_{1,j}\cdot\mathbf{B}_{1,j}\\\
\vdots&\ddots&\vdots\\\
\mathbf{A}_{i,1}\cdot\mathbf{B}_{i,1}&\cdots&\mathbf{A}_{i,j}\cdot\mathbf{B}_{i,j}\\\
\end{array}\right].$ , and $\mathbf{I}$ is the evidential identity matrix

$\mathbf{I}_{i,j}=\begin{cases}\langle 1,0\rangle&\text{if }i=j\\\ \langle
0,0\rangle&\text{otherwise}.\end{cases}$

In words, the self-citation inference rule states that evidence for
$\mathcal{A}^{{\texttt{wrote}}}$ can be modulated by the total evidence for
the path that goes from an author, to their written articles, to the articles
that those articles cite, and then finally, to the authors of those cited
articles. However, in order to ensure that those cited authors are the
original author from the start of the path (i.e. self-citation), it is
important to filter on the identity matrix $\mathbf{I}$. Hadamard entry-wise
multiplication is used to apply a matrix filter to a path. Note that the
transpose of an evidence matrix, not the converse transpose of the evidence
matrix, is used when taking the converse of a non-inheritance statement. The
reason for this is that the positive and negative evidence for the statement
“marko wrote this article” is the same for “this article was written by
marko”. Next, inferred evidence for wrote must be independent of the evidence
used to calculate it. Thus, $\mathcal{A}^{{\texttt{wrote}}}$ is mapped to a
$(\langle 1,0\rangle,\langle 0,0\rangle)$-matrix using the clip $c$ operation,
where

$c(\mathcal{A}^{k})_{i,j}=\begin{cases}\langle 1,0\rangle&\text{if
}\mathcal{A}^{k}_{i,j}\neq\langle 0,0\rangle\\\ \langle
0,0\rangle&\text{otherwise}.\end{cases}$

Finally, the total evidence for the self-citation path is summed with the
current evidence for the wrote edge. Thus, the agent has used self-citations
as further, revising evidence for wrote.

#### Coauthorship Paths

Two people are considered coauthors if they have both written an article
together. The evidence for coauthorship is determined by the total evidence
across all their jointly written articles. In its algebraic form, the evidence
for coauthor can be determined by

$\mathcal{A}^{{\texttt{coauthor}}}_{(t+1)}=\left(\left(\mathcal{A}^{{\texttt{wrote}}}_{(t)}{\mathcal{A}^{{\texttt{wrote}}}_{(t)}}^{\top}\right)\circ
n(\mathbf{I})\right)+\mathcal{A}^{{\texttt{coauthor}}}_{(t)},$

where $n(\mathbf{I})$ “nots” the evidential identity matrix such that such
that every $\langle 1,0\rangle$ is a $\langle 0,0\rangle$ and every $\langle
0,0\rangle$ is a $\langle 1,0\rangle$. The reason for the $n(\mathbf{I})$
filter is that to represent a coauthor path from a person to their authored
papers and then to other authors of those papers, the path must exclude the
original author as an author is not a coauthor of themselves. In other words,
it must filter out the identity evidence matrix. The two inferred coauthor
edges between marko and joe are diagrammed in Figure 7.

Figure 7:  Coauthoring in an inheritance network.

The article (?) provides an in-depth review of different inferences that can
be made with arbitrary paths, various filters, and how the theorems of the
general path algebra can be applied to derive equivalent, yet more
computationally efficient paths. The examples presented in (?) can be applied
to an evidence tensor as long as the definitions of $+$ and $\cdot$, as
defined in this article, are respected.

## Conclusion

Reasoning with axiomatic logics is computationally expensive (?; ?). With
respect to the Semantic Web, and with the integration capabilities brought
forth by the Linked Data initiative, such reasoning is intractable. The
original assumption driving the development of NARS is the Assumption of
Insufficient Knowledge and Insufficient Resources (AIKIR) (?). Given space and
time constraints, an agent cannot reason over the entire Semantic Web, and
potentially, not even over its entire internal knowledge network. The benefit
of the inheritance-based syllogisms are that they do not require a global
analysis of the knowledge network, can be executed independently of each
other, and at the their core, are very simple and computationally efficient.
With respect to inferencing with arbitrary path expressions, the efficiency is
dependent on the length of the path and the number of applied filters. It is
important to note that the matrix formalism presented in this article is very
much intractable as the best known algorithm to compute ordinary matrix
multiplication is approximately $O(|V|^{2.807})$ (?). As stated previously,
this matrix model can be approximated using various techniques such as grammar
walk algorithms which do not compute the inferences over the entire network,
but instead, on local subgraphs (i.e. paths starting from particular vertices)
(?). With evidential logic, such walks can be executed when resources are
available and only in those areas of the knowledge network where it is deemed
necessary (e.g. $f\sim 0.5$ and/or low $c$ areas).

Finally, to actually represent a multi-relational, evidence network in RDF and
on the Semantic Web, some form of reification can be used. A popular technique
is the quad-form of a “triple” where a statement maintains a fourth component
known as a named graph (?). With reification, statements can be attached to
statements and thus, a $\langle w^{+},w^{-}\rangle$ evidence tuple can be
assigned to an RDF statement.

This article presented a non-axiomatic evidential logic that can be
implemented within the constructs of RDF and thus, can be used as an
evidential reasoning system in the Semantic Web. The benefit of this system is
that it works with arbitrary, partial, and contradictory knowledge while, at
the same time, in a non-matrix implementation, supports a tractable
approximate reasoning process.

## Acknowledgements

Vadas Gintautas provided useful insight during the development of these ideas.
This work was funded by a Director’s Fellowship from the Los Alamos National
Laboratory, EPSRC grant EP/D504376/1, and British Telecom plc.

## References

  * [Berners-Lee, Hendler, and Lassila 2001] Berners-Lee, T.; Hendler, J. A.; and Lassila, O.  2001\.  The Semantic Web.  Scientific American 34–43.
  * [Bochenski 1970] Bochenski, I. M.  1970\.  A History of Formal Logic.  New York, USA: Chelsea Publishing Company.
  * [Carroll et al. 2005] Carroll, J. J.; Bizer, C.; Hayes, P.; and Stickler, P.  2005\.  Named graphs, provenance and trust.  In The Fourteenth International World Wide Web Conference (WWW05), 613–622.  Chiba, Japan: ACM Press.
  * [Chartrand 1977] Chartrand, G.  1977\.  Introductory Graph Theory.  Dover.
  * [Dau 2006] Dau, F.  2006\.  RDF as graph-based, diagrammatic logic.  Lecture Notes in Computer Science 4203\.
  * [Donini 2002] Donini, F.  2002\.  The Descripton Logic Handbook.  Cambridge University Press.  chapter Complexity of Reasoning, 101–141.
  * [Fensel and van Harmelen 2007] Fensel, D., and van Harmelen, F.  2007\.  Unifying reasoning and search to web scale.  IEEE Internet Computing 11(2):94–96.
  * [Hayes and Gutierrez 2004] Hayes, J., and Gutierrez, C.  2004\.  Bipartite graphs as intermediate model for RDF.  In Proceedings of the International Semantic Web Conference ISWC04, 47–61.
  * [Kolda, Bader, and Kenny 2005] Kolda, T. G.; Bader, B. W.; and Kenny, J. P.  2005\.  Higher-order web link analysis using multilinear algebra.  In Proceedings of the Fifth IEEE International Conference on Data Mining ICDM’05.  IEEE.
  * [Morale and Serodio 2006] Morale, A. A. M., and Serodio, M. E. V.  2006\.  A directed hypergraph model for RDF.  In Simperl, E.; Diederich, J.; and Schreiber, G., eds., Proceedings of the Knowledge Web PhD Symposium (KWEPSY07).
  * [Patzig 1968] Patzig, G.  1968\.  Aristotle’s Theory of The Syllogism.  Boston, Massachusetts: D. Reidel Publishing Company.
  * [Rodriguez and Shinavier 2008] Rodriguez, M. A., and Shinavier, J.  2008\.  Exposing multi-relational networks to single-relational network analysis algorithms.  Technical Report LA-UR-08-03931, Los Alamos National Laboratory.
  * [Rodriguez, Bollen, and Van de Sompel 2009] Rodriguez, M. A.; Bollen, J.; and Van de Sompel, H.  2009\.  Automatic metadata generation using associative networks.  ACM Transactions on Information Systems 27(2).
  * [Rodriguez 2008] Rodriguez, M. A.  2008\.  Grammar-based random walkers in semantic networks.  Knowledge-Based Systems 21(7):727–739.
  * [Strassen 1969] Strassen, V.  1969\.  Gaussian elimination is not optimal.  Numerische Mathematik 13:354–356.
  * [Wang 1993] Wang, P.  1993\.  Non-axiomatic reasoning system (version 2.2).  Technical Report 75, Center for Research on Concepts and Cognition at Indiana University.
  * [Wang 1994] Wang, P.  1994\.  From inheritance relation to non-axiomatic logic.  International Journal of Approximate Reasoning 11:281–319.
  * [Wang 2004a] Wang, P.  2004a.  Cognitive logic versus mathematical logic.  In Proceedings of the Third International Seminar on Logic and Cognition.
  * [Wang 2004b] Wang, P.  2004b.  Experience-grounded semantics: a theory for intelligent systems.  Cognitive Systems Research 6(4):282–302.
  * [Wang 2006] Wang, P.  2006\.  Rigid Flexibility.  Springer.