# Neural Operator: Is data all you need to model the world? An insight into
the impact of Physics Informed Machine Learning

Hrishikesh Viswanath<EMAIL_ADDRESS>Md Ashiqur Rahman  Abhijeet Vyas
Andrey Shor  Beatriz Medeiros  Stephanie Hernandez  Suhas Eswarappa Prameela
Aniket Bera  Department of Computer Science, Purdue University, West
Lafayette, IN, USA  Department of Materials Science and Engineering, MIT,
Cambridge, MA, USA  Department of Aeronautics and Astronautics, MIT,
Cambridge, MA, USA  Hopkins Extreme Materials Institute, Johns Hopkins
University, Baltimore, MD, USA

###### Abstract

Numerical approximations of partial differential equations (PDEs) are
routinely employed to formulate the solution of physics, engineering and
mathematical problems involving functions of several variables, such as the
propagation of heat or sound, fluid flow, elasticity, electrostatics,
electrodynamics, and more. While this has led to solving many complex
phenomena, there are some limitations. Conventional approaches such as Finite
Element Methods (FEMs) and Finite Differential Methods (FDMs) require
considerable time and are computationally expensive. In contrast, data driven
machine learning-based methods such as neural networks provide a faster,
fairly accurate alternative, and have certain advantages such as
discretization invariance and resolution invariance. This article aims to
provide a comprehensive insight into how data-driven approaches can complement
conventional techniques to solve engineering and physics problems, while also
noting some of the major pitfalls of machine learning-based approaches.
Furthermore, we highlight, a novel and fast machine learning-based approach
($\sim$1000x) to learning the solution operator of a PDE operator learning. We
will note how these new computational approaches can bring immense advantages
in tackling many problems in fundamental and applied physics.

###### keywords:

Machine learning, Neural networks, Neural operators, Fourier neural operator,
Geo-FNO, Graph neural operator, Physics informed neural operator, Finite
element method, Finite volume method, Finite difference method, DeepONet,
Spectral neural operator, Adaptive Fourier neural operator, Burgers equation,
Darcy Flow equation, Navier Stokes equation, Kolmogorov Flow

††journal: Communications Physics

##  1 Introduction

Partial differential equations (PDEs) are an integral tool in mathematically
modeling the physical world. They allow one to describe how a quantity changes
with respect to multiple variables and have allowed physicists to model
various phenomena in fluid flow, electrodynamics, and quantum mechanics. An
example family of generic PDEs can be represented as shown in equation 1,

$\displaystyle(L_{a}u)(x)=f(x),~{}~{}~{}x\in D,$ (1) $\displaystyle
u(x)=0,~{}~{}~{}x\in\delta D$

for some $a\in A,f\in L$, where $A,L$ are Banach spaces, $D$ is the domain of
the PDE and $u:D\rightarrow\mathbf{R},u\in U$ is the solution function. While
PDEs are all around us, it is oftentimes very difficult for one to solve them
analytically. The best that one can achieve is an approximation of the true
solution of the PDE. The most popular approaches to solving PDEs are numerical
methods such as finite difference methods (FDMs) Godunov & Bohachevsky [1959],
finite element methods (FEMs) Zienkiewicz et al. [2005], and finite volume
methods (FVMs) Eymard et al. [2000] as they are able to approximate solutions
to PDEs with high amounts of accuracy. However, they are computationally
expensive.

Finite difference methods solve PDEs by converting them into linear algebraic
equations called finite difference equations. These equations are obtained by
discretizing the domain of the functions involved in the PDEs and representing
the derivatives as differences according to the first principles of calculus
Jordan & Jordán [1965]. The different discretization schemes result in
different methods tailored to specific applications such as gas-dynamics Sod
[1978] and heat transfer Özişik et al. [2017]. An exhaustive study of applying
this method to various families of PDEs has been published by Smith et al.
[1985]. Finite element methods, on the other hand, generate algebraic
equations by applying fundamental physics laws on static or moving quantum of
a system Hughes [2012] and have been used to solve several different problems
in physics such as the fluid flow problem Donea & Huerta [2003] and strain
localization in metals Roters et al. [2011].

Numerical methods have traditionally been used to solve PDEs, but in an effort
to reduce the computational cost and obtain the solutions more quickly,
researchers are exploring data-driven approaches to approximate the solutions
to PDE. These methods come under the overarching paradigm of machine learning
approaches, which utilize data-driven algorithms that allow a program to learn
and improve from experience. The modern face of machine learning is neural
networks, which have led to a variety of advances in many scientific
endeavors. Recent advances in deep learning, a subfield of machine learning
which studies algorithms known as artificial neural networks, have allowed
researchers to develop neural network architectures capable of approximating
the solution of PDEs through large amounts of data. Deep neural networks that
can approximate any function to an arbitrary precision Cybenko [1989] consist
of multiple hidden layers and serve as a mapping between inputs and outputs
LeCun et al. [2015]. Deep neural networks have been applied to a multitude of
problems in material physics, thermodynamics, and fluid dynamics problems Cai
et al. [2022], Mao et al. [2020], Cai et al. [2021b], Thais et al. [2022] due
to their innate ability to learn complex relationships between physical
entities. To solve PDEs using neural networks, a dataset of inputs
$\\{x_{i}\\}_{1}^{n}$ and their solution mappings $\\{u(x_{i})\\}_{1}^{n}$ is
used to train or learn an approximate function map $u^{+}:D\rightarrow R$ of
the solution function $u$, where $R$ is the range of $u$. The accuracy of the
predictions of $u$ depends on the complexity of the function class $u^{+}$,
which in turn depends on the neural network architecture. The ability of the
network to predict $u(x)$ for $x\notin\\{x_{i}\\}_{1}^{n}$ is known as
generalization and is vital to solving PDEs accurately. In particular,
physics-informed neural networks (PINNs) Raissi et al. [2019] and Deep-ONet Lu
et al. [2019] architectures have shown great success in learning PDEs. A
recently proposed neural network-based learning technique called operator
learning Li et al. [2020a] has shown promise in approximating the PDE operator
through learning infinite-dimensional function spaces.

While neural networks are able to approximate any function, which is a map
between finite-dimensional spaces Cybenko [1989], to approximate an operator,
which is a map between infinite dimensional spaces, a network of infinite
length is required Guss & Salakhutdinov [2019]. The need for more accurate
solutions has motivated the development of neural operator learning technique:
a generalization of neural network architectures to map between infinite
dimensional spaces Kovachki et al. [2021]. Neural operators are neural
networks, composed of linear integral operators and nonlinear activation
functions. The result is an operator capable of approximating highly nonlinear
solution operators of PDEs. Furthermore, neural networks trained as neural
operators are resolution-invariant and up to $\approx$1000x faster than
typical neural networks in approximating the solution of PDEs Li et al.
[2020a]. Current state-of-the-art neural operator architecture includes the
graph neural operator (GNO) Li et al. [2020c], Fourier neural operator (FNO)
Li et al. [2020a] and it’s variant geo-Fourier neural operator (Geo-FNO) Li et
al. [2022], and physics informed neural operator (PINO) Rosofsky & Huerta
[2022]. While these neural network based operator models are faster, they have
some limitations, such as the need for a comprehensive dataset to prevent
overfitting. Moreover, noisy datasets can prevent the models from converging
to the correct operator. Furthermore, the multi-layer configuration of these
neural networks makes the models less interpretable. They are ultimately black
box models.

This paper aims to provide a brief overview of popular numerical methods and
elaborate on several machine learning-based methods to numerically approximate
PDEs. Figure 1 provides an overview of both conventional and machine learning-
based approaches to solving PDEs. We will note how these new computational
approaches can bring immense advantages in tackling many problems in
fundamental and applied physics. Sections 2 and 3 describe some of the
conventional and neural network-based architectures, while sections 4 through
8 talk about different types of neural operator models. Section 9 discusses
various fields where neural operators have been successful. Tables 7 and 6, in
the appendix section, provide the definitions of the mathematical notations
and the GitHub links to neural operator models respectively.

##  2 Conventional Solvers

In this section, we will discuss some of the conventional approaches that have
been used to solve PDEs, such as finite difference methods, finite element
methods and finite volume methods.

###  2.1 Finite difference method

The finite difference method (FDM) is a class of grid-point techniques in
numerical methods used to approximate the solution to differential equations.
FDMs work by discretizing a function in an infinite domain into a finite,
space-time grid Moczo et al. [2004]. Each point in the grid represents a value
of the particular function being approximated. FDMs are able to approximate
the solution to a PDE at each grid-point by approximating the derivatives with
finite difference formulas Godunov & Bohachevsky [1959]. The more grid-points
available, the more accurate the solution.

FDMs have been used to help understand earthquake physics Aochi et al. [2013],
conservation laws Sod [1978], and fluid flow simulations Fadlun et al. [2000].
Due to its simplicity, the FDM is very fast at solving PDEs on structured
grids. However, FDM struggles with complex geometries and is often less
accurate than finite element and finite volume methods.

###  2.2 Finite element method

The finite element method (FEM) is a numerical method for solving PDEs in
which a complex domain is discretized into a collection of small, simple
domains. These simple domains are referred to as finite elements and are used
to construct approximation functions over each element Reddy [2019]. The FEM
creates a sparse matrix representing the discretization and solves it via a
sparse matrix solver to achieve a global solution. The FEM allows one to solve
complex PDEs since the method is adaptable over difficult domains.
Furthermore, the FEM offers very accurate results dependent on discretization.
The disadvantages of the FEM are that it is computationally expensive and that
the accuracy of the solution depends on the resolution of the mesh. The FEM is
utilized heavily in problems involving complex geometries and for modeling
heat transfer Wilson et al. [1974], electromagnetic potential Li et al.
[2017], and quantum mechanics Searles & von Nagy-Felsobuki [1988].

###  2.3 Finite volume method

Figure 1: The above figure represents the logical flow of how PDEs can be
solved using various methods, highlighting existing ML techniques and various
families of neural operator-based techniques

Finite volume methods (FVM) are capable of evaluating PDEs numerically through
the use of algebraic equations. FVMs utilize a volume integral formation of
the PDE and discretize the geometry of the PDE through the use of a finite
collection of volumes. Then, FVMs evaluate each volume integral at each
volume. FVMs are often used to approximate solutions to physical problems that
arise from conservation laws. These include computational fluid dynamics
Acharya et al. [2007], dynamic solid mechanics Slone et al. [2003], and stress
analysis Demirdžić & Muzaferija [1994]. The key advantage of FVMs is the
ability to be used on unstructured grids and complex geometries. The accuracy
of FVM methods depend on various factors such as the order, the complexity of
the PDE and the grid resolution. Some methods, such as the Gudonov method, are
first-order accurate, while methods, such as the MUSCL-Hancock scheme, are
typically second-order accurate. WENO and ENO schemes are accurate up to the
fifth order.

##  3 Data Driven ML-based approaches

In this section, we go over some of the most commonly used neural network-
based architectures for approximating the solution to PDEs, such as fully
connected neural networks and convolutional neural networks. A neural network
is a mesh of layers, where each layer consists of a set of nodes, which
computes the weighted sum of the input and applies a non-linear activation to
it. These networks are also called feed-forward networks because the input
flows forward from the input layer to the output layer through one or more
hidden layers Bebis & Georgiopoulos [1994].

Neural Network-based approaches to solve PDEs are primarily data-driven. That
is, the computational domain is discretized, and the functions are represented
as vectors. The values are computed on a discretized mesh, and the neural
network is trained to learn these function mappings. The training process
involves passing the input vector to the neural network as the input
parameter. The output is designed to have the same dimensions as the ground
truth vector, which is part of the pre-computed training set. The difference
between the predicted (output) and the ground truth vectors, which is termed
as loss, forms the basis for learning through gradient descent techniques.

These techniques have been shown to perform quite well in parametric PDE
approximations and high-dimensional PDEs. Neural Networks allow for solving
the PDE repeatedly but with different initial and boundary conditions. Rather
than solving a single PDE model, these methods learn to map the initial
conditions and the model parameters to the solution, using large training
datasets.

While these techniques offer promising results, in most situations, they do
not outperform higher-order classical numerical methods Jentzen et al. [2023].
Due to their mesh-dependent nature, these methods suffer from the curse of
dimensionality. Operator learning was proposed to overcome this issue by
learning mesh invariant solutions to PDEs Chen et al. [2023]. However, neural
operators, which refer to neural networks that are trained to learn infinite-
dimensional function mappings, work well in interpolation paradigms where the
prediction lies within the support of the training data; however, they may
fail in scenarios where the data is present outside the scope of the training
sets, requiring extrapolation Zhu et al. [2023].

The following subsections discuss various neural network-based architectures
and table 1 highlights the differences between classical solvers and data-
driven approaches

###  3.1 Fully Connected neural networks

Fully connected neural networks (FCNNs) are the most basic neural network
architecture. They consist of a multitude of layers, including an input layer,
one or more hidden layers, and an output layer Yegnanarayana [2009]. These
types of networks are considered ”feed-forward” because the information in
this structure only moves forward, from one layer to the next, until it
reaches the output layer Sazli [2006]. This architecture has been an
attractive approach to solving PDEs because of its generalizability Lagaris et
al. [1998].

Many approaches have been proposed to approximate the solution to PDEs using
FCNs. One such approach is to create a model consisting of the sum of two
terms. The first term is used to satisfy the boundary conditions, whereas the
second term is an FCN that is trained to satisfy the PDE itself Lagaris et al.
[1998]. An FCN can be used here to approximate the solution to the PDE because
FCNs are universal function approximators Hornik et al. [1989]. Through this
approach, one can achieve a differentiable, closed analytic form of the
solution to the PDE Lagaris et al. [1998]. While this approach can be applied
to both PDEs and ODEs, it struggles to handle PDEs of higher dimensions as the
number of training points increases greatly. This, in turn, makes the proposed
method computationally exhaustive. Furthermore, this approach is less precise
than Finite Element methods in approximating a solution of the PDE Lagaris et
al. [1998].

Another common issue is the difficulty in solving PDEs with complex boundary
conditions. In fact, there are times when numerical methods cannot approximate
solutions to such PDEs Tadmor [2012]. The method proposed in Sirignano &
Spiliopoulos [2018] is known as the deep Galerkin method (DGM). This method
utilizes an FCN that is trained on a set of randomly sampled time and space
points. The key strength of this method is that it is able to satisfy the
differential operator and initial/boundary conditions. Furthermore, this
approach is meshfree; hence it can approximate solutions to PDEs that standard
numerical methods may struggle with. Lastly, this method is capable of solving
PDEs in high dimensional spaces, unlike the approach described by Lagaris et
al. [1998].

While these approaches have shown success in solving PDEs, there are still
many limitations. One such limitation is that FCNs are unable to capture
spatial and temporal data, primarily because of how FCNs process data. FCNs
deal with data in a one-dimensional spatially-invariant format. Therefore it
is hard for FCNs to capture spatial information Livingstone et al. [1997].
Another issue with FCNs is that they cannot outperform numerical methods in
low-dimensional spaces Lagaris et al. [1998]. Lastly, FCNs, which are not
designed as neural operators, primarily depend on a specific discretization of
the grid from which training points are sampled and hence need to be retrained
each time the discretization changes Li et al. [2020b].

###  3.2 Convolutional neural networks

While FCNs can approximate PDEs, they fail to capture potentially vital
information, such as spatiotemporal features. CNNs are the key to learning
spatial features O’Shea & Nash [2015]. Traditionally, the CNN architecture has
been utilized in image processing tasks Browne & Ghidary [2003], Naranjo-
Torres et al. [2020], Ciresan et al. [2011], but have in recent years been
trained to solve PDEs due to their ability to learn complex patterns in image-
formatted data. CNNs have been shown to be able to approximate the solution of
elliptical PDEs Winovich et al. [2019] and have been the foundation for hybrid
architectures such as PhyGeoNet Gao et al. [2021].

Standard CNN architecture differs from FCN architecture by utilizing three
distinct layers: convolutional layers, pooling layers, and fully-connected
layers O’Shea & Nash [2015]. As the name suggests, the convolutional layers
apply convolution to the input. This allows the network to learn local
features in an image. The pooling layers are applied to convolutional layers
in order to further generalize the local features. Pooling can be a function
such as max, min or avg, which is applied to subsets of the input. The
resulting matrix will have fewer elements than the original. Lastly, the fully
connected layers are utilized in order to aid with classification problems
Aghdam & Heravi [2017].

ConvPDE-UQ is a framework that utilizes CNNs to construct lightweight
numerical solvers that can solve elliptical PDEs Winovich et al. [2019]. These
lightweight solvers utilize the representative power of a neural network to
approximate the solution to a PDE on a general domain in a single forward pass
by using Green’s functions Winovich et al. [2019]. This allows the solvers to
be more computationally efficient than conventional solvers such as FEMs. One
key limitation of this architecture is that it cannot be easily extended to
modeling systems that are inhomogeneous and contain mixed boundary conditions
Winovich et al. [2019].

Physics-informed geometry-adaptive convolutional neural networks (PhyGeoNet)
are an extension of CNNs. This architecture utilizes convolutional neural
networks (CNNs) with a physics constraint. CNNs are often utilized when one
needs to learn spatiotemporal data O’Shea & Nash [2015]. The key disadvantage
of CNN architecture is that it can only handle rectangular grids; hence
utilizing a pure CNN architecture on complex geometries is challenging.
PhyGeoNet is a physics-constrained CNN architecture capable of learning
solutions to parametric PDEs on irregular geometries and nonuniform grids
without the need to utilize labeled data Gao et al. [2021]. This architecture
is capable of learning the solutions to parametric PDEs by utilizing an
elliptic mapping from the irregular physical domain to a regular reference
domain Gao et al. [2021]. While PhyGeoNet shows promise in solving PDEs, the
architecture is constrained by the fact that it is only capable of solving
steady-state parametric PDEs and thus cannot be applied to dynamic systems.

There have been recent works that have focused on designing architectures for
solving spatio-temporal problems. PhyCRNet or physics informed convolutional
recurrent network, proposed by Ren et al. [2022], aims to learn spatiotemporal
data through low-dimensional spatial feature extraction and temporal evolution
learning. However, this model is currently restricted to regular spatial
domains at fixed discretizations of the physical domain. Mavi et al. [2023]
propose convolutional-LSTM networks with non-local peridynamic differential
operator filter to capture time-dynamics in smaller latent space. They show
that the use of LSTMs coupled with convolution layers improves the
extrapolation ability of the network. However, this model fails to generalize
well to different initial conditions.

###  3.3 Making neural networks physics informed

PINN or physics informed neural networks are deep neural network based
architectures that utilize the laws of physics to help guide their training,
through the use of physics informed loss functions. They incorporate domain
knowledge as constraints on the empirical loss functions. These models can use
any underlying neural network architecture such as fully connected neural
networks, convolutional neural networks etc. but their learning is defined by
loss functions that are based on PDE residuals. In deep learning paradigms,
neural networks learn by minimizing a convex loss function. In machine
learning, the loss function typically contains the loss (ex: L2-Norm) and a
regularization term which helps alleviate the risk of overfitting. With PINNs,
the loss function also contains a knowledge-based term Raissi et al. [2019].
The knowledge-based term allows one to incorporate existing physical laws into
the training of the neural network. This allows for a mapping that allows the
neural network to be consistent with existing physical laws.

Since PDEs most frequently define physical laws, PINNs can be used to learn
the solution to PDEs. The loss function in PINN architecture embeds boundary
conditions, initial conditions, PDE residuals for a finite amount of points,
and space-time domain boundary Cuomo et al. [2022] into the knowledge-based
term. Upon training, PINNs are able to map between an input point in the
integration domain and the estimated solutions of the differential equation at
that point. Unlike other supervised learning techniques, PINNs are capable of
taking into account the underlying physics of the problem instead of just
using existing data.

Traditional PINNs have shown limited accuracy in solving PDEs, especially on
multi-scale dynamic systems. A variation that is capable of addressing this
limitation is gradient-enhanced PINNs (gPINNS). Traditional PINNs use the
knowledge-based term to encode the PDE residual into the loss function;
however, gPINNs also utilize the gradient information of the PDE residual in
the loss function Yu et al. [2022], allowing for much more accurate results
after training.

Another typical issue with PINNs when solving nonlinear PDEs is when those
PDEs have discontinuous solutions. Hybrid PINNs utilize ideas from
convolutional neural networks (CNNs) and FVMs in their architecture. Instead
of using automatic differentiation to solve the PDE, this method utilizes an
approximation to the differential operator, hence allowing the architecture to
have a convergent rate and avoiding the problem of discontinuity Fang [2021].

The process of adding a soft constraint to the loss function to guide the
model is known as regularization. However, unlike typical regularization
techniques, in PINN the use of a differential operator may not correspond to a
convex set. This term rather, corresponds to actual physical quantities. The
use of these types of regularization have been shown to fail in problems such
as one dimensional convection, where the model fails for larger values of
convection coefficient. It fails in a similar manner when trained to learn 1D
reaction-diffusion problems. Krishnapriyan et al. [2021] show that these
failures are due to optimization issues that arise from using physics based
loss constraints.

Conventional Methods | Data Driven Methods
---|---
Slower on Fine grids | Slow training time, resolution
| and discretization invariant
Requires explicit form | Only requires training data
Solve a single instance | Generalize to an
| entire family of PDEs
Higher computation power | Once trained, comparable
for larger meshes | computation power
| for different resolutions

Table 1: This table highlights the main differences between conventional
methods and data-driven methods

##  4 Pitfalls of data-driven methods

Neural networks have worked well on high-dimensional PDEs and parametric PDEs.
Networks trained on finite-dimensional function spaces, however, suffer from
the curse of dimensionality and become mesh dependent. To overcome this issue,
operator learning was introduced, wherein a neural network is trained to learn
mappings between infinite dimensional spaces and this network is termed a
neural operator. The primary advantage of these data-driven techniques is that
their approximation of certain high dimensional and parametric PDEs are
comparable to classical techniques and they can repeatedly solve these PDEs
with different initial and boundary conditions, without requiring additional
re-training. Furthermore, once trained, they generate the solutions faster
than classical methods and require less computing power.

These models, however, suffer from the same issues that are faced by data-
driven deep learning-based models. Firstly, they require a comprehensive
training dataset that accurately describes the physical quantities. If the
dataset isn’t comprehensive enough, these models may overfit to the training
data and fail to generalize to data beyond the training set. As shown by Zhu
et al. [2023] on DeepONets, extrapolation errors, or errors that occur when
predicting solutions that lie outside the support of the training set can be
minimized by larger training datasets.

Another issue with deep learning-based methods is the degree of
parameterization. That is the number of trainable parameters or the size of
the neural network. While there are no hard and set rules on the number of
parameters a neural network needs to have to provide optimal solutions, it is
understood that overparameterized neural networks have lower generalization
errors. This is due to the fact that as the number of parameters increases,
there are more locally optimal solutions; therefore, the network is more
likely to converge to a near-optimal solution. However, these large models are
not as interpretable as smaller models. Moreover, the computational cost of
training such large networks is very high. In their work, Kontolati et al.
[2023], explore the effects of over-parameterization. They conclude that over-
parameterization generally leads to better generalization. In the case of
under-parameterized models, increasing the dataset size doesn’t have a
significant improvement in generalization. Furthermore, they conclude that
deepONet and their variants have the best generalization accuracy,
irrespective of parameterization.

In their work, Grossmann et al. [2023], perform experiments to compare the
accuracy of physics-informed neural networks against finite element method.
They show that some versions of PINNs achieve lower error than FEM methods but
have larger training times. They also conclude that having more layers doesn’t
lead to significant improvement in performance. However, their comparison
metric for solution time was training time. If evaluation time was considered,
FEMs were slower than PINNs. Furthermore, they noted that PINNs had better
accuracy at higher dimensions.

Chuang & Barba [2022] discuss various drawbacks of PINNs in solving physics
problems. They highlight issues that occur during the design of model
architecture, such as the selection of number of layers and parameters, which
is purely done on a trial-and-error basis with no way of determining why it
works. They also discuss the inability to control the outcomes generated by
the model. Domain knowledge plays little role in improving the solution. The
authors conclude that while machine learning-based models cannot replace
conventional models, they can serve as complements to the classical models.

Neural operator models have been shown to have their own drawbacks as well.
While they offer better generalization, Fourier neural operators are
susceptible to aliasing errors and opaque outputs, as shown by Fanaskov &
Oseledets [2022]. These errors occur when the input grid is refined, causing
the outputs to deviate from the ground truth. Many of these operator models
suffer from vanishing gradients due to large number of layers.

The above-mentioned drawbacks and a lack of theoretical and statistical
guarantees for data-driven approaches should be taken into consideration when
choosing data-driven models as potential replacements for classical
techniques. These models are a good alternative when solving higher
dimensional PDEs and have been successfully tested on certain applications,
which will be discussed in the following sections.

##  5 Neural Networks for Operator Learning

To enable neural network-based architectures to learn infinite-dimensional
function mapping while solving PDEs, a new learning mechanism called operator
learning was proposed. The intuition behind operator learning was to design
mesh-independent, resolution-invariant learning techniques to solve PDEs. This
means one could train a model on a 40x40 matrix but test it against 256x256
matrices. A neural operator is designed to learn the mapping between
functional spaces. This operator could have FCN (Fully Connected Network) or
CNN (Convolutional Neural Network) or PCA-Net or GNN (Graph Neural Network) as
the underlying layers but the key characteristic of a neural operator is that
it attempts to learn the mapping between two infinite dimensional spaces using
a finite collection of input-output pairs. For example, the input could be a
Gaussian field, and the output could be the solution of the PDE on the field.
An important advantage of neural operators is that they are mesh-independent
because a single set of parameters may be used for different discretizations
Li et al. [2020a]. The training requires only the input-output pairs and no
knowledge of the PDEs being learned.

Let $G^{+}$ be a (potentially non-linear) map from the input space $A$ to the
solution space $U$, where $A$ and $U$ are separable Banach function spaces and
take the values $\mathcal{R}^{d_{a}}$ and $\mathcal{R}^{d_{u}}$, i.e., if we
draw $a_{j}$ and the corresponding solution $u_{j}$ from $A$ and $U$
respectively, then $u_{j}=G^{+}(a_{j})$. The neural operator is a parametric
map that aims to approximate $G^{+}$. This approximation is the operator G
denoted as G: $A$ x $\Theta\xrightarrow{}U$. The learning problem would then
be an optimization problem where the cost function $\mathcal{C}:U\times
U\rightarrow\mathcal{R}$ is an $L_{2}$ Bochner norm-based distance generating
function defined on $U$. The problem then becomes
min${}_{\theta\in\Theta}\mathbf{E}_{a}[\mathcal{C}(G_{\theta}(a),G^{+}(a)]$,
which would be the discrepancy in the actual map versus the operator. While a
neural network learns a map from the input domain $D$ to an element in $U$
using samples of input-output pairings $\\{x_{i},u(x_{i})\\}_{i=1}^{n}$, a
neural operator learns a mapping between $A$ to $U$ using samples of
$\\{u_{i},a_{i}\\}_{i=1}^{n}$.

To summarize, neural operators have shown potential in the following scenarios

  * 1.

Fast and fairly accurate approximation of PDE solutions at high resolutions,
without the need for additional re-training

  * 2.

Approximations of high dimensional PDEs, discretized on arbitrarily sized
meshes

  * 3.

Generating reasonably accurate solutions from low-resolution, less accurate
training data

###  5.1 DeepONet

Figure 2: Architecture of the basic Fourier neural operator

Deep operator network (DeepONet) is formulated around the observation that a
neural operator is a universal operator approximator Lu et al. [2019]. The
universal approximation theorems for neural operators are derived following
the key result that a single non-linear layer in a neural network is capable
of approximating any continuous operator Chen & Chen [1995].

The DeepONet architecture consists of two encoder networks. The first encoder
is used for encoding the input function and the second encoder is used for
encoding the location of the output functions. This architecture has been
shown to significantly outperform traditional fully connected neural network
architectures on dynamic systems and PDEs. The key strength of this
architecture lies in its capability of accurately approximating complex
mappings between infinite dimensional Banach spaces, hence making it a natural
choice for being utilized in learning the solution operator to a PDE.

While the original DeepONet architecture has shown its ability to learn PDEs,
its main downside is large training data requirement. Oftentimes, generating
this data can be very computationally expensive, and even with large training
datasets, DeepONet may fail to learn the underlying physical principles of the
equation. Physics-informed DeepONets are advanced DeepONets that utilize these
physical principles as a regularization mechanism, hence leading to
predictions that abide by the governing physical laws described. Automatic
differentiation is used to integrate these physical laws as penalties while
the model trains Wang et al. [2021]. Also, DeepONets, like regular neural
networks, can only take input at a fixed discretization. In other words, they
can not process input functions at variable discretization. To address this
issue, (Variable-Input Deep Operator Networks) VIDON is proposed which use
functions defined on sets in the encoder part to encode functions at a
different resolution Prasthofer et al. [2022].

DeepONet and its variants have been used to solve a plethora of problems,
including material physics Goswami et al. [2022], electrodynamics Cai et al.
[2021a], and aerothermodynamics Sharma Priyadarshini et al. [2021]. A
particularly interesting use of DeepONet comes from utilizing this model for
medical image simulation, particularly for aortic dissection. Aortic
dissection is a lethal condition which is characterized by the tear of the
aorta, which is a blood vessel used to deliver oxygen to the rest of the body
DeSanctis et al. [1987]. The DeepONet architecture has been used to predict
dissection progression in a heterogeneous aortic wall, which is a very complex
physical environment Yin et al. [2022].

Since the DeepONet architecture is capable of learning the non-linear operator
between infinite dimensional Banach spaces, it is capable of learning
oscillatory continuous functions. These types of functions are often found in
earthquake physics. An interesting problem that DeepONet has been able to
solve is how buildings respond to seismic excitation over a certain time
period. The DeepONet architecture utilizes scaling techniques that convert a
high-frequency function to a low-frequency function, which then allows this
architecture to learn a range of frequencies for the low-frequency function.
This multi-scale architecture, known as multi-scale DeepONet, has allowed
scientists to map seismic excitation to building responses Liu & Cai [2021].

###  5.2 Graph neural networks for operator learning

Graph neural network is a machine learning architecture that is built upon the
traditional neural network. The additional information that a graph neural
network aims to capture are interactions between constituent objects, which in
the case of the PDE, would be the kernel, which is described in sub-section
5.2.1. Traditional neural networks do not capture such relationships. A graph
neural network represents these relationships as edges between features, which
are represented as vertices. Learning relationships between these objects
occurs through a process called message passing. This technique allows nodes
to learn the information held by neighboring nodes through an aggregating
function, which combines the embeddings (feature vector) of the neighboring
nodes. An update function learns the new embeddings for the nodes. This
process is described in equation 2.

$v_{t+1}(x)=\sum_{y\in N(x)}F(h_{x},h_{y},e_{xy})$ (2)

In the above equation, the new embedding is an aggregate function of the
current embedding with the embeddings of all neighboring nodes and F is an
arbitrary function of hidden layer embeddings $h_{x}$, $h_{y}$ and the edge
$e_{xy}$. Message passing allows the network to learn the kernel, which in
turn approximates the solution to the PDE.

Graph neural networks, however, ignore long-range relationships because of
scaling issues. These approximations of relationships are ineffective when
trying to solve a PDE since they lead to issues with generalization. To
capture long-range relationships, Li et al. [2020c] discuss multipole methods.
They propose an architecture of linear complexity called multipole graph
neural operator, which captures long-range relationships between objects. The
model recursively adds points to the kernel matrix and is equivalent to the
multi-level formulation. The long-range relationships are modeled through the
addition of inducing points, which form sub-graphs.

####  5.2.1 Kernel operator

Kernel integration is a key component of the iterative learning process that
occurs in the Graph Neural operator. This process helps learn the mapping
between function spaces in PDEs. The process is described in equation 14 and
is equivalent to message passing in graphs. Every spatial point in the
discretized input can be represented by x-y coordinates. The values at that
point can be represented as functions a(x) and a(y) from the input function
space. A kernel operator can be defined to act on these data values in the
following manner

$(K_{a}u)(x)=\int_{D}k_{\phi}(a(x),a(y),x,y)u(y)dy$ (3)

Equation 3 can then be converted to an iterative neural network-based
architecture in equation 4, where the terms have meanings denoted in table 7.

$v^{(t)}=\sigma((W+K_{a})v^{(t-1)})$ (4)

###  5.3 Fourier neural operator

Fourier neural operators (FNOs) are another class of neural operators that use
the properties of the Fourier transforms to perform the calculation of the
integration operation at each layer of the neural operator. The motivation
behind using Fourier transforms instead of convolution is that they are
faster, and PDEs are inherently continuous, and representing them in Fourier
space is more efficient. Convolution in physical space is equivalent to
multiplication in Fourier space. Linear transformations are therefore
performed in the Fourier space. However, activation functions are applied in
the original space because they help recover non-periodic boundaries, which
are left by the Fourier space. The entire pipeline is shown in Figure 2.

####  5.3.1 Architecture

Fourier neural operator utilizes the operator learning technique implemented
by Li et al. [2020a]. It has been applied in spatial and temporal settings,
such as modeling the flow of fluids with time, specifically for weather
modeling and forecast. As specified in Li et al. [2020a], The iterative
updates to a state $v_{i}$ to reach the state $v_{i+1}$ is denoted by equation
5.

$v_{i+1}(x)=\sigma(Wv_{i}(x)+\mathcal{\kappa}v_{i}(x))$ (5)

In the above equation,$\mathcal{\kappa}$ denotes the kernel integral
transformation which is defined in equation 6.

$[\mathcal{\kappa}v_{t}](x)=\int_{D}\mathcal{\kappa}(y)v_{t}(y)dy$ (6)

The kernel integral operator is replaced with a convolution operator defined
in the Fourier space. This becomes the basis for the Fourier neural operator.
Equations 7 and 8 define Fourier transform and inverse Fourier transform.

$(\mathcal{F}f)_{j}(k)=\int_{D}f_{j}(x)e^{-2i\pi\langle x,k\rangle}dx$ (7)
$(\mathcal{F}^{-1}f)_{j}(x)=\int_{D}f_{j}(k)e^{2i\pi\langle x,k\rangle}dk$ (8)

The kernel integral operator can be represented as follows

$[\mathcal{\kappa}v_{t}](x)=\mathcal{F}^{-1}(\mathcal{F}(\kappa)\mathcal{F}(v_{t}))(x)$
(9)

In equation 9, $\mathcal{F}(\kappa)$ is the Fourier transform of a periodic
function $\kappa$ and can therefore be approximated as a Fourier series. By
truncating the series at a maximal number of modes $k_{max}$, the series can
be represented with a finite-dimensional parameterization.

###  5.4 Variants of neural operator

The neural operator has been fine-tuned and modified for specific instances
and situations. The following subsections discuss the different variations of
the neural operator, built on different neural network architectures. Figures
3 and 4 illustrate the flow of various neural operators. 6 provides links to
the source code for various neural operators.

Figure 3: The logical pipeline of the main neural operator architectures. The
above figure highlights the differences in the building blocks of the neural
architecture Figure 4: The logical pipeline of geo-FNO, implicit FNO and
multivariate FNO, which are all modified versions of the basic neural operator
architecture

###  5.5 GANO and UNO

Generative adversarial neural operator Rahman et al. [2022a] and U-shaped
neural operator Rahman et al. [2022c] are the operator variants of Generative
adversarial networks and U-Net, which are typically used for generative
modeling. These two architectures allow for memory-efficient implementations
of deeper neural operators. The generator network of GANO and the UNO network
have similar architecture, comprising an encoder network and a decoder network
with skip connections. However, in this paradigm, the input function spaces
are mapped to vector-valued function spaces with steadily decreasing domains
in the encoder network and increasing domains in the decoder network.

GANOs also use a discriminator neural functional, analogous to the
discriminator network in a GAN, to facilitate adversarial learning. This
discriminator differentiates between the generated solutions to the PDE and
the ground truth PDE solution. This network is optimized for Polish function
spaces. The adversarial learning is done using the Wasserstein formulation,
which allows for a bounded norm in infinite dimensional space. This
architecture is both resolution and discretization invariant, similar to the
basic FNO an is good at learning probability measures on function spaces.
Another advantage of GANOs over regular GANs is that they do not suffer from
modal collapse.

###  5.6 Adaptive Fourier neural operator

AFNO is a powerful sequence-to-sequence generative model similar to a Vision
Transformer that is built by stacking individual operator networks. The
network learns the representations through token mixing in the Fourier domain.
The model treats tokens as continuous objects in infinite space. The network
has a block diagonal structure, where the weight matrix is divided into weight
blocks and the kernel operates on them independently. This architecture is the
basis for FourcastNet Pathak et al. [2022], a weather prediction model that
has shown tremendous promise in forecasting weather, matching the accuracy of
ECMWF Integrated Forecasting System. Weather systems are applications of
complex physical entities that can be modeled as PDEs. Generative models such
as VFiT and FourCastNet can be used to model other complex physical systems
such as meteor trajectories, and volcanoes, predicting the deformations on the
surface of space vehicles, etc.

###  5.7 Implicit Fourier neural operator

Implicit Fourier neural operator (IFNO) was designed by You et al. [2022] to
address some of the limitations of the basic Fourier neural operator network
such as its tendency to overfit as the number of layers increases and its
susceptibility to vanishing gradient. IFNO is an integral operator-based
architecture where the operator learning is defined to be an implicit mapping,
modeled as a fixed point. Through this technique, the PDEs can be implicitly
expressed as a set of equations that can be solved using methods such as the
Newton-Raphson method, where approximations to the solution $\mathbf{V}$ are
successively improved until a precise value is achieved. This process is
defined in equation 10.

$\mathbf{V}[l+1]=\mathbf{V}[l]+\mathcal{R}(\mathbf{V}[l],F)$ (10)

where $\mathbf{V}[l]$ is the current approximation of the solution,
$\mathcal{R}$ is the operator that improves the approximation, and F is the
input vector. The network update equation is denoted in equation 11.

$\displaystyle f(x,\Delta t)=$ $\displaystyle f(x,\Delta t)+\sigma(Wf(x,\Delta
t)$ (11) $\displaystyle\quad\quad+\mathcal{F}^{-1}(\mathcal{F}(\kappa
v_{t})\cdot\mathcal{F}(f(.;\Delta t)))(x)+b)$

The key difference between this method and the vanilla FNO is that, in this
case, the parameters of the hidden layers are considered independent. The
total number of trainable parameters does not depend on the number of layers.
As the layer deepens, it becomes the analog of discretized ordinary
differential equations. The weight update is simply a time discretization of
the ODE.

You et al. [2022] further showed that this architecture can model heterogenity
and material defects. These claims were empirically verified against
hyperelastic, brittle and anisotropic materials.

The specific applications where this method showed promising results were as
follows

  * 1.

Porous medium flow in a 2-dimensional setting with heterogenous permeability
field.

  * 2.

Fiber material deformation in hyperelastic and anisotropic settings,
represented by the Holzapfel-Gasser-Odgen (HGO) model. Two boundary conditions
were considered - the Dirichlet boundary with uniform uniaxial displacement
applied on the right, and top edges and the Neumann boundary with uniaxial
tension applied on the top edge.

  * 3.

Fracture mechanisms in brittle glass ceramics, which as modeled as the Darcy-
Flow equation.

###  5.8 Geo Fourier neural operator

Geo-Fourier neural operator is an architecture developed by Li et al. [2022]
and is designed to be ’geometry-aware’. This architecture addressed the input
limitations of regular FNO, which is that it only works with rectangular
domains with uniform meshes. FNOs were made to work in irregular domains by
embedding them within larger rectangular domains, however, this was an
inefficient way to represent irregularity, and geo-FNO was designed to fix
this issue. Rather than embedding in a larger regular mesh, geo-FNO deforms
the non-uniform meshes into uniform meshes on which FFT can be applied. This
architecture can be used when the input is represented by non-uniform meshes
or point clouds or if the input domain is represented as signed distance
functions. The network performs two operations - deforming the non-uniform
input into a uniform mesh, followed by operator learning in latent space.
Traditional methods do not work when the input is deformed because the
deformed mesh in Fourier space does not correspond to the original system.
However, since the FNO approximates through training data, it is not
constrained by this limitation.

The deformation from the physical space to the computational space is done
using an adaptive moving mesh defined by a coordinate transformation. Geo-FNO
can be used in both structural and fluid mechanics problems, and it performs
well on Euler’s Equation for flow over the airfoil and the plastic forging
problem defined by equation 12.

$\rho^{s}\frac{\partial^{2}\mathcal{U}}{\partial t^{2}}+\nabla.{\varsigma}=0$
(12)

In the above equation, $\rho^{s}$ refers to the mass density, $\mathcal{U}$ is
the displacement and $\varsigma$ is the stress tensor. However, geo-FNOs have
only been tested on regular homeomorphic topologies and more studies are
needed to exhaustively determine the range of problems that can be solved by
this architecture.

###  5.9 Multiwavelet neural operator

The motivation for this architecture stems from the fact that the solution to
any PDE can be found by learning the inverse operator mapping from input to
output. To do this, the authors of Gupta et al. [2021] suggest decomposing the
kernel using fine wavelets. Embedding the inverse wavelet filters allows for
projecting the kernel into multiwavelet polynomial bases. Repeated
multiwavelet transform aids in learning complex dependencies and resolution-
independent solutions. The purpose of this architecture is to achieve compact
representations of the operator and to exploit orthogonality properties &
vanishing moments to capture complex information about physical systems and
data streams. The model maps the multiwavelet transform of the input to the
output at a very fine scale and has two components - the decomposition network
and the reconstruction network. The network computes multiwavelet coefficients
of the output at a coarse level using four neural networks. The reconstruction
network uses the output of the four networks to compute the multiwavelet
coefficients of the output at a finer level. This has a recurrent structure,
and it continues until the finest level is achieved. This model was
empirically shown to perform well on the 2D-Navier Stokes equation, where the
wavelet transform was applied to velocity. It has also shown promise in
generating finer resolution outputs when trained on low-resolution data.
However, it cannot generalize to high-frequency signals from low-frequency
ones.

###  5.10 Spectral neural operator

Spectral neural operator (SNO) is a recently proposed neural architecture that
has been designed to address opaque outputs and aliasing errors which can be
potentially observed in vanilla FNOs due to parameterization of the output
function Fanaskov & Oseledets [2022]. This model is designed to not suffer
from aliasing errors and perform lossless operations on functions.

Typically, when FNOs are applied on inputs with coarser grids, then, the
activation functions mitigate these aliasing errors. However, when the grid is
refined, these errors disappear but FNOs would try to mitigate them further,
causing outputs to deviate from the ground truth, resulting in errors in the
final solution.

To elaborate further, any function when represented using Fourier Series with
k $<$ $|N|$ terms, can be structured on a uniform grid of 2N + 1 points.
However, when activation function is applied, the output will have higher
frequency points and this means, the grid will need to have more than 2N + 1
points. However, while training, the FNO will learn to fix these errors in
representing higher harmonics. In situations where the higher harmonics also
fit the grid with 2N + 1 points, the bias of the FNO induced during the
training will cause it to try to fit the non-existent high harmonics.

To fix this issue, the Spectral neural operator is defined with a fixed number
of harmonics. It also decouples the interpolation process from the function
mapping process. The mapping is proposed in equation 13.

$\sum_{i}g_{i}(x)d_{i}=\sum_{i}g_{i}(x)b_{i}$ (13)

In the above equation, $g_{i}(x)$ are either complex exponential or Chebyshev
polynomial. $d_{i}$ and $b_{i}$ can be stacked with finite-dimensional
vectors. Spectral neural operator performs the above mapping and, importantly,
preserves the structure of the series. To perform this mapping on finer grids,
Chebyshev or Trigonometric interpolation should be used.

SNO has been shown to outperform vanilla FNO on integration, differentiation,
parametric ODE, elliptic equation, KdV equation and non-linear Schrodinger
equation. However, it does not perform as well on Burger’s equation with low
viscosity. The limitations of this version of SNO are that it is subject to
Gibb’s phenomenon and can only work on smooth input and output data. Basis
functions used for SNO are non-adaptive.

###  5.11 Physics Informed neural operator

In this section, we will discuss a hybrid neural operator architecture called
physics-informed neural operator (PINO), which learns the mapping between
function spaces through data-driven training while being subjected to physics
constraints. This architecture is not to be confused with physics informed
neural network (PINN), which learns the solution function as opposed to the
solution operator. PINO was designed to address the following key limitations
of existing approaches Li et al. [2021b]

  * 1.

Data-driven methods perform poorly if the training data is noisy or
insufficient

  * 2.

Physics-based approaches require a lot of computation power and may not
optimize when the constraints are more complex.

The advantages of PINO over PINN is that in the case of multi-scale dynamic
PDE systems, PINO offers better optimizations and it more accurately
propagates information from the initial and boundary conditions to unseen
parts of the interior Li et al. [2021b]. Moreover, like PINNs, it is
resolution invariant. This means that even when the operator is trained on
low-resolution or coarse data, it accurately manages to predict the solution
to high-resolution test instances. Furthermore, low-resolution data can be
combined with high-resolution PDE constraints without any degradation in
accuracy. It has been empirically shown Li et al. [2021b] that PINO has better
generalization capabilities than regular FNO and requires far less training
data.

PINO architecture comprises of a sequence of linear integral operators
followed by non-linearity, which allows the operator to learn non-linear
continuous operators. Operator learning, however, is done with two loss
functions - a data-based loss function, similar to the ones used by other
neural operators, and a physics-based loss function, used by PINN-based
architectures.

The data-based loss function is given by equation 14.

$\mathcal{L}_{data}=||u-G_{\theta}(a)||^{2}=\int_{D}(u(x)-G_{\theta}(a)(x))^{2}dx$
(14)

In the above equation, u denotes the actual solution, $G_{\theta}$ represents
the operator and $G_{\theta}(a)$ represents the predicted solution.

The physics-based loss function is defined on both stationary and dynamic
systems. A simple stationary system may be defined with bounded domain D as
follows.

$\displaystyle\mathcal{P}(u,a)=0~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}in~{}~{}D\subset\mathbf{R}^{d}$
(15) $\displaystyle u=g~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}in~{}~{}\partial
D~{}~{}~{}~{}~{}~{}$

The physics-based loss function for the stationary system presented in
equation 15 is defined as follows.

$\mathcal{L}_{pde}=||\mathcal{P}(a,u_{\theta})||^{2}_{L^{2}(D)}+\alpha||u_{\theta}|\partial
D-g||^{2}_{L^{2}(D)}$ (16)

In equation 16, $\mathcal{P}$ is a partial differential operator. a is the
instance and u is the solution. $u_{\theta}$ is the neural network
parameterized by $\theta$.

Let a dynamic system be defined as follows

$\displaystyle\frac{du}{dt}$
$\displaystyle=\mathcal{R}(u)~{}~{}in~{}~{}D\times(0,\infty)$ (17)
$\displaystyle u$ $\displaystyle=g~{}~{}in~{}~{}\partial D\times(0,\infty)$
$\displaystyle u$ $\displaystyle=a~{}~{}in~{}~{}\bar{D}\times{0}$

In equation 17, a = u(0) is the initial condition, $\mathcal{R}$ is the non-
linear partial differential operator defined on Banach spaces. The Loss on
this system is calculated in equation 18.

$\displaystyle\mathcal{L}_{pde}(a,u_{\theta})=||\frac{du_{\theta}}{dt}-\mathcal{R}(u_{\theta})||^{2}_{L^{2}(T;D)}+~{}~{}~{}~{}$
(18) $\displaystyle\alpha||u_{\theta}|\partial D-g||^{2}_{L^{2}(T;\partial
D)}+\beta||u_{\theta}|_{t=0}-a||^{2}_{L^{2}(D)}$

The above losses allow PINO to be constrained by physics while also being
trained with data. Furthermore, instance-wise fine-tuning is done after
training by using an operator loss. To compute the exact derivatives for the
loss functions, point-wise differentiation approaches are used.

PINO architecture, when fine-tuned, has shown promise in learning specific
equations such as Long Temporal Transient Flow, Chaotic Kolmogorov Flow,
Transfer Reynolds Numbers and Lid Cavity Flows. However, it has yet to be
tested on different kinds of geometry and it remains to be seen whether it can
function well on irregular geometry.

##  6 Applications of data-driven approaches in physics problems

In this section, we discuss various applications of neural operators in
material physics problems. Table 5 summarizes the advantages, limitations and
applications of various neural operator architectures. Figure 11 highlights
the key applications where neural operators have been successful in
outperforming conventional methods.

###  6.1 Solving partial differential equations

We shall first discuss the key PDEs for which neural operator architectures
have been designed. A large part of this subsection is devoted to exploring
the Navier-Stokes family of equations due to their broad range of
applications, which will be described in the following sections. Figure 5
depicts the relative errors of various machine learning models in learning
these PDEs.

The Navier-Stokes family of equations were developed to model the behavior of
viscous fluids. They represent the conservation of mass and momentum of
Newtonian Fluids. The 2D Navier Stokes equation is described in equation 19.

$\displaystyle\frac{\partial w(x,t)}{\partial t}+u(x,t).\nabla w(x,t)-$
$\displaystyle\nu\Delta w(x,t)=f(x)$ (19) $\displaystyle\nabla.u(x,t)=$
$\displaystyle 0,~{}~{}~{}~{}x\in(0,1)^{2},t\in[0,T]$ $\displaystyle
w_{0}(x)=$ $\displaystyle w(x,t=0),~{}~{}~{}~{}x\in(0,1)^{2}$

Figure 5: The figure highlights the performance of neural operator-based
architecture against neural network-based architectures. (a) represents how
the relative error decreases with the number of Epochs on the Navier-Stokes
Equation. It can be seen that FNO outperforms other architectures. (b)
represents how the relative error varies with resolution on Burgers’ Equation.
FNO has a significantly lower error than Multipole Graph neural operator
(MGNO), Graph neural operator (GNO), Low-Rank neural operator (LNO), etc. (c)
shows that FNO outperforms all other models on Darcy Flow Equation. Image
taken from Kovachki et al. [2021]

In the above equations, w represents the vorticity. The neural operator learns
to map the vorticity up to time T to vorticity at a time t$>$T. These
equations are derived from Newton’s second law to fluid mechanics, under the
assumption that stress is dependent on viscosity and pressure. Navier Stokes
Equations have various applications beyond modeling the flow of a liquid
through a pipe. They are also useful in aerodynamics by modeling the flow of
air around a wing, and they can even be used to model weather Pathak et al.
[2022]. One of its derivatives, the Burgers equation, is particularly useful
for modeling the one-dimensional flow of fluids as well as wave propagation
and even vehicular traffic movement Musha & Higuchi [1978], Taigbenu [1999].

Another derivative of Navier Stokes, known as Darcy Flow, is useful for
modeling the flow of fluids through a porous medium. Because of the wide range
of physical phenomena that can be modeled by these three equations, methods of
solving them are critical for several academic and commercial applications.
However, these equations, which take the form of non-linear PDEs, are
expensive to solve numerically.

In the following subsections, we will discuss the empirical results of neural
operators on these families of PDEs. The following two sections describe the
Poisson equation and Darcy flow, which belong to the class of equations
described in (LABEL:equation:PDEgeneric) and the FNO approximates the maps
$G^{+}:f\rightarrow a$ and $G^{+}:u\rightarrow a$ respectively. The remaining
sections deal with evolution equations where $a=u_{0}=u(.,0)$ are the initial
conditions and $u(.,t)$, the evolved states at a fixed time $t$, are the
solution function. The operator approximates a map $G^{+}:a=u(.,0)\rightarrow
u=u(.,t)$ for some $t$. The datasets, ie. $\\{f_{i},u_{i}\\}_{i=1}^{n}$ or
$\\{a_{i},u_{i}\\}_{i=1}^{n}$ as discussed earlier are generated using
appropriate numerical solvers.

####  6.1.1 Burgers equation

The Burgers equation is a non-linear PDE, which is represented in equation 20.

$\displaystyle\partial_{t}u(x,t)+\partial_{x}(u^{2}(x,t)/2)=$
$\displaystyle\nu\partial_{x}^{2}u(x,t)~{}x\in(0,1),t\in(0,1]$ (20)
$\displaystyle u(x,0)=$ $\displaystyle u_{0}(x)~{}x\in(0,1)$

with periodic boundary conditions where the initial condition $a=u_{0}\in
L^{2}_{per}((0,1);\mathbf{R})$ is an element in the class of Bochner
measurable functions whose norms $\|f\|_{(0,1);\mathbf{R}}$ lie in the
standard $L^{2}$ space and $\nu\in\mathbf{R}_{+}$ is the viscosity
coefficient. We aim to learn the operator mapping the initial condition to the
solution at time $t=1$,
$G^{+}:L^{2}_{per}((0,1);\mathbf{R})\times\mathbf{R}_{+}\rightarrow
H^{r}_{per}((0,1);\mathbf{R})$ defined by $u_{0}\rightarrow u(.,1)$ for any
$r>0$.

Li et al. [2020a] showed that FNO architectures outperformed other
architectures (Traditional CNN based architectures, autoencoder, Graph neural
networks) and provided the following loss measures against the Burgers
Equation, as shown in table 2.

Model | s=256 | s=512 | s=1024 | s=2048
---|---|---|---|---
FNO | 0.0149 | 0.0158 | 0.0160 | 0.0146
MGNO | 0.0243 | 0.0355 | 0.0374 | 0.0360
GNO | 0.0555 | 0.0594 | 0.0651 | 0.0663
DeepONet | 0.0569 | 0.0617 | 0.0685 | 0.0702
Table 2: Performance of different neural operators against Burgers Equation.
The data has been taken from Kovachki et al. [2021]

####  6.1.2 Darcy Flow equation

Li et al. [2020a] trained a Fourier neural operator to learn the solution
operator to the steady state 2D Darcy Flow Equation, shown in equation 21,
with a Dirichlet boundary $a\in L^{\infty}((0,1)^{2};\mathcal{R}_{+})$, where
a is the diffusion coefficient and $f\in L^{2}((0,1)^{2};\mathcal{R})$ is the
forcing function.

$-\nabla.(a(x)\nabla u(x))=f(x)$ (21)

Table 3 highlights the performance of various neural operator architectures
against the Darcy Flow equation.

Model | s=85 | s=128 | s=141 | s=211 | s=256 | s=421
---|---|---|---|---|---|---
FNO | 0.0108 | 0.0111 | 0.0109 | 0.0109 | 0.0107 | 0.0098
MGNO | 0.0416 | 0.0547 | 0.0428 | 0.0428 | 0.0542 | 0.0420
GNO | 0.0346 | - | 0.0332 | 0.0342 | - | 0.0369
DeepONet | 0.0476 | - | 0.0479 | 0.0462 | - | 0.0487
UNO | 0.0075 | - | 0.0072 | 0.0070 | - | 0.0068
MWT | - | 0.0074 | - | - | 0.0072 | -

Table 3: Performance of different neural operator-based architectures against
Darcy Flow Equation. It can be seen that UNO outperforms other models, with
Multiwavelet models (MWT) having similar performance values as UNO. The values
represent the relative L2 error. The data has been taken from Kovachki et al.
[2021], Gupta et al. [2021] and Rahman et al. [2022c]

####  6.1.3 Navier Stokes equation

The Navier-Stokes equation is defined as follows.

$\partial_{t}w(x,t)+u(x,t).\nabla w(x,t)=\nu\Delta w(x,t)+f(x)$ (22)

In equation 22, $u\in$ C([0, T ]; $H^{r}_{per}$((0, 1)2; $\mathbf{R}^{2}$))
for any $r>0$ is the velocity field, $w=\nabla\times u$ is the vorticity,
$w\in$ $L_{per}((0,1)^{2}$; $\mathbf{R}$) is the initial vorticity,
$\nu\in\mathbf{R}_{+}$ is the viscosity coefficient, and $f\in
L_{per}^{2}((0,1)^{2};\mathbf{R})$ is the forcing function.

Li et al. [2020a] aims to map the vorticity at a given time T to vorticity at
a later time in the future. They argue that vorticity is more challenging to
model than velocity.

Table 4 contrasts the performance of FNO and UNO on the 2D and 3D variants of
the Navier Stokes equation.

Model |  $\nu$=1e-3; |  $\nu$=1e-4; |  $\nu$=1e-5;
---|---|---|---
| T=50 | T=30 | T=20
FNO-3d | 0.0086 | 0.1918 | 0.1893
FNO-2d | 0.0128 | 0.1559 | 0.1556
UNO-3d | 0.0080 | 0.0830 | 0.1120
UNO-2d | 0.0050 | 0.0590 | 0.0700
Table 4: Performance of FNO and UNO architectures against Navier Stokes
Equation at fixed N = 1000. It can be observed that UNO outperforms FNO. The
values represent the relative L2 error. The data has been taken from Rahman et
al. [2022c] Kovachki et al. [2021]

####  6.1.4 Flow equations

In their work Wen et al. [2022] explore the possibility of applying FNOs to
solve multiphase flow equation, specifically with $CO_{2}$ and Water, given by
the following equations 23 and 24.

$\frac{\partial(\varphi\Sigma_{p}S_{p}\rho_{p}X_{p}^{CO_{2}})}{\partial
t}=-\nabla.[\Phi^{CO_{2}}|_{adv}+\Phi^{CO_{2}}|_{dif}]+q^{CO_{2}}$ (23)

$\frac{\partial(\varphi\Sigma_{p}S_{p}\rho_{p}X_{p}^{water})}{\partial
t}=-\nabla.[F^{water}|_{adv}+F^{water}|_{dif}]+q^{water}$ (24)

$\varphi$ is the porosity, $S_{p}$ is the saturation of phase $p$, and
$X_{p}^{\eta}$ is the mass fraction of component $\eta$ (water or $CO_{2}$)in
phase $p$. For both components, the advective mass flux $F^{\eta}|_{adv}$ is
obtained by summing over phases $p$. To solve this type of PDE, two types of
variables are sampled - scalars and fields. Field variables are horizontal &
vertical permeability, porosity, and injection perforation. The domain of this
problem is defined to be a bounded and open set and a modified version of FNO
called U-FNO is proposed to solve this problem.

The primary difference between a regular Fourier layer and a U-Fourier Layer
is that the U-Fourier layer also includes a U-Net embedded within it to
enhance the representation power of the layer through local convolutions.
However, since it is a CNN architecture, it is less flexible than an Operator.

####  6.1.5 Photoacoustic equation

2D Photoacoustic equation is another PDE that can be solved using neural
operators, as shown by Guan et al. [2021]. A slight modification is made to
the way the input is fed to the neural network. The training data is not
normalized beforehand (a term that refers to centering the data w.r.t its
moments) with Gaussian normalization prior to training. They show that both
FNO-2D and FNO-3D can be used to solve this equation. The former performs 2d
convolutions in space for a fixed interval of time, which is then recurrently
propagated in time to solve the next interval. In the case of 3D FNO, the
network performs 3D convolutions in space-time. They argue that while both can
be applied, the FNO-3D performs better than its 2D counterpart.

The photoacoustic pressure wave that Guan et al. [2021] used as the basis for
their experiments was defined in equation(s) 25.

$\displaystyle(\partial_{tt}-c_{0}^{2}\Delta)p(r,t)=0,$ (25) $\displaystyle
p(r,t=0)=x,$ $\displaystyle\partial_{t}p(r,t=0)=0$

Here $p(r,t)$ is the photoacoustic pressure wave at position $r$ and time $t$.
$c$ is the speed of sound.

###  6.2 Weather modelling

Pathak et al. [2022] discuss an FNO-based architecture - FourcastNet, which
aims to model weather data at a resolution of 0.25∘, which is approximately
equal to the area of 900Km2 near the equator. At such high resolutions, the
predictions of their model can be compared with those of the Integrated
Forecasting System (IFS). The key advantage of FourCastNet is that it is
45,000 times faster than NWP models on a node-hour basis, making it efficient
for the generation of large ensemble forecasts. The model can be used to
generate forecasts of massive weather phenomena such as Tornadoes, hurricanes,
and extreme precipitations. Once trained, it consumes less power than IFS to
generate forecasts. Some of the other advantages of the FourCastNet, as
mentioned by Pathak et al. [2022], are 1\. It has eight times the resolution
of typical DL-based weather modeling systems. 2\. It predicts lead times of up
to a week with exceptional levels of accuracy. 3\. It is rapid and inexpensive
once trained.

###  6.3 High resolution simulation of SARS-COV-2 replication transcription
complex dynamics

The replication of the SARS-COV-2 virus is done primarily by a multi-domain
protein. Conventional tools such as all-atom molecular dynamics (AAMD) have
shown to be limited in representing the molecular dynamics in high resolution
and timescale. Trifan et al. [2021] propose a graph neural operator-based
model to simulate the molecular dynamics from the AAMD data. The model was
used to learn time-dependent conformational changes within the molecules. The
GNO model was able to predict protein backbone conformation up to 5ps.

###  6.4 Thermochemical curing of composites

Figure 6: The figure represents the degree of cure predicted by the FEM method
vs. the same done by FNO. The figure has been taken from Chen et al. [2021]

The composite curing process is strongly dependent on the temperature
gradient, which influences the strength and mechanical properties of the
resulting material. Chen et al. [2021] propose a residual FNO architecture
that learns the mapping between the curing cycle and the temperature history.
The model leverages the time resolution invariance property of neural
operators and learns with smaller training datasets. The general heat transfer
equation that the model learns is as follows

Figure 7: These graphs represent the degree of cure predicted by FNO and FEM
methods at x = 35mm and 21mm. It can be observed that the two curves almost
entirely overlap for the whole duration. The maximum prediction errors were
within 0.02. This figure has been taken from Chen et al. [2021]

$\rho C\frac{\partial\mathbf{T}}{\partial t}=\frac{\partial}{\partial
x}(\lambda_{x}\frac{\partial\mathbf{T}}{\partial x})+\frac{\partial}{\partial
y}(\lambda_{y}\frac{\partial\mathbf{T}}{\partial y})+\frac{\partial}{\partial
z}(\lambda_{z}\frac{\partial\mathbf{T}}{\partial z})+\mathcal{Q}$ (26)

In equation 26, $\mathbf{T}$ is the temperature, $\rho$ and C are density and
specific heat capacity, respectively, $\lambda$ is the directional thermal
conductivity, and $\mathcal{Q}$ is the internal heat source. The boundary
conditions used for this problem were Dirichlet boundaries, Neumann
boundaries, and Robin boundaries. The architecture comprises K Fourier Layers,
similar to a ResNet neural network. The cure period is discretized into finite
intervals and each cure cycle is sampled as a vector from the space of n
discrete cycles. A key advantage of residual FNO is that it leverages the
domain knowledge to train the model using less training data. Rather than
learning the mapping over the entire hypothesis space, it learns the mapping
with respect to the specific input. For instance, if $T_{a}$ is the cure cycle
and $\mathcal{G_{\theta}}(T_{a})$ is the learned temperature history, the
network learns the difference $\mathcal{G_{\theta}}(T_{a})-T_{a}$, restricting
the size of the hypothesis space.

The authors empirically showed that this is an efficient way to model the
curing process due to the limited number of Fourier modes. Using domain
knowledge, the model was able to generate more accurate results. Figure 6
visually represents the differences between the degree of cure predicted by
the finite element method and by the neural operator method. Figure 7 is the
plot of the trends in predicting the degree of cure with time by the two
methods.

###  6.5 Coastal flood Modelling

Jiang et al. [2021] developed a digital twin of the earth’s coastlines using
FNO to predict sea surface levels on coastlines. They extended the basic FNO
to learn multivariate dynamics. They built surrogate models of NEMO (Nucleus
for European Modelling of the Ocean) and used these models to train the FNO to
predict sea surface height. Their empirical evaluations showed that FNO-based
models are approximately 45 times faster than systems such as NEMO and are
better suited for real-time predictions of coastal flooding.

###  6.6 Continuous spatio-temporal dynamics

Stochastic PDEs are used to model various physical systems under the influence
of randomness. Finite difference and spectral Galerkin methods require high
computation power and high-resolution meshes. SPDEs can be defined by the
following equation

$du_{t}=(\mathbf{L}u_{t}+F(u_{t}))dt+G(u_{t})d\mathcal{W}_{t}$ (27)

In equation 27, $\mathcal{W}_{t}$ represents a Weiner process, F and G are
continuous operators and $\mathbf{L}$ is the linear differential operator.
From the operator perspective, W is the continuous embedding of the spatio-
temporal data stream. To solve the system, u(t), with the initial condition,
is projected into the latent space and the ODE solver solves it in the latent
space, and the solution is mapped back into the original space. Salvi et al.
[2021] have proposed a neural operator to solve the following equations

Model | Advantages | Limitations | Applications
---|---|---|---
Fourier neural operator |  | Faster
---
Resolution invariant
discretization invariant
Data driven
Doesn’t need to know the underlying PDE
Zero shot Super resolution Li et al. [2020a]
| Parameterization may lead to opaque outputs
---
and aliasing errors Fanaskov & Oseledets [2022]
Only works on rectangular domains with uniform
meshes Li et al. [2022]
Overfits with deeper networks
Susceptible to Vanishing Gradient Rahman et al. [2022a]
Constrained by availability of training data. Li et al. [2021b]
| Burger’s Equation
---
Darcy Flow Equation
Navier Stokes Equation Li et al. [2020a]
Coastal Flood modelling Jiang et al. [2021]
Photoacoustic Equation Guan et al. [2021]
Chaotic systems Li et al. [2021a]
Seismic wave progressions Yang et al. [2021]
GANO/UNO |  | Memory efficient implementations of
---
deeper networks
Optimized for Polish and Banach spaces
Works well for bounded norms in infinite spaces
Good at learning probability measures
Don’t suffer from modal collapse Rahman et al. [2022a]
| Only works on rectangular domains with uniform
---
meshes Li et al. [2022]
Parameterization may lead to opaque outputs and
aliasing errors. Fanaskov & Oseledets [2022]
| Volcanic deformations Rahman et al. [2022a]
---
Video Interpolation Viswanath et al. [2022]
Dyadic Human motion prediction Rahman et al. [2022b]
DeepONet |  | Accurately approximate mappings
---
between infinite dimensional banach
spaces
Learns oscillatory continuous functions
Can learn the mapping from high frequency
Not resolution invariant: Only takes input function at a
fixed discretization
functions to low frequency functions Lu et al. [2019]
| Requires high amount of training data
---
May fail to learn underlying physical
principles
| Material Physics Goswami et al. [2022]
---
Electrodynamics Cai et al. [2021a]
Aerothermodynamics Sharma Priyadarshini et al. [2021]
Medical imaging DeSanctis et al. [1987]
Effects of seismic waves on buildings Liu & Cai [2021]
Graph neural operator |  | Learns long range dependencies in graph like data
---
Linear time complexity
Discretization invariant
Can learn Mesh invariant solutions Li et al. [2020c]
| Outperformed by Fourier neural operator on all PDEs
---
with regular meshes. Li et al. [2020a]
| Burgers Equation
---
Darcy Flow Equation Li et al. [2020c]
Protein dynamics in SARS-COV-2
virus Trifan et al. [2021]
PINO |  | Doesn’t suffer from generalization
---
errors that other operators suffer from
Overcomes the limitations of purely physics based
and purely data driven approaches.
Incorporate constraints at different resolutions -
combine coarse resolution data and high resolution
data. Li et al. [2021b]
| Has not been tested rigorously on High dimensional
---
PDEs Li et al. [2021b]
| Long Temporal Transient Flow
---
Kolmogorov Flows
Wave Equation
Non-Linear Shallow Water Equation Li et al. [2021b]
Adaptive FNO |  | Powerful generative model Pathak et al. [2022]
---
Efficient Token Mixer
Adaptive Weight sharing among tokens
Quasi-Linear Time Complexity
highly parallelized
Outperforms self-attention mechanisms Guibas et al. [2021]
| Can be modified through wavelet transforms
---
to better capture locality Guibas et al. [2021]
| Climate Modelling
---
Weather forecast
Hurricane prediction Pathak et al. [2022]
Generative imaging Guibas et al. [2021]
geo-FNO |  | geometry Aware
---
Input can be irregular meshes, point clouds
As fast as FNO but more efficient and accurate Li et al. [2022]
| Only been tested on regular homeomorphic topologies
---
Can be potentially expanded into PINOs but hasn’t been
empirically verified. Li et al. [2022]
| Structural and Fluid Mechanics
---
Problems
Euler’s equation for Airfoil flow Li et al. [2022]
Implicit FNO |  | Doesn’t suffer from Vanishing Gradient
---
Less prone to overfitting
Hidden Layer parameters are independent
Has the ability to learn material responses directly
from DIC displacement tracking measurements. You et al. [2022]
| Has long training times despite having fewer parameters
---
due to the iterative algorithm used for learning. You et al. [2022]
| Model Heterogenity and
---
Material Defects in anisotropic
and hyperelastic setting
Porous Medium Flow
Fracture mechanisms You et al. [2022]
Multiwavelet FNO |  | Compact Representation of data
---
Resolution-independent solutions
Learn complex dependencies Gupta et al. [2021]
| Performance degrades if the Kernel
---
used for data generation is changed.
Cannot generalize to high frequency
signals from low frequency ones Gupta et al. [2021]
| Burgers Equation (1D)
---
Navier-Stokes Equation (2D)
Darcy Flow Equation (2D)
Korteweg-de Vries Equation (1D) Gupta et al. [2021]
Spectral FNO |  | Doesn’t suffer from aliasing errors
---
Lossless operations on Functions
Preserves the structure of the functions Fanaskov & Oseledets [2022]
| Doesn’t perform well on Burger’s Equations
---
Suffers from Gibbs Phenomenon
Only works on smooth input/output
Basis functions used are non-adaptive Fanaskov & Oseledets [2022]
| Basic Integration, Differentiation
---
Parametric ODEs
Elliptic Equations
KdV Equation
Non-Linear Schrodinger Equation
Fanaskov & Oseledets [2022]
|  |  |

Table 5: The table highlights the key advantages and limitations of the most
recent operator-based neural architectures for solving PDE and other physics
problems

  * 1.

Stochastic Ginzburg-Landau equation

$\displaystyle\partial_{t}u-\Delta u$ $\displaystyle=3u-u^{3}+\xi$ (28)
$\displaystyle u(t,0)$ $\displaystyle=u(t,1)$ $\displaystyle u(0,x)$
$\displaystyle=u_{0}(x),(t,x)\in[0,T]\times[0,1]$

Where the operator learns the solution along the noise path $\xi$.

  * 2.

Stochastic Korteweg-De Vries equation

$\displaystyle\partial_{t}u+0.1\partial^{3}_{x}u$
$\displaystyle=6u\partial_{x}u+\xi$ (29) $\displaystyle u(t,0)$
$\displaystyle=u(t,1)$ $\displaystyle u(0,x)$
$\displaystyle=u_{0}(x),(t,x)\in[0,T]\times[0,1]$

This equation describes the propagation of non-linear waves on the surface of
fluids under random perturbations. The equation becomes stochastic when the
noise is defined to be the partial sum approximation of a Q-Weiner Process.

  * 3.

Stochastic Navier Stokes equation in 2D

$\partial_{t}u-\nu\Delta w=-u.\nabla w+f+0.05\xi$ (30)

This equation describes the incompressible flow of fluids under force, where
$w$ refers to the vorticity and $\nu$ is the viscosity coefficient. The noise
is set to a Q-Weiner process.

###  6.7 Disspative dynamics in chaotic systems

Chaotic systems tend to be unpredictable because they are susceptible to minor
perturbations Li et al. [2021a]. However, their long-term trajectories depend
on an invariant measure called the global attractor. This problem has been
previously approached using recurrent neural networks but has only worked for
very short trajectories. The dissipativity and Markovian properties of these
systems can be modeled using neural operator because they exhibit invariance.
Dissipativity can be described as a compact set into which all other bounded
sets evolve over time. The solution operator maps these initial conditions
into the solution set.

Figure 8: This figure shows the simulation of Kolmogorov flow by the Markov
neural operator (MNO) with an initial condition generated from a random
Gaussian field. The model captures the energy spectrum that converges to the
cascade rate of $k^{-5/3}$. Image taken courtesy of Li et al. [2022]

To train the neural operator, dissipativity is imposed on the mesh by
augmenting data on the outer shell. The authors consider finite-dimensional
Lorenz-63 system, infinite-dimensional Kuramoto-Sivashinsky equations, and
Kolmogorov flows. For their work, the authors limit themselves to ergodic
systems, and the model does not make predictions outside the trained
distribution. The operator learns the mapping in single time steps so that the
input would be the system at time step t, and the operator maps it to the
system at time step t+1. Long-term predictions are made with repeated
composition. Figure 8 highlights the simulation of Kolmogorov flow in a
40-second span from an initial Gaussian Random Field.

###  6.8 Seismic Wave propagation

Figure 9: The figure compares the waveform inversion obtained through FNO vs
the same obtained through SEM and adjoint tomography. The waveforms on the
left (a) and (c) represent the true velocity and the source locations are
denoted by the red circles. The relative misfit between true and inverted
velocity in the case of SEM is 0.028 while for FNO it is 0.0319. Image taken
from Yang et al. [2021]

Earthquakes and seismic activities are described by various forms of wave
equations. However, solving these equations can be computationally expensive.
Yang et al. [2021] discuss neural operators as a way to overcome the
computation constraints by training the networks on an ensemble of wave
equation data represented as low-resolution matrices.

Figure 10: This Figure illustrates the 3D deformation experienced by a plate
due to projectile impact. Each figure denotes the deformation in a
25-microsecond interval. Each subfigure represents the axial cross-section
with Von Mises stress measure. Figure taken courtesy of Liu et al. [2022]

The neural operator is trained to learn the 2D acoustic wave equation, defined
as the mapping from velocity, defined on an irregular mesh, to the wavefield
solution. The network is also used to perform waveform inversions through
reverse mode automatic differentiation. Figure 9 illustrates the difference
between the inverted waveforms obtained through SEM and neural operator-based
methods.

###  6.9 Inelastic impact problems

The training data for the operator is generated as random fields with the von
Karman covariance function. The solution is obtained using the Spectral-
Element method by applying the Ricker Wavelet source. An added advantage of a
neural operator is that its ability to perform automatic differentiation,
which is equivalent to the adjoint state method, allows for full waveform
inversion without having to compute the adjoint wavefield.

In their work, Liu et al. [2022] design neural operators to learn the
inelastic deformation of polycrystalline solids, where polycrystalline refers
to materials composed of disjoint grains, where each grain is made of the same
material but exhibits different orientations based on the reference viewpoint.
Plastic deformations occur through the mechanisms of slip and twinning. The
problem is defined by the kinetic relation between crystallographic
orientation, inelastic deformation gradient, slip activity, and twin volume
fractions. The study is motivated by the fact that fine-scale behavior
influences the behavior on a coarser scale. The authors propose two models to
learn this relation.

  * 1.

2DFFT This model represents the problem in 2 dimensions with two slip systems
and no twinning. The problem is solved using Fast Fourier Transform. The input
to the network is the deformation history. The operator uses this information
to approximate Cauchy stress.

  * 2.

3D Taylor This is the 3-dimensional setting to study the behavior of materials
such as magnesium. To solve this system, Taylor averaging assumption is used
to assume that the deformation gradient is uniform on the unit cell.

Figure 10 portrays how the neural operator simulates projectile impact on a
plate. It shows the axial cross section of the plate subjected to impact.

Figure 11: The above figure highlights the performance of various neural
operator based architectures in solving different types of problems. It
provides a visual comparison between generated solution an ground truth. The
images on the top are the ground truth values while the ones on the bottom row
are generated by the operator models. The first image on the left represents
the airfoil Flow simulation generated by the geo-FNO architecture Liu et al.
[2022]. The second one is the Darcy Flow simulation generated by the GANO
architecture Rahman et al. [2022a]. The third one represents the Kolmogorov
Flow generated by the PINO architecture Li et al. [2021b], the fourth pair
represents the Navier-Stokes equations simulated by the vanilla FNO Li et al.
[2020a] and the last one is the weather forecast model generated by the
FourcastNet Pathak et al. [2022].

###  6.10 Forecasting subcritical cylinder Wakes

In their work, Renn et al. [2023] propose using Fourier neural operators to
temporally forecast velocity fields to study the Karman vortex street
phenomenon. They use this approach to determine how physical fluid flows
evolve with time. The neural operator is trained on flow data obtained through
particle image velocimetry and predicts ten-time steps. They show that the
model predicts the velocity fields over the entire range of Reynolds numbers.
While the model achieves low errors in predicting, it fails to learn unsteady
turbulent dynamics. This could be because of the insufficient spatial
resolution of the model. The authors speculate that training the model to
learn smaller secondary flows could help predict the flows at larger lengths.

###  6.11 Video interpolation

More recently, neural operator-based models have been used for computer vision
problems. In their work, Viswanath et al. [2022] model the problem of video
frame interpolation as implicitly learning a PDE, where the complex motions
exhibited by the moving objects within a video follow a PDE. A UNO was used to
learn these complex trajectories to predict intermediate frames in a video.

###  6.12 Human motion analysis in dyadic setting

Prediction of reactions to human actions in dyadic or two-person settings is
an interesting problem that has applications in gaming and 3D simulations. In
their work, Rahman et al. [2022b] explore using Conditional GANO-based
architectures to predict the reactions of humans conditioned on the movements
of their dyadic partners. The model accurately learns from long sequences of
motions at any arbitrary temporal resolution.

##  7 Scalability

Finite element approaches are generally slow and imprecise as they suffer from
a trade-off between mesh resolution and computation time. The higher the
resolution, the higher the computation cost per evaluation. Moreover, the
results of a computation are only valid for a single instance of a PDE.
Changing initial or boundary conditions or any other parameter requires the
expensive computation to be re-run.

Neural network based methods are capable of approximating a PDE with decent
accuracy. Although training them is computationally expensive, each
computation of the forward evaluation is much faster than the conventional
approaches. Some of these methods, such as the ConvPDE-UQ framework, are even
able to parameterize the initial and boundary conditions as part of the input
to the model in order to enable computation of any system within a given
family of PDEs on any domain without having to retrain the model (Winovich et
al. [2019]). However, neural network approaches are still mesh-dependent. The
resolution that can be achieved from the model is determined by the resolution
of the training data, which is computed via conventional methods. Thus the
computational cost for training is still sensitive to resolution. This is
where a neural operator is advantageous as it attempts to learn the function
and hence, the mapping between infinite dimensional spaces. The result is that
they are resolution invariant in that they can be trained with data that has a
relatively low resolution and will still be able to evaluate at a higher
resolution with the same error rate as in low resolution. Li et al. [2021b]
demonstrated that their neural operator variant, the Fourier neural operator,
is capable of accurately learning PDEs with zero-shot super-resolution. They
further proved that the Fourier neural operator achieves superior accuracy
compared to neural network-based solvers while still enjoying the same
benefits in terms of fast computation of the forward evaluation and
generalization to any instance within a PDE family.

All of these features of FNOs- low evaluation cost, zero-shot super
resolution, and generalization- have significant implications in terms of
computational efficiency. Li et al. [2021b] highlight this by presenting the
Bayesian Inversion Problem, where they use a function space Markov chain Monte
Carlo (MCMC) method (Cotter et al. [2013]) to draw samples from the posterior
distribution of the initial vorticity in Navier-Stokes. MCMC are a set of
algorithms for sampling data from distributions. The approach involves
constructing Markov chains for the desired distribution, and a sample of this
distribution would be a set of states of the Markov chain. Metropolis-Hastings
is a well-known MCMC algorithm. In this experiment, they compare FNOs and
conventional solvers. While both are able to achieve similar results, they
vary substantially in terms of computation time. The MCMC using the
traditional solver took 2.2 seconds per computation, whereas the FNO took only
0.005s per computation. The FNO, however, requires a one-time training, which
in this scenario took approximately 12 hours. The traditional solver, on the
other hand, requires no such training. Hence, for a small number of data
points, the conventional solver is more efficient. The benefits of FNOs are
instead realized at a larger scale because the model only has to be trained
once and can be used to quickly evaluate any instance of the PDE. To
illustrate this point, let T represent the total computation time in seconds
to make n evaluations. For the traditional solver, the computation time has
the form of a fixed rate ($T=2.2n$ in this particular setting), whereas, for
FNO, computation time would behave as a smaller rate plus a one-time cost
($T=43200+0.005n$ in their setup). The authors generated 30,000 data points
using each method. Using the FNO, this took 12 hours for training plus an
additional 2.5 minutes for the 30,000 forward evaluations, whereas the
traditional solver took a total of 18 hours for the same number of evaluations
(Li et al. [2021b]).

These advantages in computation time extend to cost savings, especially at
large scale. To illustrate the difference in cost at a large scale, consider
the cost of running both models with the same experimental setup as above on a
p3.2xlarge EC2 instance on AWS, which has 8 vCPU and 61 GiB (or approx. 65 GB)
of memory. The P3 instances feature the NVIDIA V100 GPU making it the most
similar of the AWS compute options to the NVIDIA V100 GPU with 16 GB memory
used by Li et al. [2021b]. For the purposes of this demonstration, assume
these instances have the same hardware performance as the hardware used by Li
et al. The on-demand rate for this instance, which is the cheapest of the P3s,
is $3.06 per hour. Thus, training the FNO would cost approximately $36.72.
Once trained, the FNO can theoretically perform up to 720000 computations per
hour, while the conventional solver would take 440 hours to do the same number
of evaluations. For 100,000 evaluations, the FNO would cost under $40,
including training time, whereas the conventional solver would cost
approximately $187. Beyond the cost savings, this computational efficiency has
important implications for environmental impact, as cloud computing is known
to consume significant energy resulting in a large carbon footprint.

In designing mechanical systems involving aerodynamics or fluid dynamics, it
is common for engineers to have to compute the forward operation of Navier
Stokes several thousands of times, varying the coefficients with each
evaluation in order to recover certain properties of the system. Fourestey &
Moubachir [2005], for example, investigated inverse problems of the Navier
Stokes equations in the context of designing bridge decks under wind loads.
Solving the inverse problem of a given PDE is also critical to tuning
predictive modeling systems such as those for weather and climate modeling
(Kashinath et al. [2021]). Using conventional numerical methods to solve
inverse problems of a given PDE can be prohibitively slow and expensive. The
scalability of FNOs can make this problem not only feasible but cost-
effective, which has significant implications for mechanical design.

##  8 Future work - Adaptability

In this paper, we have illustrated the advantages that FNOs have over
traditional numerical solvers in terms of computational performance, as well
as their superior accuracy when compared to neural network-based solvers. The
remaining question to be answered is how these benefits can be realized in
academic research and commercial applications. The development of numerical
methods such as FDM and FEM began as early as the 1940s Hrennikoff [1941].
However, these methods did not see wide-spread use until the 1960s when open-
source programs for FEM began to be developed, such as Nastran developed by
NASA Butler & Michel [1971] and SAP IV developed at UC Berkeley Gran & Yang
[1978]. Since then, many tools for the numerical modeling of physical systems
have been built and made widely available to researchers and engineers,
including MATLAB, COMSOL, and Autodesk Simulation. The question now becomes-
how can the same be done with FNO-based solvers?

As FNOs are still relatively new computational methods, there is more work
that is needed to develop them further, and more variants of them are likely
to be developed over the next few years. Nevertheless, existing variants
already have the potential to greatly accelerate physics-based modeling.
However, as with many newly developed neural networks, they are unlikely to
see practical use or widespread adoption by physics researchers until they are
integrated into a software package that can reliably be used without a firm
background in machine learning. Developing this software and enabling it to be
used for practical applications would further motivate future research and
development of these approaches for PDE approximation. An important factor in
achieving this is open-source. Li et al. have already made their code open-
source for their work in Li et al. [2020a]. Open-source accelerates
development and enables standardization in software packages. This
standardization is central to the repeatability of results and is hence,
critical for research.

It is important to note that numerical solvers are still required to produce
the training data for FNOs. As with all machine learning approaches, the
quality of this data, in terms of the size and spread of training data, as
well as accuracy, impacts the performance of the FNO. In this respect, machine
learning paradigms such as that of active learning Settles [2012], which uses
learning theoretic arguments to come up with strategies to select training
data, can be used to improve the sample complexity (a term used in machine
learning for the number of samples required to achieve a particular accuracy)
of learning based techniques. In particular, the learning algorithm would
decide on the pairs of $\\{u_{i}\\}_{i=1}^{N}$ for which the solver should
generate the corresponding $\\{f_{i}\\}$ or $\\{a_{i}\\}$ with the aim of
selecting the right pairs such that the network is able to approximate the
solver function with the lowest number of pairs $N$. Meta-learning is another
paradigm that can be used to improve the sample complexity of learning
algorithms. This method aims to learn some underlying connections to different
machine-learning tasks. While there are several notions of meta-learning, a
popular algorithm is MAML Finn et al. [2017], which aims to learn an
initialization for gradient-based learning approaches from different tasks
sharing a common structure. This learned initialization, when used for the
gradient-based learning approaches, allows for better sample complexity
Saunshi et al. [2020].

Due to the dependency of FNOs on numerical solvers, one potential future for
FNOs would be to integrate them into FEM software packages. This would make
them more accessible to physics researchers allowing them to leverage the
computational benefits of FNOs to accelerate their work. The software could
train an FNO for a user-inputted PDE with data it produces using FEM and a
sampling method based on active learning or meta-learning. The trained FNO
would then be used in the computation of the forward evaluation with a mesh
and other parameters defined by the user via the same interfaces they use
today, as the software would handle the parameterization of those inputs into
the structure expected by the FNO. Of course, there are many variants of FNOs,
each created for different types of systems involving PDEs, where they achieve
superior performance over the vanilla version. Users of FEM software would not
be aware of the distinctions between them. Hence, this software would have to
be able to provide the user with some recommendation of which variant, if any,
to use in which scenarios. A deeper study of the performance of different
variants across different use-cases would be required in order to build this
knowledge base. This paper is a starting point for that.

While the accuracy of FNO-based solvers is invariant to resolution, it is
still true that a given FNO-based solver could have higher accuracy on certain
systems over others due to the non-deterministic nature of machine learning.
This would be an additional consideration for researchers using this software.
One approach to contend with this would be to have the software also produce
test data along with the training data and provide a report of accuracy on
this test set so that the researcher can weigh the accuracy-efficiency trade-
off between FNO and conventional methods in order to decide which approach to
use. If they opt for the FNO approach, they could also include this report in
their findings.

##  9 Conclusion

The goal of this paper was to characterize the numerous methods that have been
developed for approximating PDEs in order to highlight the potential
opportunities that exist for further development of these technologies. We
have provided a dense overview of these methods, from conventional solvers to
more novel approaches developed in recent years that leverage neural networks
to approximate the mapping between finite spaces as well as those capable of
approximating the mapping between function spaces by leveraging the neural
operator. We have provided details about several variants of each type of PDE
solver and have compared the performance of these variants across several
applications, particularly those in the field of physics. We have also
articulated the advantages of FNO-based solvers as compared to conventional
solvers and the implications this could have for the future of physics
modeling. Furthermore, we proposed a high-level vision for how this technology
could eventually be used to accelerate computation-based physics research.

## References

  * Acharya et al. [2007] Acharya, S., Baliga, B., Karki, K., Murthy, J., Prakash, C., & Vanka, S. P. (2007).  Pressure-based finite-volume methods in computational fluid dynamics, .
  * Aghdam & Heravi [2017] Aghdam, H. H., & Heravi, E. J. (2017).  Guide to convolutional neural networks.  New York, NY: Springer, 10, 51.
  * Aochi et al. [2013] Aochi, H., Ulrich, T., Ducellier, A., Dupros, F., & Michea, D. (2013).  Finite difference simulations of seismic wave propagation for understanding earthquake physics and predicting ground motions: Advances and challenges.  In Journal of Physics: Conference Series (p. 012010).  IOP Publishing volume 454.
  * Bebis & Georgiopoulos [1994] Bebis, G., & Georgiopoulos, M. (1994).  Feed-forward neural networks.  Ieee Potentials, 13, 27–31.
  * Browne & Ghidary [2003] Browne, M., & Ghidary, S. S. (2003).  Convolutional neural networks for image processing: an application in robot vision.  In Australasian Joint Conference on Artificial Intelligence (pp. 641–652).  Springer.
  * Butler & Michel [1971] Butler, T. G., & Michel, D. (1971).  NASTRAN: A summary of the functions and capabilities of the NASA structural analysis computer system volume 260\.  Scientific and Technical Information Office, National Aeronautics and Space ….
  * Cai et al. [2022] Cai, S., Mao, Z., Wang, Z., Yin, M., & Karniadakis, G. E. (2022).  Physics-informed neural networks (pinns) for fluid mechanics: A review.  Acta Mechanica Sinica, (pp. 1–12).
  * Cai et al. [2021a] Cai, S., Wang, Z., Lu, L., Zaki, T. A., & Karniadakis, G. E. (2021a).  Deepm&mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks.  Journal of Computational Physics, 436, 110296.
  * Cai et al. [2021b] Cai, S., Wang, Z., Wang, S., Perdikaris, P., & Karniadakis, G. E. (2021b).  Physics-informed neural networks for heat transfer problems.  Journal of Heat Transfer, 143.
  * Chen et al. [2021] Chen, G., Li, Y., Meng, Q., Zhou, J., Hao, X. et al. (2021).  Residual fourier neural operator for thermochemical curing of composites.  arXiv preprint arXiv:2111.10262, .
  * Chen et al. [2023] Chen, K., Wang, C., & Yang, H. (2023).  Deep operator learning lessens the curse of dimensionality for pdes.  arXiv preprint arXiv:2301.12227, .
  * Chen & Chen [1995] Chen, T., & Chen, H. (1995).  Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems.  IEEE Transactions on Neural Networks, 6, 911–917.
  * Chuang & Barba [2022] Chuang, P.-Y., & Barba, L. A. (2022).  Experience report of physics-informed neural networks in fluid simulations: pitfalls and frustration.  arXiv preprint arXiv:2205.14249, .
  * Ciresan et al. [2011] Ciresan, D. C., Meier, U., Masci, J., Gambardella, L. M., & Schmidhuber, J. (2011).  Flexible, high performance convolutional neural networks for image classification.  In Twenty-second international joint conference on artificial intelligence.
  * Cotter et al. [2013] Cotter, Roberts, Stuart, & White (2013).  Mcmc methods for functions: Modifying old algorithms to make them faster.  Statist. Sci., 28, 424 – 446.
  * Cuomo et al. [2022] Cuomo, S., Di Cola, V. S., Giampaolo, F., Rozza, G., Raissi, M., & Piccialli, F. (2022).  Scientific machine learning through physics-informed neural networks: Where we are and what’s next.  arXiv preprint arXiv:2201.05624, .
  * Cybenko [1989] Cybenko, G. (1989).  Approximation by superpositions of a sigmoidal function.  Mathematics of control, signals and systems, 2, 303–314.
  * Demirdžić & Muzaferija [1994] Demirdžić, I., & Muzaferija, S. (1994).  Finite volume method for stress analysis in complex domains.  International journal for numerical methods in engineering, 37, 3751–3766.
  * DeSanctis et al. [1987] DeSanctis, R. W., Doroghazi, R. M., Austen, W. G., & Buckley, M. J. (1987).  Aortic dissection.  New England Journal of Medicine, 317, 1060–1067.
  * Donea & Huerta [2003] Donea, J., & Huerta, A. (2003).  Finite element methods for flow problems.  John Wiley & Sons.
  * Eymard et al. [2000] Eymard, R., Gallouët, T., & Herbin, R. (2000).  Finite volume methods.  Handbook of numerical analysis, 7, 713–1018.
  * Fadlun et al. [2000] Fadlun, E., Verzicco, R., Orlandi, P., & Mohd-Yusof, J. (2000).  Combined immersed-boundary finite-difference methods for three-dimensional complex flow simulations.  Journal of computational physics, 161, 35–60.
  * Fanaskov & Oseledets [2022] Fanaskov, V., & Oseledets, I. (2022).  Spectral neural operators.  arXiv preprint arXiv:2205.10573, .
  * Fang [2021] Fang, Z. (2021).  A high-efficient hybrid physics-informed neural networks based on convolutional neural network.  IEEE Transactions on Neural Networks and Learning Systems, .
  * Finn et al. [2017] Finn, C., Abbeel, P., & Levine, S. (2017).  Model-agnostic meta-learning for fast adaptation of deep networks.  In International conference on machine learning (pp. 1126–1135).  PMLR.
  * Fourestey & Moubachir [2005] Fourestey, G., & Moubachir, M. (2005).  Mcmc methods for functions: Modifying old algorithms to make them faster.  Computer Methods in Applied Mechanics and Engineering, 194, 877–906.
  * Gao et al. [2021] Gao, H., Sun, L., & Wang, J.-X. (2021).  Phygeonet: Physics-informed geometry-adaptive convolutional neural networks for solving parameterized steady-state pdes on irregular domain.  Journal of Computational Physics, 428, 110079.
  * Godunov & Bohachevsky [1959] Godunov, S., & Bohachevsky, I. (1959).  Finite difference method for numerical computation of discontinuous solutions of the equations of fluid dynamics.  Matematičeskij sbornik, 47, 271–306.
  * Goswami et al. [2022] Goswami, S., Yin, M., Yu, Y., & Karniadakis, G. E. (2022).  A physics-informed variational deeponet for predicting crack path in quasi-brittle materials.  Computer Methods in Applied Mechanics and Engineering, 391, 114587.
  * Gran & Yang [1978] Gran, C. S., & Yang, T. (1978).  Nastran and sap iv applications on the seismic response of column-supported cooling towers.  Computers & Structures, 8, 761–768.
  * Grossmann et al. [2023] Grossmann, T. G., Komorowska, U. J., Latz, J., & Schönlieb, C.-B. (2023).  Can physics-informed neural networks beat the finite element method?  arXiv preprint arXiv:2302.04107, .
  * Guan et al. [2021] Guan, S., Hsu, K.-T., & Chitnis, P. V. (2021).  Fourier neural operator networks: A fast and general solver for the photoacoustic wave equation.  arXiv preprint arXiv:2108.09374, .
  * Guibas et al. [2021] Guibas, J., Mardani, M., Li, Z., Tao, A., Anandkumar, A., & Catanzaro, B. (2021).  Adaptive fourier neural operators: Efficient token mixers for transformers.  arXiv preprint arXiv:2111.13587, .
  * Gupta et al. [2021] Gupta, G., Xiao, X., & Bogdan, P. (2021).  Multiwavelet-based operator learning for differential equations.  Advances in Neural Information Processing Systems, 34, 24048–24062.
  * Guss & Salakhutdinov [2019] Guss, W. H., & Salakhutdinov, R. (2019).  On universal approximation by neural networks with uniform guarantees on approximation of infinite dimensional maps.  arXiv preprint arXiv:1910.01545, .
  * Hornik et al. [1989] Hornik, K., Stinchcombe, M., & White, H. (1989).  Multilayer feedforward networks are universal approximators.  Neural networks, 2, 359–366.
  * Hrennikoff [1941] Hrennikoff, A. (1941).  Solution of problems of elasticity by the framework method, .
  * Hughes [2012] Hughes, T. J. (2012).  The finite element method: linear static and dynamic finite element analysis.  Courier Corporation.
  * Jentzen et al. [2023] Jentzen, A., Riekert, A., & von Wurstemberger, P. (2023).  Algorithmically designed artificial neural networks (adanns): Higher order deep operator learning for parametric partial differential equations.  arXiv preprint arXiv:2302.03286, .
  * Jiang et al. [2021] Jiang, P., Meinert, N., Jordão, H., Weisser, C., Holgate, S., Lavin, A., Lütjens, B., Newman, D., Wainwright, H., Walker, C. et al. (2021).  Digital twin earth–coasts: Developing a fast and physics-informed surrogate model for coastal floods via neural operators.  arXiv preprint arXiv:2110.07100, .
  * Jordan & Jordán [1965] Jordan, C., & Jordán, K. (1965).  Calculus of finite differences volume 33.  American Mathematical Soc.
  * Kashinath et al. [2021] Kashinath, K., Mustafa, M., Albert, A., Wu, J., Jiang, C., Esmaeilzadeh, S., Azizzadenesheli, K., Wang, R., Chattopadhyay, A., Singh, A. et al. (2021).  Physics-informed machine learning: case studies for weather and climate modelling.  Philosophical Transactions of the Royal Society A, 379, 20200093.
  * Kontolati et al. [2023] Kontolati, K., Goswami, S., Shields, M. D., & Karniadakis, G. E. (2023).  On the influence of over-parameterization in manifold based surrogates and deep neural operators.  Journal of Computational Physics, 479, 112008.
  * Kovachki et al. [2021] Kovachki, N., Li, Z., Liu, B., Azizzadenesheli, K., Bhattacharya, K., Stuart, A., & Anandkumar, A. (2021).  Neural operator: Learning maps between function spaces.  arXiv preprint arXiv:2108.08481, .
  * Krishnapriyan et al. [2021] Krishnapriyan, A., Gholami, A., Zhe, S., Kirby, R., & Mahoney, M. W. (2021).  Characterizing possible failure modes in physics-informed neural networks.  Advances in Neural Information Processing Systems, 34, 26548–26560.
  * Lagaris et al. [1998] Lagaris, I. E., Likas, A., & Fotiadis, D. I. (1998).  Artificial neural networks for solving ordinary and partial differential equations.  IEEE transactions on neural networks, 9, 987–1000.
  * LeCun et al. [2015] LeCun, Y., Bengio, Y., & Hinton, G. (2015).  Deep learning.  nature, 521, 436–444.
  * Li et al. [2017] Li, S., Cui, X., & Li, G. (2017).  Multi-physics analysis of electromagnetic forming process using an edge-based smoothed finite element method.  International Journal of Mechanical Sciences, 134, 244–252.
  * Li et al. [2022] Li, Z., Huang, D. Z., Liu, B., & Anandkumar, A. (2022).  Fourier neural operator with learned deformations for pdes on general geometries.  arXiv preprint arXiv:2207.05209, .
  * Li et al. [2020a] Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., & Anandkumar, A. (2020a).  Fourier neural operator for parametric partial differential equations.  arXiv preprint arXiv:2010.08895, .
  * Li et al. [2020b] Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., & Anandkumar, A. (2020b).  Neural operator: Graph kernel network for partial differential equations.  arXiv preprint arXiv:2003.03485, .
  * Li et al. [2021a] Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., & Anandkumar, A. (2021a).  Markov neural operators for learning chaotic systems.  arXiv preprint arXiv:2106.06898, .
  * Li et al. [2020c] Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Stuart, A., Bhattacharya, K., & Anandkumar, A. (2020c).  Multipole graph neural operator for parametric partial differential equations.  Advances in Neural Information Processing Systems, 33, 6755–6766.
  * Li et al. [2021b] Li, Z., Zheng, H., Kovachki, N., Jin, D., Chen, H., Liu, B., Azizzadenesheli, K., & Anandkumar, A. (2021b).  Physics-informed neural operator for learning partial differential equations.  arXiv preprint arXiv:2111.03794, .
  * Liu et al. [2022] Liu, B., Kovachki, N., Li, Z., Azizzadenesheli, K., Anandkumar, A., Stuart, A. M., & Bhattacharya, K. (2022).  A learning-based multiscale method and its application to inelastic impact problems.  Journal of the Mechanics and Physics of Solids, 158, 104668.
  * Liu & Cai [2021] Liu, L., & Cai, W. (2021).  Multiscale deeponet for nonlinear operators in oscillatory function spaces for building seismic wave responses.  arXiv preprint arXiv:2111.04860, .
  * Livingstone et al. [1997] Livingstone, D. J., Manallack, D. T., & Tetko, I. V. (1997).  Data modelling with neural networks: advantages and limitations.  Journal of computer-aided molecular design, 11, 135–142.
  * Lu et al. [2019] Lu, L., Jin, P., & Karniadakis, G. E. (2019).  Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators.  arXiv preprint arXiv:1910.03193, .
  * Mao et al. [2020] Mao, Z., Jagtap, A. D., & Karniadakis, G. E. (2020).  Physics-informed neural networks for high-speed flows.  Computer Methods in Applied Mechanics and Engineering, 360, 112789.
  * Mavi et al. [2023] Mavi, A., Bekar, A. C., Haghighat, E., & Madenci, E. (2023).  An unsupervised latent/output physics-informed convolutional-lstm network for solving partial differential equations using peridynamic differential operator.  Computer Methods in Applied Mechanics and Engineering, 407, 115944. URL: https://www.sciencedirect.com/science/article/pii/S0045782523000671. doi:https://doi.org/10.1016/j.cma.2023.115944.
  * Moczo et al. [2004] Moczo, P., Kristek, J., & Halada, L. (2004).  The finite-difference method for seismologists.  An Introduction, 161.
  * Musha & Higuchi [1978] Musha, T., & Higuchi, H. (1978).  Traffic current fluctuation and the burgers equation.  Japanese journal of applied physics, 17, 811.
  * Naranjo-Torres et al. [2020] Naranjo-Torres, J., Mora, M., Hernández-García, R., Barrientos, R. J., Fredes, C., & Valenzuela, A. (2020).  A review of convolutional neural network applied to fruit image processing.  Applied Sciences, 10, 3443.
  * O’Shea & Nash [2015] O’Shea, K., & Nash, R. (2015).  An introduction to convolutional neural networks.  arXiv preprint arXiv:1511.08458, .
  * Özişik et al. [2017] Özişik, M. N., Orlande, H. R., Colaco, M. J., & Cotta, R. M. (2017).  Finite difference methods in heat transfer.  CRC press.
  * Pathak et al. [2022] Pathak, J., Subramanian, S., Harrington, P., Raja, S., Chattopadhyay, A., Mardani, M., Kurth, T., Hall, D., Li, Z., Azizzadenesheli, K. et al. (2022).  Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators.  arXiv preprint arXiv:2202.11214, .
  * Prasthofer et al. [2022] Prasthofer, M., De Ryck, T., & Mishra, S. (2022).  Variable-input deep operator networks.  arXiv preprint arXiv:2205.11404, .
  * Rahman et al. [2022a] Rahman, M. A., Florez, M. A., Anandkumar, A., Ross, Z. E., & Azizzadenesheli, K. (2022a).  Generative adversarial neural operators.  arXiv preprint arXiv:2205.03017, .
  * Rahman et al. [2022b] Rahman, M. A., Ghosh, J., Viswanath, H., Azizzadenesheli, K., & Bera, A. (2022b).  Pacmo: Partner dependent human motion generation in dyadic human activity using neural operators.  arXiv preprint arXiv:2211.16210, .
  * Rahman et al. [2022c] Rahman, M. A., Ross, Z. E., & Azizzadenesheli, K. (2022c).  U-no: U-shaped neural operators.  arXiv e-prints, (pp. arXiv–2204).
  * Raissi et al. [2019] Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019).  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  Journal of Computational physics, 378, 686–707.
  * Reddy [2019] Reddy, J. N. (2019).  Introduction to the finite element method.  McGraw-Hill Education.
  * Ren et al. [2022] Ren, P., Rao, C., Liu, Y., Wang, J.-X., & Sun, H. (2022).  Phycrnet: Physics-informed convolutional-recurrent network for solving spatiotemporal pdes.  Computer Methods in Applied Mechanics and Engineering, 389, 114399.
  * Renn et al. [2023] Renn, P. I., Wang, C., Lale, S., Li, Z., Anandkumar, A., & Gharib, M. (2023).  Forecasting subcritical cylinder wakes with fourier neural operators.  arXiv preprint arXiv:2301.08290, .
  * Rosofsky & Huerta [2022] Rosofsky, S. G., & Huerta, E. A. (2022).  Applications of physics informed neural operators.  arXiv preprint arXiv:2203.12634, .
  * Roters et al. [2011] Roters, F., Eisenlohr, P., Bieler, T. R., & Raabe, D. (2011).  Crystal plasticity finite element methods: in materials science and engineering.  John Wiley & Sons.
  * Salvi et al. [2021] Salvi, C., Lemercier, M., & Gerasimovics, A. (2021).  Neural stochastic pdes: Resolution-invariant learning of continuous spatiotemporal dynamics.  arXiv preprint arXiv:2110.10249, .
  * Saunshi et al. [2020] Saunshi, N., Zhang, Y., Khodak, M., & Arora, S. (2020).  A sample complexity separation between non-convex and convex meta-learning.  In International Conference on Machine Learning (pp. 8512–8521).  PMLR.
  * Sazli [2006] Sazli, M. H. (2006).  A brief review of feed-forward neural networks.  Communications Faculty of Sciences University of Ankara Series A2-A3 Physical Sciences and Engineering, 50.
  * Searles & von Nagy-Felsobuki [1988] Searles, D. J., & von Nagy-Felsobuki, E. I. (1988).  Numerical experiments in quantum physics: Finite-element method.  American Journal of Physics, 56, 444–448.
  * Settles [2012] Settles, B. (2012).  Active learning.  Synthesis lectures on artificial intelligence and machine learning, 6, 1–114.
  * Sharma Priyadarshini et al. [2021] Sharma Priyadarshini, M., Venturi, S., & Panesi, M. (2021).  Application of deeponet to model inelastic scattering probabilities in air mixtures.  In AIAA AVIATION 2021 FORUM (p. 3144).
  * Sirignano & Spiliopoulos [2018] Sirignano, J., & Spiliopoulos, K. (2018).  Dgm: A deep learning algorithm for solving partial differential equations.  Journal of computational physics, 375, 1339–1364.
  * Slone et al. [2003] Slone, A., Bailey, C., & Cross, M. (2003).  Dynamic solid mechanics using finite volume methods.  Applied mathematical modelling, 27, 69–87.
  * Smith et al. [1985] Smith, G. D., Smith, G. D., & Smith, G. D. S. (1985).  Numerical solution of partial differential equations: finite difference methods.  Oxford university press.
  * Sod [1978] Sod, G. A. (1978).  A survey of several finite difference methods for systems of nonlinear hyperbolic conservation laws.  Journal of computational physics, 27, 1–31.
  * Tadmor [2012] Tadmor, E. (2012).  A review of numerical methods for nonlinear partial differential equations.  Bulletin of the American Mathematical Society, 49, 507–554.
  * Taigbenu [1999] Taigbenu, A. E. (1999).  Burgers equation.  In The Green Element Method (pp. 195–216).  Boston, MA: Springer US.  URL: https://doi.org/10.1007/978-1-4757-6738-4_7. doi:10.1007/978-1-4757-6738-4_7.
  * Thais et al. [2022] Thais, S., Calafiura, P., Chachamis, G., DeZoort, G., Duarte, J., Ganguly, S., Kagan, M., Murnane, D., Neubauer, M. S., & Terao, K. (2022).  Graph neural networks in particle physics: Implementations, innovations, and challenges.  arXiv preprint arXiv:2203.12852, .
  * Trifan et al. [2021] Trifan, A., Gorgun, D., Li, Z., Brace, A., Zvyagin, M., Ma, H., Clyde, A., Clark, D., Salim, M., Hardy, D. J. et al. (2021).  Intelligent resolution: Integrating cryo-em with ai-driven multi-resolution simulations to observe the sars-cov-2 replication-transcription machinery in action.  bioRxiv, .
  * Viswanath et al. [2022] Viswanath, H., Rahman, M. A., Bhaskara, R., & Bera, A. (2022).  Nio: Lightweight neural operator-based architecture for video frame interpolation.  arXiv preprint arXiv:2211.10791, .
  * Wang et al. [2021] Wang, S., Wang, H., & Perdikaris, P. (2021).  Learning the solution operator of parametric partial differential equations with physics-informed deeponets.  Science advances, 7, eabi8605.
  * Wen et al. [2022] Wen, G., Li, Z., Azizzadenesheli, K., Anandkumar, A., & Benson, S. M. (2022).  U-fno—an enhanced fourier neural operator-based deep-learning model for multiphase flow.  Advances in Water Resources, 163, 104180.
  * Wilson et al. [1974] Wilson, E., Bathe, K., & Peterson, F. (1974).  Finite element analysis of linear and nonlinear heat transfer.  Nuclear engineering and design, 29, 110–124.
  * Winovich et al. [2019] Winovich, N., Ramani, K., & Lin, G. (2019).  Convpde-uq: Convolutional neural networks with quantified uncertainty for heterogeneous elliptic partial differential equations on varied domains.  Journal of Computational Physics, 394, 263–279.
  * Yang et al. [2021] Yang, Y., Gao, A. F., Castellanos, J. C., Ross, Z. E., Azizzadenesheli, K., & Clayton, R. W. (2021).  Seismic wave propagation and inversion with neural operators.  The Seismic Record, 1, 126–134.
  * Yegnanarayana [2009] Yegnanarayana, B. (2009).  Artificial neural networks.  PHI Learning Pvt. Ltd.
  * Yin et al. [2022] Yin, M., Ban, E., Rego, B. V., Zhang, E., Cavinato, C., Humphrey, J. D., & Em Karniadakis, G. (2022).  Simulating progressive intramural damage leading to aortic dissection using deeponet: an operator–regression neural network.  Journal of the Royal Society Interface, 19, 20210670.
  * You et al. [2022] You, H., Zhang, Q., Ross, C. J., Lee, C.-H., & Yu, Y. (2022).  Learning deep implicit fourier neural operators (ifnos) with applications to heterogeneous material modeling.  arXiv preprint arXiv:2203.08205, .
  * Yu et al. [2022] Yu, J., Lu, L., Meng, X., & Karniadakis, G. E. (2022).  Gradient-enhanced physics-informed neural networks for forward and inverse pde problems.  Computer Methods in Applied Mechanics and Engineering, 393, 114823.
  * Zhu et al. [2023] Zhu, M., Zhang, H., Jiao, A., Karniadakis, G. E., & Lu, L. (2023).  Reliable extrapolation of deep neural operators informed by physics or sparse observations.  Computer Methods in Applied Mechanics and Engineering, 412, 116064.
  * Zienkiewicz et al. [2005] Zienkiewicz, O. C., Taylor, R. L., & Zhu, J. Z. (2005).  The finite element method: its basis and fundamentals.  Elsevier.

##  10 Appendix

Model | Link
---|---
FNO | https://github.com/neural-operator/fourier_neural_operator
FourCastNet | https://github.com/NVlabs/FourCastNet
GANO | https://github.com/kazizzad/GANO
geo-FNO | https://github.com/neural-operator/Geo-FNO
GNO | https://github.com/neural-operator/graph-pde
MWT-NO | https://github.com/gaurav71531/mwt-operator
PINO | https://github.com/neural-operator/PINO
SNO | https://github.com/vlsf/sno
UNO | https://github.com/ashiq24/UNO

Table 6: This table provides links to the source code for various neural
operator architectures

Term | Meaning
---|---
A | Banach Function Space
a(x) | Input function
$\alpha$ | Hyperparameter for neural network
b | Bias
$\beta$ | Hyperparameter for neural network
C | Specific heat capacity
$\mathcal{C}$ | Cost Function
c | Speed of Sound
D | Domain
$d_{i}$, $b_{i}$ | Finite dimensional vectors
$e_{xy}$ | Edge from node x to node y in a graph neural network
$\mathcal{F}$ | Fourier Transform
f(x) | Function
$\mathcal{F}^{-1}$ | Inverse Fourier Transform
G | neural operator
$G^{+}$ | Non-Linear map between Function Spaces
$g_{i}$ | Complex Exponential/Chebyshev polynomial
$h_{i}$ | Hidden layer Embedding
k | directional Thermal Conductivity
K, $\kappa$ | Kernel
L | Banach Function Space
$\mathbf{L}$ | Linear Differential Operator
$\mathcal{L}$ | Loss Function
$\lambda$ | Directional Thermal Conductivity
N | Natural Numbers
n | Arbitrary Natural number
P | Lifting Map
p(r,t) | photoacoustic pressure wave at position r, time t
$\mathcal{P}$ | Partial Differential Operator
$\Phi$ | flux
$\varphi$ | Porosity
$\mathcal{Q}$ | internal heat source
Q | Projecting Map
R | Non-Linear Partial Differential Operator
$\mathcal{R}$ | Real Numbers
$\rho$ | density
$\varsigma$ | stress tensor
$S_{p}$ | Saturation of phase p
$\sigma$ | Non-Linear Activation
$\mathbf{T}$ | temperature
t, T | time
U | Banach Function Space
u(x) | solution function
$\mathcal{U}$ | Velocity
$u_{\theta}$ | neural network parameterized by $\theta$
$v_{i}$, $v^{(i)}$ | ith output of a neural network
$\mathbf{V}$ | neural network Approximation of Solution
$\nu$ | Viscosity coefficient
W | Weight
$\mathcal{W}$ | Weiner Process
w | Vorticity
X | mass fraction
$\xi$ | noise
$x_{i}$ | ith input element
$\nabla$ | Gradient Operator
$\Delta$ | Laplacian Operator

Table 7: A summary of mathematical notations used in the article