*Omer San, School of Mechanical & Aerospace Engineering, Oklahoma State University, Stillwater, OK 74078, USA.

# Hybrid analysis and modeling, eclecticism, and multifidelity computing
toward digital twin revolution

Omer San  Adil Rasheed  Trond Kvamsdal  School of Mechanical & Aerospace
Engineering, Oklahoma State University, Stillwater, OK 74078, USA  Department
of Engineering Cybernetics, Norwegian University of Science and Technology,
7465 Trondheim, Norway  Department of Mathematical Sciences, Norwegian
University of Science and Technology, 7491 Trondheim, Norway  Department of
Mathematics and Cybernetics, SINTEF Digital, 7034 Trondheim, Norway
<EMAIL_ADDRESS>
###### Abstract

Most modeling approaches lie in either of the two categories: physics-based or
data-driven. Recently, a third approach which is a combination of these
deterministic and statistical models is emerging for scientific applications.
To leverage these developments, our aim in this perspective paper is centered
around exploring numerous principle concepts to address the challenges of (i)
trustworthiness and generalizability in developing data-driven models to shed
light on understanding the fundamental trade-offs in their accuracy and
efficiency, and (ii) seamless integration of interface learning and
multifidelity coupling approaches that transfer and represent information
between different entities, particularly when different scales are governed by
different physics, each operating on a different level of abstraction.
Addressing these challenges could enable the revolution of digital twin
technologies for scientific and engineering applications.

###### keywords:

hybrid analysis and modelling, scientific machine learning, neurophysical
computing, digital twin, interface learning, reduced order modeling

††articletype: Article

##  1 Introduction

According to Peter Shaver 1 there have been two monumental steps in the rise
of science: 1) The qualitative and descriptive worldview of Aristotle, 2) The
quantitative and predictive worldview of Newton. Newton’s laws of motion is
the foundation for physics-based models in engineering and earth science that
we now may complement with artificial intelligence (AI) and machine learning
(ML) techniques through interaction with the physical reality in question by
means of real-time sensor data. We consider the combined use of physics-based
and data-driven models, herein denoted _hybrid analysis and modeling_ (HAM),
as a third monumental step of science characterised by being interactive and
evolving. _Interactive_ in the sense that the combined model is based both on
a priori human perception of the physical reality as well as on real-time
experimental (through sensors) observations. Evolving in the sense of changing
with time according to more available observational data utilised by the data-
driven methods as well as changes of the involved objects (e.g. degradation)
and processes. A key ingredient in our perspective is to develop HAM tools for
scientific and engineering applications and make these techniques available
for industry to be used in design and operation by means of developing highly
capable _digital twins_ (DTs). We postulate and foresee that the HAM approach
will be of disruptive nature for _computational science and engineering_ (CSE)
and initiate an exponential growth of scientific/engineering knowledge and
innovations, and that DTs will be the natural means to facilitate the take up
of the new knowledge and innovations into design and operation by the
industry.

Although a healthy criticism about _data-driven modeling_ goes around in
scientific communications, ML has started to become a new tradition in many
scientific disciplines. If we are not able to understand, let’s learn it!
However, if not well grounded, such data-driven learning might yield a
_perpetual engine_ that can be viewed as a sort of _fishing expedition_ ,
especially for modeling large scale systems. Moreover, if there is already a
_white-box_ principled procedure available to model a certain process, it is
arguably safer that we should avoid using a black-box approach that is blindly
reconstructed from data. On the other hand, the trade-off between accuracy and
efficiency could be a game changer for numerous applications. Yet, for
example, it might not be that far from today to use applications in our smart
phones to contribute value of crowd sourced data in improving operational
weather forecasts. More importantly, in some cases, we might lack having a
principled approach for an underlying process, hence data becomes essential to
build a model. Crucially, we highlight that the data-driven models need to
possess important characteristics (e.g., symmetry, conservation, stability,
and scalability properties) in order to revamp the existing predictive tools.
To address some of these challenges, here we look into _eclecticism_ as a key
computing philosophy, and explore an emerging HAM framework that combines the
generalizability and interpretability of physics-based models with the
versatility of data-driven modeling. This can be viewed as a divide-compute-
merge-conquer approach, for which the _compute_ part highly benefits from the
advances in computational mathematics, and the _merge_ part utilizes the
evolutionary shift in modern computational infrastructures to effectively
account for representation losses and closures in multi-x workflows. Herein
the _multi-x_ refers to the systems where the underlying models involve
multiple characteristic scales, multiple spatiotemporal domains, multiple
physical laws, and even multiple disciplines. _Divide_ part might consist
clustering, spatial or spatio-temporal partitioning, and we term _conquer_ in
referring to the acceleration of the overall process or improving its system
level predictive performance. For example, it is very common to use a coarse-
grid solver to resolve the atmospheric boundary layer flow and employ the
fine-grid solver to resolve the flow field around wind turbines in the wind
farm. For such problems, it is very important to exchange the information
between two solvers that leads to physically consistent boundary conditions,
and statistical tools like deep learning can be exploited for coupling
different solvers and mathematical descriptions.

In a nutshell, our prospective survey aims at exploring imminent algorithmic
needs in _predictive engineering_ for multi-x systems; complex natural or
engineered systems comprising multiple physical or computational heterogeneous
entities. Such predictive engines serve as the inner-workhorse for outer-
workflow loops such as optimal design, control, and discovery. While immense
advances in computational mathematics and scientific computing have come to
fruition, these methods often suffer a curse of dimensionality limiting
turnaround. The field of _multifidelity computing_ therefore aims to address
this computational challenge by exploiting the relationship between high-
fidelity and low-fidelity models. Here, we focus on physical systems where the
computational domain consists of a set of nested hierarchical models or
solvers. Examples range from systems within the Earth’s core, its shallow
subsurface and atmosphere, and beyond. A grand challenge in modeling and
computing such complex multi-x systems is the difficulty of determining proper
interface boundary conditions, and consistent restriction and prolongation
operators among solvers. We often come to a point in modeling where we ask the
questions about the legitimate ways to exploit the use of statistical
inference algorithms to fuse two heterogeneous computational entities having
an interface, which often consist of nested models, multiple physics, and
parametric and spatiotemporal heterogeneity. This survey intends primarily to
share our perspectives on some of these potential questions.

Eclecticism, according to the Cambridge English Dictionary, refers to a
conceptual philosophical approach that does not hold rigidly to a single
paradigm or set of assumptions, but instead draws upon multiple theories,
formulations, or ideas to gain complementary insights into a subject, or
applies different theories in particular cases. In computational sciences,
eclecticism may mean combining whatever seems the best or most useful to
improve predictions against limited computational resources. This can be
effectively achieved using a _domain decomposition_ approach 2, 3, 4, 5, 6, 7,
8, 9, 10, 11, 12, 13, 14, 15, 16, a powerful concept often used to solve
partial differential equations (PDEs) with complex geometries. Here, we can
interpret the term _domain decomposition_ in a broad sense such that it also
includes the simultaneous advances in the multiscale and multiphysics problems
17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28. Many approaches in the field
of domain decomposition are well-suited for heterogeneous problems, which have
dissimilar governing equations or models (a.k.a. abstractions) in different
regions of interest.

In terms of classification, two main coupling approaches are devised to merge
these abstractions 29: (i) the monolithic or intrusive framework 30, 31, 32,
33, 34, 35 that generates a unified system for the coupled problem, and (ii)
the partitioned or non-intrusive framework 36, 37, 38, 39, 40, 41, 42 in which
we solve multiple abstractions separately, and exchange interface boundary
conditions between them (a.k.a. transmission conditions) using sequential or
parallel Schwarz iterations 43, 44, 45. In practice, the latter is often
preferred due to the flexibility of using pre-existing solvers for each
abstraction, which is also the approach we will pursue in accomplishing
_scalable eclecticism_. Two conflicts are in order. First, the iterative
nature of this approach poses significant challenges on the scalability of the
ultimate joint solver, even though an extreme scalability may be already
presented in the individual abstractions. Second, although optimal
transmission conditions can be derived for linear or simplified model systems
46, 47, 48, 49, 50, 51, 52, it is often unclear how good these transmission
conditions are for realistic applications, especially when we consider their
accuracy, stability, and convergence rate characteristics. However, these
conflicts offer new challenges as well as opportunities for researchers
working on the next-generation statistical inference approaches for scientific
applications. We arguably hypothesize that it might be viable to introduce
statistical inference tools into multi-x workflows to develop robust
predictive engines that require no iteration between such abstractions. Within
the context of domain decomposition, we refer readers to recent discussions on
exploratory studies on ML to couple numerical solutions of partial
differential equations 53, 54. Moreover, neural networks have been utilized to
predict the geometric location interfaces in developing adaptive domain
decomposition methods 55. As highlighted in our recent works 56, 57, ML tools
can be effectively utilized in various forms of interfacial error correction
to form the building blocks of an integrated approach among mixed fidelity
descriptions toward predictive DT technologies.

From modeling perspective, overall, two distinct approaches are in vogue in
the scientific community today: physics-based modeling and data-driven
modeling. Both these approaches have their own strengths and weaknesses and
hence in the present paper we argue for a need to combine these two modeling
approaches in ways that benefit from the best of both approaches. To this end,
we aim to highlight tentative ways of combining them. We also stress that the
coupling of these complementary approaches in an eclectic way also poses novel
computational, mathematical and phenomenological issues that should be
explored. So far, the computational science and engineering community has been
driven mostly by a _physics-based modeling_ approach. This approach consists
of observing physical phenomena, developing approximate mathematical models
from first principles, and ultimately solving them. We often observe only a
part of the physics, and model even a smaller part of underlying phenomena. To
make the simulations computationally tractable, more assumptions are made
leading to further loss of physics. In any physics-based approach, therefore,
we take into account only a fraction of all the physical processes, and that
partly explains the mismatch between predictions and observations. On the
other hand, we have a purely _data-driven modeling_ approach that thrives on
the availability of data. It is expected that data is a manifestation of all
the physics driving a particular process and hence even without any knowledge
of the governing physics, one can model their effects. This approach, which
was challenged until recently, is now gaining attention primarily because of
the access to unprecedented amounts of data, open source cutting-edge ML and
data analytics libraries, and computational resources. Together with their
modularity and simplicity, data-driven models also offer a unique advantage in
multidisciplinary collaborative environments and provide richer platforms
securing intangible assets and intellectual property rights 58, 59. As we
highlight in our technology watch survey 60, such an interconnected HAM
paradigm constitutes a key enabler for the emerging DT revolution. In this
perspective letter, we aim to provide an overview of HAM strategies relevant
to scientific and engineering applications. The topic spans a wide spectrum,
and there are a great number of review articles on relevant discussions,
methodologies and applications 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72,
73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91,
92, 93, 94, 95, 96, 97, 98. Therefore, it is not our intention to give a
complete biography, but rather present somehow our _subjective_ perspectives
with an emphasis on the emerging methodologies and enabling technologies from
modeling perspectives. In particular, there has been recent interest in _big
data_ , AI, and ML applications in numerous disciplines, and therefore our
survey is intended to encourage cross-disciplinary efforts between many
stakeholders including practitioners, physicists, mathematicians and data
scientists.

We emphasize that advanced ML models like deep learning networks are powerful
tools for finding patterns in complicated datasets. However, as these
_universal approximators_99, 100, 101, 102 become complex, the number of
trainable parameters (weights) quickly explodes, adversely affecting their
interpretability and hence their trustworthiness. Using these models in
combination with other traditional models compromises the trustworthiness of
the overall system. Some of the active areas of research in this context are:
cost function modification to accommodate the model Jacobian 103, grow-when-
required network 104, physics-informed neural networks (PINNs) 105, 106, 107,
embedding _hard_ physical constraints in a neural network108, leveraging
uncertainty information 109, developing visualization tool for the network
analysis 110, and nonparametric modeling approaches for bridging data science
and dynamical systems 111. While these techniques have been emerging in both
scientific computing and ML fields, they offer many opportunities to fuse
topics in numerical linear algebra and theoretical computer science 112
towards improving the explainability of these models and implementing built-in
sanity checks on the overall system where these models are employed. With this
in mind, we present our perspectives to provide a glimpse into this exciting
field, which came to known as _HAM_.

This paper is organized as follows. In Section 2, we describe how HAM
approaches can tackle important issues in physics-based or data-driven models,
for instance, how the generalization properties of data-driven models can be
improved by using domain knowledge. Sections 3 and 4 are devoted to two
promising HAM frameworks, physics-guided ML and interface learning,
respectively. Sections 5 and 6 illustrate the application of HAM approaches in
reduced order modeling and big data cybernetics, and Section 7 describes the
idea of DTs and different capability stages of DTs. The final section then
gives concluding remarks and an outlook with remarks on open topics in the
field of HAM.

##  2 Hybrid analysis and modeling

_Quo vadis_ in modeling: physics-based or data-driven? One camp develops and
improves first principle models that provide the best generalizability and
trustworthiness characteristics. In the other camp, researchers try to explain
phenomena from archival data using statistical approaches. With the abundant
supply of big data, open-source cutting edge and easy-to-use ML libraries,
data-driven modeling has become very popular. Compared to the physics-based
modeling approach, these models thrive on the assumption that data is a
manifestation of both known and unknown physics and hence when trained with an
ample amount of data, the data-driven models might learn the full physics on
their own. This approach, involving in particular deep learning, has started
achieving human-level performance in several tasks. Notable among these are
image classification 113, 114, 115, 116, dimensionality reduction 117, medical
treatment 118, 119, 120, 121, smart agriculture 122, physical sciences 64,
123, 124 and beyond. Some of the advantages of these models are online
learning capability, computational efficiency for inference, accuracy even for
very challenging problems as far as the training, validation and test data are
prepared properly. However, due to their data-hungry and black-box nature,
poor generalizability, inherent bias and lack of robust theory for the
analysis of model stability, their acceptability in multi-x systems is fairly
limited. In fact, the numerous vulnerabilities of deep neural networks have
been exposed beyond doubt in several recent works 125, 126, 127. To address
such challenges, we explore a new HAM approach based on a synergistic
combination of deterministic and statistical model components.

We term HAM as a modeling approach that combines the interpretability, robust
foundation and understanding of a physics-based approach with the accuracy,
efficiency, and automatic pattern-identification capabilities of advanced
data-driven ML and artificial intelligence algorithms. Our overall hypothesis
for developing hybrid methodologies is to form a solid foundation that
balances technology push and domain-specific application pull. In particular,
we consider the combined use of physics-based and data-driven models in a
broad sense, hereafter denoted as HAM, as a science and engineering disruptor
that can be discerned in Figure 1. In this figure, we display the different
ideas that can be considered the pillars of the HAM paradigm. The interface
learning framework allows us to take advantage of models of different levels
of fidelity to optimize the available computational resources. The
neurophysical modeling framework aims at bringing the best of physics-based
and data-driven approaches to deliver computationally economical and
generalizable robust models. The reduced order modeling framework can
facilitate the construction of efficient surrogate models that can serve as
the inner-workhorse of outer loop applications like model predictive control.
The data assimilation framework lets us continuously update the models built
from the HAM paradigm by exploiting online measurements to reach the optimal
set point. Feature engineering, which is a process of using domain knowledge
to extract features from data, is critical to all these frameworks that guide
the overall design of hybrid models with the objective of bringing
trustworthiness in hybrid models. Finally, all these ideas can be interlinked
or utilized in isolation to bring realism to a DT of any physical system.

Figure 1: Overview of the _HAM_ paradigm as a science and engineering
disruptor while balancing technology push and domain-specific application
pull. As a core methodological facilitator in combining first principles with
data-driven approaches, the HAM framework benefits from the advances in both
modeling disciplines towards building more explainable and trustworthy
predictive tools. Since DTs often comprise heterogeneous computational
entities to maximize the utilization of available resources, the interface
learning paradigm allows seamless coupling of these individuals components
while neurophysical modeling approach aims at elevating the reliability and
computational efficiency of such entities by combining physics-based and data-
driven methodologies. For the latter, feature engineering improves their
interpretability by minimizing the number of trainable parameters and reducing
the size of search space. Data assimilation can also be viewed as one of the
well-established HAM algorithms that fuse models and data to cure the
deficiencies in both, which can be further implemented for prediction and
control purposes. Nonetheless, their inner workhorses are often
computationally expensive for realistic applications. Fortunately, reduced
order modeling provides light surrogates to mitigate this burden, where again
HAM plays an important role in improving the accuracy, stability, and
trustworthiness of these proxy models. The integration and synergy of all
these pave the way towards the realization of next-generation DT technologies
in the first place.

To illustrate the HAM concept at an abstract level, let’s consider a generic
prognostic equation relevant to general circulation models (GCMs):

$\frac{\partial\boldsymbol{u}}{\partial
t}+\mathcal{F}(\boldsymbol{u};\boldsymbol{x},t)=\mathcal{S}(\boldsymbol{u};\boldsymbol{x},t)$
(1)

where $\boldsymbol{u}$ is a three-dimensional (3D) dependent variable (e.g.,
density, velocity, temperature, moisture); $\boldsymbol{x}$ is a 3D
independent variable (i.e., latitude, longitude, height); $\mathcal{F}$ is the
model’s dynamical core (i.e., a semi-discretized function of PDEs of mass,
momentum, energy conservation laws); $\mathcal{S}$ is the model’s physics
(a.k.a. parametrization) that refers to physical and chemical processes (e.g.,
turbulence, precipitation, long and short wave atmospheric radiation, clouds,
constituency transport, and chemical reactions). Although most physical
processes have been included in GCMs as their simplified one-dimensional (1D)
parameterized versions (in the vertical direction), still computing
$\mathcal{S}$ takes about 70%–90% time of the total forward model computations
128, 129, 130, 131. Moreover, these simulations generate a huge volume of
climate data that poses storage challenges. This motivates us to develop a HAM
approach to improve the handling of $\mathcal{S}$ through the _discovery of
better parameterizations_ , and the _data-driven acceleration_ approaches in
the calculation of model physics components. Another major modeling challenge
that drives us to establish a bridge between physics-based models and data-
driven models is the difficulty of transferring source term effects from a
local model to the global model (e.g., turbulence flux, momentum flux, and
energy flux) or vice versa for which the global model provides boundary
conditions to the local one.

More specifically, we split $\mathcal{S}$ as

$\mathcal{S}(\boldsymbol{u};\boldsymbol{x},t)=\mathcal{D}(\boldsymbol{u};\boldsymbol{x},t)+\mathcal{N}(\boldsymbol{u};\boldsymbol{x},t)$
(2)

where $\mathcal{D}$ refers to the deterministic part of the physics that is
manageable to compute or already well established, and $\mathcal{N}$ refers to
uncertain processes representing model physics, or refers to a crude
approximation that can be improved. Recent studies by the authors 132, 133,
134, 135, 136 and other researchers 137, 138, 139, 140, 141, 142, 143, 144,
145, 146, 147, 148, 149, 150 show that feasible learning is possible for
$\mathcal{N}$ that improves the overall accuracy of the forward model, in
which a computational speed-up of several orders of magnitude has been
achieved. By replacing $\mathcal{N}$ with trainable neural networks and
symbolically discovered parameterizations, we can create a modular HAM
framework that offers opportunities in modeling and computing many complex
systems via developing (i) more sophisticated physical parameterizations, (ii)
more frequent incorporation of model physics, (iii) allowing the dynamical
core to run with higher resolution, (iv) easier ensemble simulations with
reduced uncertainties, and (v) alleviating computational complexity of running
adjoint (tangent linear) models when integrated within variational data
assimilation workflows. The outlook of the HAM approach can be established by
tackling the following key questions:

_How to manage and process data in real-time?_ Real-time control, for example,
is necessary for environmental situational awareness for improved decision-
making scenarios, such as wind-turbines adaptation 151. To achieve this, an
increasing number of observations should be assimilated instantaneously since
the lifespan of many weather phenomena can be less than a few minutes. Using a
full order model (FOM) 152, 153, 154 with the required resolution is generally
not compatible with the time-scales of natural phenomena in realistic
scenarios. Therefore, a computationally efficient _reduced order model_ (ROM)
155, 156, 157, 158, 159, 160, 161, 162, 79, 163, 164, 165, capable of
duplicating the physical system with acceptable accuracy, could offer a
realistic solution to assimilate these short-time observations. By using
adaptive finite element FOM166, 167, 168, 169, 170, 171, 172, 173, 174, 175,
176, 177, 178, 179, 180, 181, 182, 183, 184 (in the offline stage) together
with certified ROM185, 159, 186, 187, 188, 189, 190, 191, 192 (in the online
stage) we may indeed control the numerical discretization error within
prescribed tolerances. However, often the role of ROM is to give quick
guidelines on whether or not a decision needs to be taken. This is
specifically important given the intrinsic stochastic characteristics of
weather systems, since the model should be able to monitor those seemingly-
random sequential observations 193, 194, 195. The use of ROMs in real-time
simulations becomes a promising approach in several applications 196, 197,
198, 199. Data-driven tools do an excellent job in terms of online time-cost,
however, relying only on data and disregarding the known and well-established
physics is not a wise step. Hybridization techniques have been demonstrated to
give superior performance to either individual components 150, 129, 134, 137,
123, 200. Therefore, a DT based on HAM methodologies can be built to provide
nature-informed indicators for near-time events.

_How to improve the computational efficiency and accuracy of a model?_ In
order to gain online computational efficiency at the cost of offline
simulations, a physical kernel approach can be imposed in training ML models
to enable accurate but fast coarse scale simulation while still intelligently
modeling the representation losses and residuals using deep neural networks
(i.e., incorporating numerous functional forms of known closure kernels in the
training as opposed to static phenomenological models201, 136, 202, 203). The
training data can be also compressed by using both linear (e.g., matrix
factorization204) and nonlinear techniques (e.g., variational autoencoders205,
manifold learning206, 207, 208 and stochastic neighbor embedding
algorithms209, 210, 211), and the causality assessment tools212 can be
employed to explore the correlation between input kernels and output variables
in order to remove irrelevant input features from the training. These efforts
can immensely reduce the size of the training data. The overall imposition of
mass, momentum and energy conservation principles will be used as an inbuilt
sanity check mechanism on the computations made by the neural network.
Moreover, instability and inaccuracy resulting from the dimensionality
reduction in the proposed HAM models can be corrected by a long-short term
memory (LSTM) architecture as illustrated in our recent studies 59, 213. To
address transient characteristics, modal deformation and generalization of the
ROM across a wide range of scenarios, it is possible to enhance HAM by
utilizing principal interval decomposition 214, 215, 58 and Grassmann manifold
interpolation 216, 133, 132. One can also utilize a _method of manufactured
learning_ approach, which is primarily focused on devising statistical
sampling and upscaling strategies to efficiently learn neural network
architectures without requiring expensive simulations as training data. This
approach has roots in the _method of manufactured solutions_ 217 (i.e.,
generating an auxiliary source function for a given chosen solution to
validate the solver) that is often used to validate scientific codes. From
this perspective, HAM will be an effective tool for the unbiased analysis of
multi-x systems, especially with sparse datasets. The sparsity challenges can
be handled by incorporating generative adversarial networks (GANs)218. The
method of manufactured learning can leverage both mathematical functions
(e.g., Chebyshev, Dickens, wavelet families) or physical sets (e.g., proper
orthogonal decomposition modes 96, 219, 220, Lagrangian coherent structures
221, 222, 223, exact coherent structures 224, 225, 226, 227, terrain induced
vortices 228, 229, 230, traveling waves 231, 232, 233, periodic or relative
periodic orbits 234) to train $\mathcal{N}$ without requiring an expensive
“truth" labeled data. It is also pivotal when we construct self-evolving
surrogate models on locally embedded structures. In principle, this approach
can be exploited as a _new training mechanism_ for large data sets (e.g.,
geophysical turbulence) to address the generalizability and interpretability
concerns of any ML approaches.

_How to improve the trustworthiness of models?_ In addition to enriching the
input feature space as explained above, we can improve the trustworthiness of
data-driven parts by imposing constraints and physical laws on the methods
through the regularization of the cost function 105 as well as by developing a
scalable and optimal neural architecture search 235 approach for model
structure selection. To develop a more quantitative insight into the effects
of these approaches, we can exploit the use of piece-wise affine
representation of networks as shown in our recent work 236. Moreover, a
relatively straightforward approach for constructing error correction models
based on formulating a data assimilation procedure can be proceeded in order
to achieve long-term accurate prediction beyond training zone. To tackle
stability issues for the online deployment of data-driven closures, it is
possible to exploit the _sequential data assimilation_ techniques 237 to
correct the HAM predictions in Eq. 1. In particular, we can use a set of
neural network architectures to learn the correlation between prognostic
variable $\boldsymbol{u}$ and the data-driven parameterizations of
$\mathcal{N}$, and formulate a _nudging data assimilation_ approach to improve
long-range predictions in chaotic systems. The core idea behind nudging is to
penalize the dynamical model evolution with the discrepancy between the
model’s predictions and observations 238, 239, 240. In other words, the
forward model given in Eq. 1 is supplied with a nudging (or correction) term
rewritten in the following discrete form,

$\mathbf{u}^{n+1}=\mathbf{\mathfrak{M}}(\mathbf{u}^{n})+\mathbf{G}(\mathbf{z}^{n+1}-h(\mathbf{u}_{b}^{n+1})),$
(3)

where $\mathbf{\mathfrak{M}}$ is the forward model, $\mathbf{u}_{b}^{n+1}$ is
the prior model prediction computed using imperfect background model, defined
as $\mathbf{u}_{b}^{n+1}=\mathbf{\mathfrak{M}}(\mathbf{u}^{n})$, $\mathbf{G}$
is called the nudging (gain) matrix, $\mathbf{z}$ is the set of measurements,
and $n$ refers to the time index where we have these observations, while
$h(\cdot)$ is a mapping from model space to observation space. For example,
$h(\cdot)$ can be a reconstruction map, from ROM space to FOM space as shown
in our recent studies 241, 200. In other words, $h(\mathbf{u})$ represents the
“model forecast of the measured quantity”, while $\mathbf{z}$ is the “actual”
observations. Despite the simplicity of Eq. 3, the determination of the gain
matrix $\mathbf{G}$ is not as simple 242, 243, 244, 245. Here, instead, we can
define a functional nudging map $\mathbf{\mathfrak{C}}(\mathbf{u},\mathbf{z})$
as,

$\mathbf{u}^{n+1}=\mathbf{\mathfrak{M}}(\mathbf{u}^{n})+\mathbf{\mathfrak{C}}(\mathbf{u}_{b}^{n+1},\mathbf{z}^{n+1}),$
(4)

where the map $\mathbf{\mathfrak{C}}(\mathbf{u},\mathbf{z})$ will be inferred
(e.g., LSTM model). We have tested this _predictor-corrector nudging_
methodology in our recent studies 133, 246. In a demonstration for solving the
multiscale Lorenz 96 system247, the chaos contaminates the solution in a short
time when $\mathcal{N}$ is solely modeled by a neural network to account for
the fast fluctuating dynamics. However, we showed a significant improvement in
the long-term prediction of the underlying chaotic dynamics with our forecast
error correction framework empowered by a deterministic ensemble Kalman filer
(DEnKF) 248. For more detailed discussions on algorithms of such nonlinear
filtering frameworks, we refer readers to our recent comparative study of the
LSTM nudging and other sequential data assimilation methods for the Kuramoto-
Sivashinsky equation 249. Moreover, we demonstrated that the HAM approach can
handle the _non-Gaussian statistics_ of subgrid scale processes, and
effectively improve the accuracy of outer data assimilation workflow loops in
a modular non-intrusive way. While these preliminary results indicate the
basic viability of the approach, they do not cover many key aspects that need
to be addressed to scale this predictor-corrector nudging idea into more
realistic situations. It is also interesting to note that the similarities and
equivalence between ML and the formulation of statistical data assimilation as
used widely in physical and biological sciences have been postulated 250, 251.

_How to make generalizable models?_ The problem of building a model from the
data can be posed as a multi-objective optimization problem where several
aspects of the knowledge-injection processes must be involved in data
preparation, feature engineering, training and testing steps. One can
investigate how the reinforcement learning (RL) agent trained with an
environment with the coarse mesh behaves in the environment with fine
resolution. These efforts can be built on our recent works in RL architectures
252, 253, 254 to devise new approaches for transfer learning from a low-
fidelity model to a high-fidelity model. It would be a nice attempt to explore
constructing new reward functions that are more suited to multi-x environments
to take into account the efficient sampling of big data. A _design of
experiments_ methodology can be used to keep the number of numerical
experiments (required to develop HAM) as small as possible without
compromising the resulting quality. The agent will then be trained with the
new objective of adaptive sampling. Moreover, to put the learning of the agent
in the form of ordinary differential equations one can exploit the use of a
_symbolic regression_ method for hidden physics discovery 255. A symbolic
regression approach built on _gene expression programming_ 256 and _graph
networks_ 257 has an edge over other ML methods in the sense that it gives a
functional form of the mapping from the input to the output space making
stability analysis for the system possible. With these augmented RL agents one
can synthesize data and take advantage of the expressivity of symbolic and
multivariate regression tools to discover new parameterizations. As an
alternative approach, we can take advantage of _equation-free multiscale
methods_ 258, 259, 260 that allow us to compute a system on a level for which
no complete governing equation exists. This coarse graining approach (a.k.a.
_projective integration_) requires identification of the latent space that
represents the system without ever deriving it in a closed form. The success
of this approach often depends upon the existence of a separation of time
scales on the system dynamics. To determine when the errors introduced by the
coarse projective steps have healed, and therefore another projective step can
be taken, one can monitor, for example, energy spectra or structure functions
in fluid dynamics applications. This will provide us a sanity check mechanism
for the spatial, temporal and internal consistency of the ML outputs.

_How to develop self-evolving models which learn and adapt as-we-go?_ This can
be accomplished by blending new measurements through dynamic data assimilation
to correct and update HAM models. While one can continuously and recursively
update the reduced dimensional basis with the most recent observations, the
models can be made to adapt to recent changes while still exploiting knowledge
learned a long time in the past. To match the output of the hybrid model with
measurement data as-we-go, one of the key aspects in accelerating overall HAM
workflow is to create a robust data assimilation module that can also be
instrumental in sensitivity, robustness and error propagation analysis.
Moreover, the resulting hybrid model can be applied (i) as an inner-engine
with the data being observations streamed in through the outer-loop; and (ii)
as a surrogate for the linear solution process within physics-based nonlinear
solvers, where now the data stream is provided by the nonlinear solution
process itself. The new perturbations to the system will be captured using the
sparse data collected by the sensors to further develop our evolve-then-
correct reduced order modeling strategy 132. Invariably, numerical
approximations of transient and nonlinear phenomena can lead to sequences of
nonlinear algebraic systems. At the start of a simulation, an offline HAM can
act as a rough preconditioner for the linear solution process. As the
simulation progresses, the nonlinear iterates might act as the online data-
stream that the HAM assimilates, thereby creating a symbiosis. As the HAM
continues to learn and adapt, it can ultimately become as an efficient
preconditioner. Furthermore, low-rank approximations can be injected into a
numerical integration itself. Therefore, the target algebraic system remains
to be the full scale problem, while the nonlinear solution process itself
exploits a ROM representation.

_How to better quantify uncertainties?_ The decision-making is a complex
process for many disciplines. In recent decades, computational models have
shown the potential of supporting the decision, providing detailed information
that may not be directly acquirable in experiments. However, the sensitivity
of the results on the different sources of uncertainties is a critical piece
of information. Uncertainty quantification (UQ) therefore certifies the
reliability of computational models working with noisy data. However, a UQ
analysis often requires many simulations to estimate the unknown coefficients
(e.g., polynomial chaos expansion coefficients 261 with a set of quadrature
points). We can use reduced order models and data-driven tools to improve the
efficiency of the underlying decision-making process by performing a complete
UQ analysis in a reduced amount of time. We envision that the HAM approach can
significantly accelerate the UQ analysis to feasible CPU times (e.g., on the
order of minutes) on local computers. This becomes even more important in
certain fields (i.e., health care and biomedical applications), where the high
performance computing (HPC) facilities might not be easily accessible for
parallel computations. Moreover, sensitive data-protection issues can make
remote HPC simulations very challenging.

The above mentioned challenges in each question are somehow interrelated, so
progress in each front must build on advances in the other ones. Generally
speaking, the black-box parameterization often becomes an ill-posed problem
and might diverge when considering long duration integrations. There are
several remedies already available. Namely, one can explore including the mean
Jacobian approximation 262, weight smoothing 263 or other regularization
techniques, and cost function modifications 103, where the first order Sobolev
norm has been used in the training instead of the standard Euclidean norm.
When deeper neural network architectures are required, the complexity might
become a practical issue, e.g., in the GCM computations, since the learning of
the weights is very time-consuming and requires a large optimization problem
that is prone to yield sub-optimal solutions. To mitigate this one can utilize
the _extreme learning machine_ closure approach 264 in a recursive manner for
an easier training process (e.g., for re-training a part of $\mathcal{N}$ to
adapt it to the new environment).

To sum up, bringing physical realism in DTs of next-generation physical and
engineering systems will need (i) new modeling approaches that are accurate
and reliable, generalizable, computationally efficient, and trustworthy, and
(ii) seamless integration of multi-scale, multi-physics, and multi-fidelity
models (multi-x systems). To this end, we will highlight two HAM approaches in
the following sections that address the above needs. While the first approach,
physics-guided ML, provides a mechanism to inform the black-box ML models like
a deep neural network in which the first principles feed the training process
to enhance the predictive model’s trustworthiness, the second approach,
interface learning, enables the coupling of multi-x systems.

##  3 Neurophysical modeling and physics-guided ML

There is a growing interest in applying ML, particularly deep learning for
scientific applications due to the abundance of data generated either from
high-fidelity numerical simulations or sparse measurements. Despite the
success of data-driven models for many problems in scientific computing, they
lack the interpretability of the physics-based models. This motivates the need
to address the generalizability and data efficiency of the data-driven
approaches for physical systems. Therefore, one of the active research thrusts
is to leverage methodologies for the combination of physics-based and neural
network models, a rapidly emerging field that came to be known as physics-
guided AI (PGAI) or physics-guided ML (PGML). To this end, we have recently
introduced a PGML framework where information from simplified physics-based
models is incorporated within neural network architectures to improve the
generalizability of data-driven models 265. The central idea in the PGML
framework is to embed the knowledge from simplified theories directly into an
intermediate layer of the neural network as shown in Figure 2. The knowledge
from the simplified theories aids in ensuring that we learn only the knowledge
required to compensate for the deficiencies of these theories instead of
learning everything from scratch. Also owing to the fact that the simplified
theories are still based on the laws of nature they are more generalizable
compared to any data-driven approach and hence they should be exploited to the
extent possible. For example, the prediction of flow around an airfoil is a
high-dimensional, and nonlinear problem that can be solved using high-fidelity
methods like computational fluid dynamics (CFD). However, a wide variety of
problems such as aerodynamic performance prediction 266, 267 for optimal
control, design optimization, uncertainty quantification might require the
prediction of the quantity of interest in real-time, and these CFD methods
become computationally infeasible for these real-time prediction purposes. To
overcome these challenges, utilizing ML approaches to build a data-driven
surrogate model is gaining widespread popularity 268, 269. Yet one of the
well-known issues with neural networks is that these universal approximators
exhibit poor _generalizability_ , i.e., they produce inaccurate prediction
when the test data is from a distribution far from the training data. This
adversely affects the trustworthiness of neural networks for scientific
applications. Therefore, it is important to incorporate the domain knowledge
into learning, and one can exploit the relevant physics-based generalizable
features from simplified models (e.g., panel methods) through the PGML
framework to enhance the generalizability of data-driven surrogate model.

Figure 2: A physics-guided ML framework, where the physics-based features are
directly embedded into hidden layers of the neural network. As an illustrative
example to this framework, we can consider constructing a learning engine to
obtain a surrogate model in replacing CFD calculations for the aerodynamic
load prediction tasks in demand. Input layer could be an airfoil shape, and
output layer could be the lift and drag coefficients. We can construct our ML
model to incorporate physics into ML models through embedding simplified
theories (e.g., panel method as a fast physics-based predictive model)
directly into neural network models.

In a typical neural network architecture, the input vector
$\mathbf{x}\in\mathbb{R}^{m}$ is fed to the supervised ML model, and the
mapping from the input vector to the output vector
$\mathbf{y}\in\mathbb{R}^{n}$ is learned through training. The neural network
is trained to learn the function $F_{\theta}$, parameterized by $\theta$, that
includes the weights and biases of each neuron. The parameters of the neural
network are optimized using the backpropagation algorithm to minimize the cost
function. Usually, for the regression problems, the cost function is the mean
squared error between true and predicted output, i.e.,
$C(\mathbf{x},\theta)=||\mathbf{y}-F_{\theta}(\mathbf{x})||_{2}$. In the PGML
framework 265, the neural network can be augmented with the output of the
simplified physics-based model. The features extracted from simplified
physics-based models are embedded into hidden layers along with latent
variable. This is in contrast to admitting physics-based features at the input
layer in conventional neural network architectures, which might lead to
underestimation of the effect of such physical information, especially for
high dimensional systems. For example, imagine stacking a few parameters to an
input vector of a dimension of $O(10^{3}-10^{6})$ (e.g., coordinates
describing the shape of an airfoil or blade). It is highly probable that the
learning algorithm overlooks the effect of such prior knowledge (scalars like
the angle of attack and Reynolds number of the flow) in the minimization
algorithm. In the PGML, a proper latent space is first identified and the
information from the physics-based model is injected into the neural network.
Since the neural network is no more required to learn these features by itself
its complexity and associated model uncertainty can be significantly reduced.

As demonstrated in predicting aerodynamic loads 265, the PGML framework is
successful in significantly reducing the model uncertainty (captured using the
prediction from multiple models initialized with different parameters) of the
neural network. It has been shown that the PGML framework also enhances the
generalizability of the neural network based surrogate model since it predicts
the quantity of interests more accurately for unseen conditions. We highlight
that the PGML framework can be applied in different branches of science and
engineering, such as wind farm layout optimization, boundary layer flows,
where simplistic models are very common. The simplified models are cheap to
run and therefore the PGML framework can be attractive for applications such
as model predictive control, data assimilation, and uncertainty
quantification. The hybrid model built using the PGML framework can act as an
enabler for predictive simulations paving the way for the DT of physical
systems. In fact PGML can be combined with PINN 105, 106, 107 to exploit the
latter’s full potential. In PINN, the cost function is regularized using the
residual of physics-based equations. The regularization can make the task of
minimizing the cost function difficult specially when the complexity of neural
network is very high. By utilizing PGML one can reduce the complexity of the
model thereby making the optimization process more efficient.

##  4 Eclecticism and interface learning

In this section, we focus on the eclectic interface modeling framework for
multi-x systems. The HAM approaches can offer the key enablers to develop
multifidelity formulations that consist of multiple abstractions with
different characteristics. Unlike traditional iterative schemes, first of all
one can develop and explore the utility of a spectrum of HAM methodologies for
non-iterative interface learning. To expedite computations, different levels
of models and descriptions should be devoted to different zones and components
of the problem in order to allocate computational resources more effectively
and economically. This might be the case for many other coupled multiphysics
systems relevant to fluid dynamics, such as heterogeneous multiscale
problems260, 270, 271, 272, 273, 274, fluid-structure interaction 275, 154,
aerospatial 276, 277, 278, subsurface 279, 280, and wind farm applications
281. Since various zones in these systems are connected through interfaces,
data sharing and consistent interface treatment among respective models are
inevitable (e.g., Dirichlet-Neumann, Dirichlet-Robin, and Robin-Robin).
Likewise, multirate and locally adaptive stepping methods can yield a mismatch
at the space-time interface, and simple interpolation or extrapolation might
lead to solution divergence or instabilities 282. Meanwhile, even if we are
interested in simulating just one portion of the domain corresponding to
specific dynamics, we still need to specify accurate interface conditions.
Running a high fidelity solver only to provide the flow state at the interface
seems to be unreasonable. Recently, we formulated an eclectic interface
modeling approach as a key enabler for emerging DT technologies in many
sectors 60. However, just like any technology, it comes with its own needs and
challenges 283, 284, 285, 286, 287, 288. Therefore, an accurate
characterization and modeling of the interface is crucial to derive the
consistent boundary conditions and upscaling laws in a large variety of
scientific applications. As discussed in our recent work 56, complex natural
or engineered systems often comprise multiple characteristic scales, multiple
spatiotemporal domains, and even multiple physical closure laws. To address
such challenges, an _interface learning_ (IL) paradigm, which consists of a
data-driven closure approach based on memory embedding, has been introduced to
provide physically consistent boundary conditions at the interface. In a
nutshell, here we can define IL as the utilization of statistical tools to
provide some _educated_ or _learned_ interface conditions to couple different
solvers or models to maximize computational efficiency. Although interface
modeling approaches are also possible with classical physics-based approaches
(e.g., see 289), we highlight that HPC environments can benefit from this
methodology to reduce communication costs among processing units in emerging
machine-learning-ready heterogeneous platforms toward exascale era. Moreover,
it is possible to include physical insights into an interface learning
framework, like _upwind learning_ concept 56 that can be considered as a
physics-informed data-driven domain decomposition approach. The chief idea has
been drawn to enforce physics into the learning process. This process,
therefore, accounts for the effects of downstream domain with an ML model that
provides the interface condition in order to be able to solve for the upstream
domain. Another key aspect of such a zonal multifidelity approach is its
ability to handle intrinsic heterogeneous physical properties, varying
geometries, and underlying governing dynamics. This heterogeneity can be
pronounced in certain applications with spatially varying states. Many
interfacial problems can be put into the following categories, explained with
a family of examples as follows:

_Reduced order model - full order model (ROM-FOM) coupling:_ As we highlight
in our recent work 60, there is a demand for lighter models that can run in
real-time. In the context of surface flows (e.g., urban modeling and weather
prediction), FOMs have been in use for a long time; however, they are
incapable of modeling phenomena associated with scales smaller than what the
coarse mesh can handle (like buildings and small terrain variations). These
fine scale flow structures can be modeled using a much refined mesh but then
the simulations become computationally intractable. To tackle this problem one
can systematically couple FOM and ROM models, and develop consistent interface
conditions (both in space and time) using, for example, the nudging type
approaches introduced in Section 2. In setting over complex terrains, the ROM-
FOM coupling could be crucial, where we can utilize a high-fidelity multi-zone
large eddy simulation (LES) solver to simulate flows in complex terrains,
which are dominated by patterns like channeling, rotors, hydraulic jumps, and
mountain waves 281. Once we build ROMs, we can develop new coupled ROM-FOM
models to perform a real-time, accurate approximation of wind turbine loadings
due to wind flow interactions, which is essential in maximizing the power
production of wind farms. Several methods to bridge low-fidelity and high-
fidelity descriptions have been recently introduced to form the building
blocks of an integrated HAM approach among mixed fidelity descriptions 57.

_Multiphysics and multiscale coupling:_ Various dichotomies with a
multiphysics coupling of interacting subsystems can be identified and
exploited in a multitude of scientific applications. A common example in this
category is the use of a particle based approach (e.g., molecular dynamics) in
part of the domain, while using a continuum based approach (e.g., Navier-
Stokes) in the rest of the domain 290, 291, 292. In a recent study289, the
feasibility of such an interface learning approach has been demonstrated by
solving the FitzHugh-Nagumo model in a bifidelity setting partitioned between
a finite difference method (FDM) and the lattice Boltzmann method (LBM), where
the authors consider the underlying dynamics to be composed of a collection of
pseudo-particles that are represented by a particle distribution function.

_Geometric multiscale or mixed-dimension modeling:_ It is desirable to develop
mathematical models for physical problems in which it is necessary to
simultaneously consider equations in different dimensions 293, 294, 295, 272,
296, 271. For example, a network flow 56 or a Darcy subsurface flow system 297
can be mathematically formulated by means of heterogeneous problems featuring
different degrees of detail and different geometric dimensions that interact
together through appropriate interface coupling conditions.

_Model fusion:_ From a fluid dynamics perspective, there is a plethora of
models to describe subgrid scale closures or nonlinear residual effects.
Turbulence modeling generally requires an apriori selection of the most suited
model to handle a particular kind of flow. However, it is seldom that one
model is sufficient for different zones in the computational domain. To
alleviate this problem, hybrid and blending models have been extensively
utilized to lift technical barriers in industrial applications, especially in
settings where the RANS approach is not sufficient and LES is expensive 298,
299, 300. By utilizing ML classifiers 136, 301 and regressors 302, 264, 303,
we can develop hybrid RANS/LES turbulence models to provide the exchange of
information accurately at the interface, and explore new hybridization
criteria.

_Nested solvers:_ To decrease the computational cost required for an accurate
representation of the numerous interconnected physical systems, e.g., oceanic
and atmospheric flows, it is pivotal to develop several classes of nested
models to form the basis of highly successful applications and research at
numerous weather and climate centers 304, 305, 306, 307, 308, 309. Numerous
HAM tools can be used to force the large-scale atmospheric states from global
climate models onto a regional forecasting model, e.g., see a canonical
example in demonstrating an LSTM-nudging approach 246.

_Space-time domain decomposition:_ An analogous situation usually occurs in
parallel computing environments with domain decomposition and distribution
over separate processors with a message passing interface to communicate
information between processors. The heterogeneity of different processing
units creates an asynchronous computational environment, and the slowest
processors will control the computational speed unless load-balancing is
performed 310, 311. The feasibility of such space-time techniques toward
noniterative domain decomposition has been demonstrated by introducing the
_Learn from Past_ (LP) concept 56. Since the interface neighboring points
might be evolved in time before the interface condition is updated in an
explicit scheme, a variant of the LP model based on a combination between old
and updated values, namely the _Learn from Past and Present_ (LPP) model 56,
can be utilized as well. It is possible to extend this mapping to take into
account the time history dependence in a non-Markovian manner through the
adoption of the recurrent neural networks (RNNs).

_Interfacing of DTs:_ DTs can be developed for different components
constituting an asset. For example in the context of wind farm there can be
component level DTs of turbine blades, drive trains, atmospheric condition,
energy trading. Interface learning can once again facilitate bidirectional
exchange of information between these component level DTs to build a holistic
asset level (e.g., wind farm) DT.

##  5 Reduced order modeling, data assimilation and control

A chief motivation behind the reduced order modeling is to be able to use them
in multi-query applications such as control and optimization. Given the
abundance of data and data-driven tools, the development of ROMs is gaining
increasing popularity nowadays. As highlighted in our introduction, there are
a great number of review articles available concerning various aspects of
model order reduction and its applications 61, 62, 63, 64, 65, 66, 67, 68, 69,
70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88,
89, 90, 91, 92, 93, 94, 95, 96, 97, 98, and more relevant to our discussion,
the enabling role of model order reduction approaches in developing next
generation DT systems has been also discussed 284, 60. Of particular interest,
model predictive control (MPC) 312, 313, 314 originated in the late seventies
and has since then evolved considerably. It is not a specific control strategy
but rather a range of control methods which make explicit use of linear and
even nonlinear models to obtain the control signal by minimizing an objective
function over a future horizon. One of the keys to success of these models is
a continuous update of the horizon and computing an optimal control input
while anticipating the future state. It is clear that the computation of an
optimal control input thus depends heavily on the accuracy and computational
efficiency of the models. Unfortunately most of the realistic models are
highly non-linear and computationally stiff to solve making the optimization
process a bottleneck in many of the MPC processes. One approach that has been
extensively used is to linearize the model around a set point and then use the
traditional linear control theories 315, which are founded on the basis of the
superposition principle. As one moves from linear to nonlinear systems these
principles no longer hold and suffers from two major issues: firstly, it
cannot predict the nonlocal behavior far from the operating point (e.g., long
horizons) and secondly it often ignores nonlinear phenomena which can take
place only in the presence of nonlinearities 316. To this end, reduced order
modeling offers an attractive alternative when designing a digital
cyberinfrastructure. Overall, ROM can play four major roles in the context of
MPCs:

  * i.

Data compression: A ROM approach can be used to compress the data from a
system. This can be useful when the communication bandwidth is limited and
storage space is expensive. Furthermore, instead of saving the ROM rather than
the data can make retrieval or archival of data really fast, something
desirable when a continuous update of the ROM is required.

  * ii.

Deducing quantity of interest: Even with the ability to collect big data
through the deployment of multiple sensors in a system, it might be difficult
to compute the quantity of interest. Complex models might be required for such
computation. ROM in this regard can help in speeding up such modelling while
still exploiting the information available in big data 317.

  * iii.

Accelerating the optimization process: The optimization process requires
running the model repeatedly to find out optimal control parameters over a
prediction horizon. ROMs which can give accuracy comparable to the high
fidelity models but still with several order of magnitude reduction of
computational cost 318 can make such practical even when the dynamics is
nonlinear.

  * iv.

Accelerating stochastic modeling: Probabilistic modeling often includes Monte
Carlo simulations which are costly using the FOM. However, in a recent study
319 the authors show how ROM together with hyperreduction can be used to speed
up stochastic modeling to take into account model-uncertainty in the
underlying physical model.

We envision that ROMs will be a key enabler to lay the groundwork for
developing a big data cybernetics infrastructure (e.g., please see Section 6),
an approach to controlling an asset or process using real-time big data. These
tools and concepts offer many new perspectives to our rapidly digitized
society and its seamless interactions with many different fields. With the
recent wave of digitalization, the latest trend in every industry is to build
systems and approaches that will help it not only during the
conceptualization, prototyping, testing and design optimization phase but also
during the operation phase with the ultimate aim to use them throughout the
whole product life cycle and perhaps much further beyond that. While in the
first phase, the importance of numerical simulation tools and lab-scale
experiments is not deniable, in the operational phase, the potential of real-
time availability of data is opening up new avenues for monitoring and
improving operations throughout the life cycle of a product, and ROMs will be
crucial to improve such DT technologies. Such a holistic approach can find
application in many fields such as wind energy for optimal control and
predictive maintenance, in autonomous vehicles for optimal and safe
maneuvering under adverse environmental conditions, in drug delivery, and
beyond.

##  6 Big data cybernetics

By combining methods from control theory and HAM, big data cybernetics is an
approach to controlling an asset or process using real-time big data. As an
explainable and interpretable approach envisioned by Martens320, it is aimed
to discover systematic/coherent structures and convert them into more
quantitative information. Although the big data cybernetics has been emerged
as an approach without involving black-box tools such as neural networks or
support vector machines, we envision the utilization of the HAM methodologies
to process and translate big data into smart data. Using the specific needs of
the industry, application, or process, big data often gets turned into smart
data when it is optimized 321, 322, 323. Here, we envision that the feature
engineering and optimal sensor placement methodologies might serve as viable
tools for this transformation. In the context of upcoming technologies like
DTs, the role of cybernetics is to steer the system toward an optimal set-
point as shown in Figure 3. The difference, called the error signal, is
applied as feedback to the controller, which generates a system input to bring
the output set-point closer to the reference. With the availability of more
and more sensors and communication technologies, an increasingly larger volume
of data (in fact big data) is being made available in real-time. At the first
step, we interpret the big data using well understood mechanistic models based
on known physics. We term the rest as interpretable residual that can be
modeled using a data-driven approach. After this second step, an
uninterpretable residual remains to be modeled using more complex and black-
box models like deep neural networks. The remaining residual is generally
noise, which can be discarded. The three steps result in a better
understanding of the data and hence an improved mechanistic model resulting
from the hybridization of physics-based and data-driven models, and will be
continuously looped with the availability of new sources of data. This concept
offers many new perspectives to our rapidly digitized society and its seamless
interactions with many different fields.

Figure 3: An overview of big data cybernetics: The model starts as a physics-
based model that is based on first principles but as time progresses the model
is continuously updated using knowledge generated from data. The figure is
adapted from Rasheed et al. 60.

We emphasize that new situations (e.g., situations that are not included in a
supervised learning machine) are pervasive in scientific applications,
especially in control problems. In many cases, there is a grand challenge in
making accurate predictions using data-driven methods based strictly on
statistical inference. Moreover, with partial or no understanding of the new
situation, it might be almost impossible to develop a pure physics-based model
due to lack of parameterizations from first principles. Therefore, hybrid
methods could be instrumental in discovering new control laws. For example,
recently, we show how a reinforcement learning (RL) agent can learn
complicated control laws through trial and error to achieve complicated tasks
of path following and collision avoidance simultaneously 252, 253, 254.
However, such generalizable learning happens in a black-box manner so their
applicability in critical applications is foreseen to be limited unless the
learned control laws can be expressed in comprehensible mathematical form. In
this context, the developments of sparse regression and symbolic regression
methods are highly encouraging 256, 324, 257. In particular, our recent work
exploited _symbolic regression_ (SR) as a powerful technique to discover
highly non-linear equations directly from sparse sensor data325. A remaining
question is how to combine RL and SR for making generalizable models without
compromising interpretability. We envision that one can make use of open
source RL packages to scale the training in a distributed manner by running
multiple environments concurrently to accelerate the training. An exploration
for constructing new reward functions that are more suited to certain multi-x
environments to take into account efficient sampling of big data. A _design of
experiments_ methodology can be used to keep the number of numerical
experiments (e.g., required to develop HAM) as small as possible without
compromising the resulting quality. The agent can then be trained with the new
objective of adaptive sampling.

Computational models are nowadays implemented in many decision making
processes like model predictive control 326, 327. For instance, flow control
deals with altering the flow field to achieve the desired objective and is of
immense importance in practical applications such as aerodynamic drag
reduction, mixing enhancement, and noise reduction 328. The problem of closed-
loop control in fluid mechanics is challenging due to strong nonlinearity,
high-dimensionality, and recent advances in system identification, model-order
reduction, ML, and compressive sensing have made significant progress in
adaptive real-time control 77, 329. Unsupervised ML algorithms, like RL, have
illustrated the potential of human-level control in playing games 330, robot
motion control 331, and for continuous control tasks 332. RL controllers are
based on experience gained from self-play or exploration, using algorithms
that can learn to execute tasks by reinforcing good actions based on a
performance metric. The main framework of the RL consists of an agent (for
example, a neural network in deep RL) that interacts with an environment to
learn a policy that will maximize the cumulative reward over a long time
horizon 333. In recent years, the RL has been explored for fluid dynamics
problems including animal locomotion 334, 335, 336, control of chaotic
dynamics 337, 338, 339, drag reduction of bluff bodies 340, 341, 342, flow
separation control 343, and turbulence closure modeling 344. Along with a
computer simulation environment, RL has been effectively applied for active
flow control around bluff bodies in an experimental setup 345. Also, Novati et
al. 344 introduced the multi-agent RL strategy to learn the turbulence closure
as a control policy enacted by cooperating agents, and all the agents are
trained to minimize the distance from direct numerical simulation (DNS) energy
spectrum. Fan et al. 345 utilized the actual towing experiment with three
cylinders as an environment of the RL framework. In their work, the rotation
of two cylinders in the wake of the main cylinder was used as an action to
minimize the overall drag of the system, and they demonstrated the importance
of noise reduction strategies like the Kalman filter for applying deep RL to
experimental environments. Bucci et al. 337 applied the deep RL to build a
model-free controller to stabilize a chaotic system using limited sensor
measurements as observable by an agent. In another work, Ma et al. 346
utilized RL to control a coupled system involving both fluid and rigid bodies,
by applying control forces only at the simulation domain boundaries. Genetic
programming (GP) has also attracted great attention for active flow control
frameworks. For example, GP was applied to control the recirculation area of a
backward facing step 347 and mitigate the separation and early reattachment of
turbulent boundary layer for a sharp-edge ramp 348. Moreover, Ren et al. 349
adopted GP to select explicit control laws for the suppression of vortex-
induced vibration of a circular cylinder in a low-Reynolds-number flow using
blowing/suction at fixed locations. All these ideas can be exploited in
emphasizing the potential for using flexible data assimilation and automatic
control environments to unlock the capabilities of a DT framework.

##  7 Digital twin revolution

We can arguably discuss that the social media is the most recent disruptive
technology for generating DTs of users. Why not generating DTs of assets or
processes? Therefore, in a broad sense, we can define a DT as a virtual
representation of an asset or its components for real-time prediction,
monitoring, control, and optimization of the asset throughout its life cycle.
This DT concept with a high level of realism has been started to be a paradigm
shift for more informed decision making in every industry. It is quite natural
that improving preconditions from simulations and models will lead to an
improvement of DT technologies. Hence, the question arises why HAM techniques
specifically could enable the revolution of DT technologies. In our recent
survey paper60, we have presented a DT framework and detailed its values,
challenges and enabling technologies.  For example, some of the major
challenges in realizing the potential of wind energy to meet the global
electricity demand are the need for a deeper understanding of the physics of
the atmospheric flow, science, and engineering of these large dynamic rotating
machines and synergistic optimization and control of fleets of wind farms
within the electricity grid 350. The decades of research and development in
fluid dynamics, systems engineering, numerical methods, manufacturing
processes, and material discovery can now be complemented with unprecedented
amounts of data generated from insitu measurements, lab experiments, and
numerical simulations to tackle these challenges. As we discussed in Section
2, the combination of physics-based and data-driven models is increasing in
every branch of science leading to numerous novel HAM approaches for many
scientific applications. Synergistically, the HAM paradigm can be applied to a
variety of tasks related to wind energy research, such as DTning for the
optimization and real-time control of wind farms. A DT of the wind farm lets
us examine _what if_ scenarios, evaluate the system’s response, and select the
corresponding mitigation strategies. As illustrated in Figure 4, the DT of
wind farms can be useful for different purposes such as optimal control of
wind turbines to achieve the maximum performance, routine maintenance of the
equipment, accurate forecast of the power production, wind farm optimization
351, and improved decision-making. The success of the DTs depends upon the
type of approach that we employ for modeling the system. The ability of the
HAM to combine the generalizability of physics-based approaches and the
automatic pattern-identification feature of data-driven approaches makes it an
attractive choice to model virtual replica of physical systems in DTs. For
example, there are several turbine wake models that capture the flow in the
wake of a single turbine and these models predict sufficiently accurate
aerodynamics of a single turbine 352, 353. However, these models are not
enough to capture the flows in wind farm due to wake superposition, complex
terrains, deep array effects, and neglected physics 354, 355. The
observational data collected from a variety of sensors can be assumed to
comprise of these complex flow interactions and the manifestation of all
physical processes. Therefore, the hybrid model will aid in the robustness of
the DT rather than a pure physics-based or pure data-driven approach.

Figure 4: An overview of a DT concept where the HAM within the big data
cybernetics framework becomes a key enabler for modeling physical realism and
optimal control. For instance, a DT of the wind farm can allow us to evaluate
different scenarios using hybrid models. The big data collected from IoT
sensors can be continuously assimilated to correct hybrid models and improve
the state estimation.

In the current paper we view a DT of a physical asset as the schematic shown
in Figure 4. Any physical asset generates data which requires real-time
processing for mainly two purposes; informed decision making and real-time
optimization and control of the physical asset. Additionally, the recorded
data can be perturbed to do offline _what if?_ analysis, risk assessment,
uncertainty quantification and process optimization. In the latter context it
is more appropriate to term the various virtual realizations of the physical
asset as _digital siblings_.

While DTs operating as digital siblings can be used for hypothetical
scenarios, _digital threads_ can be used for transferring experience from one
asset to the next generation assets. As depicted in Figure 5, the capability
of DT356 can be ranked on a scale from 0 to 5 (0-standalone, 1-descriptive,
2-diagnostic, 3-predictive, 4-prescriptive, 5-autonomy). Once multifidelity
modeling tools relevant for specific applications are developed, a DT
framework can be utilized for holistic optimization throughout the design,
installation, operation, and decommissioning phases. Therefore, the DT
representation can facilitate rapid knowledge exchange between _research and
innovation_ with _design and operations_.

Figure 5: The capability level of DTs on a scale from 0 to 5. We envision that
the HAM can leverage simulating tools from 0-2 scales to 3-5 scales.

The communication between the physical asset and its DT can allow for a rapid
testing of industry-standard workflows, as well as the state of the art data
assimilation and optimal control theories. The chief idea is to establish a
two-way coupling in real-time between the physical asset and the simulator
through data assimilation and model updates. Experimental measurements can be
immediately forwarded and seamlessly integrated into the simulator, while at
the same time, the simulator can directly control the physical asset or
provide metrics for a better informed decision-making process. Moreover, an
experimental facility or set-up can provide a unique possibility for
designing, calibrating and validating a DT simulator. Of specific interest,
the traditional industrial workflow in many sectors use simulators as a
planning and decision-making tool, wherein data are integrated via history
matching to improve the simulator’s parameterizations. This standard workflow
can be significantly improved by the concept defined by Bolton et al. 357 as
“a dynamic virtual representation of a physical object or system across its
lifecycle, using real-time data to enable understanding, learning, and
reasoning” that might yield ultimately a software package to allow for real-
time control of operating conditions. In practice, the following aspects are
of great interest:

_Validating the physical model and data integration technologies._ Due to the
availability of precise characterization in complex systems, the skill of data
assimilation technologies can be carefully assessed in comparative studies
with true physical benchmarks. Moreover, no standard for building a DT exists.
Most of the state of the art DT technologies operate on a capability scale 0-2
having little physical realism owing to lack of data and real-time models with
the ability to model unknowns. This might allow for a critical review of
industry best-practices, as well as evaluation and development of new
methodologies.

_The potential for automatic control._ The time scales of possible
experimentation (hours to days to weeks) and the ability to repeat these
experiments with nearly identical physical features allow for investigation
for more improved model-based automatic control strategies. Control algorithms
are often linear, use overly simplified models, and rarely optimize for
multiple objectives simultaneously.

_Validation of scenario and risk assessment._ Carefully designed scenario and
risk assessment studies should be performed with the same physical conditions.
A DT framework can be also used to assess sustainability.

_Enhanced training._ In many systems, the decisions are often based on
intuition obtained from indirect observations or simulations with incomplete
parameterizations. The transparency of the front panel may provide direct
feedback for the users to correlate between indirect measurements and
underlying system dynamics.

_Predictive maintenance._ The component level condition monitoring can be
speeded up by using the HAM approaches and then can be integrated together to
develop asset level condition monitoring methodologies. Predictive maintenance
decisions and remaining lifetime predictions based on condition monitoring and
statistical reliability of data interfaced with predictions of broader
interconnected systems may set important considerations toward a holistic and
probabilistic framework for decision support.

_Asset management._ Since DTs often combine different assets and their
information in a central place, it will be crucial to ensure that data is
visually represented well, secure and available to all relevant stakeholders.

_Public engagement._ Virtual reality or augmented reality technologies can be
utilized to inform the public for the future impacts of the upcoming or
hypothetical scenarios for an asset. For example, wind energy developments may
face public resistance 358. More informed communication of the impact of
upcoming wind farms to the public can benefit a lot from DT technologies.

##  8 Concluding Remarks and Outlook

In this perspective paper, we primarily focus on the combined use of physics-
based and data-driven models, denoted as HAM, as a science and engineering
disruptor that can be exploited to achieve the goals of the development of
better predictive methods and tools to form a digital transformation in
domain-specific fields. Figure 1 shows a non-inclusive list of various
directions where HAM can be pursued. The outlined HAM frameworks aim squarely
at transforming both physics-based and data-driven methodologies into general,
robust and portable computational tools for applications across the spectrum
in engineering, science, and medicine. From our point of view, there is an
urgent need to focus on HAM algorithms and architectures, which are a core
enabler of emerging DT applications. To this end, we identify a set of topics
for further exploration below.

_Knowledge-informed ML._ The problem of building a model from the high-
fidelity data can be posed as a multi-objective optimization problem where
several aspects of the knowledge-injection processes must be involved in data
preparation, training and testing steps. When it comes to developing ML
algorithms and architectures, this HAM approach is of critical importance to
tackle challenges such as incorporating physical laws within the learning
framework, producing solutions that are interpretable and explainable,
addressing nonlinearities and conservation properties, and dealing with the
massive amount of data needed for training. It is possible to gain huge
performance increases (e.g., improved generalizability with reducing neural
network model uncertainty) when we augment the knowledge of the domain-
specific simplified theories with the underlying learning process.

_Data preprocessing, compression and superresolution:_ Main issues associated
with data are noise, missing data, sparse data, and excessively dense data.
Although there are state-of-the-art methods for dealing with these issues, a
potential research focus in this domain can revolve around generative
adversarial networks, and principal component analysis (or its nonlinear
extensions 359, 360, 361, 362, 363, deep autoencoders 364, 365, 366, 367, 368
etc). While the former is a powerful method for synthesizing new data from
sparse data, the latter is a dimensionality reduction algorithm useful in
discarding irrelevant features in data.

_Improving reduced order models:_ ROMs have great promise for flow control,
data assimilation, parameter estimation, and uncertainty quantification.
However, their potential has only been realized for a limited set of problems.
For example, fluid flows typically involve nonlinear interactions over very
wide ranges of length and time scales. Although high-fidelity simulations of
complex flows are possible to a certain extent with today’s supercomputers,
computationally feasible yet accurate ROMs that operate under such strong
nonlinear interactions with a wide range of geometric and physical parameter
changes currently do not exist. When it comes to utilizing machine learning
algorithms in fluid dynamics, some of the challenges include incorporating
physical laws within the learning framework, producing solutions that are
interpretable, addressing nonlinearities, and dealing with the massive amount
of data needed for training. Therefore, synthesizing ideas from physics-based
modeling, numerical methods, data-driven tools, and machine learning can be
considered one of the most active areas in ROM community to tackle fundamental
barriers not addressed by current, evolutionary modeling practices. These
include the curse of dimensionality, lack of accuracy, modal deformation due
to transport processes, automatic feature selection, dynamically adapted error
metrics, sparse and anisotropic data, sensitivity to noise, and lack of
conservation.

_Discovering new physics using ML and multivariate data analysis:_ It is
discovered that convolutional layers closer to input modes capture small scale
features while the layers closer to the output modes captures large scale
features. Neural style transfer is a consequence of this discovery. In fluid
mechanics, a potential application of this discovery can lead to a better
understanding of turbulence dominated by multiple spatio-temporal scales.
Moreover, we can learn constitutive relations from indirect observations using
deep neural networks (for example see 369). Alternatively, to address the key
limitation of the black-box learning methods, we can exploit to use of
symbolic regression as a principle for identifying relations and operators
that are related to the underlying system dynamics. This approach combines
evolutionary computation with feature engineering to provide us a tool to
create new models (e.g., see our recent application for the closure discovery
in turbulence 325). The most prohibitive obstacle to overcome in order to
search for an appropriate closure or correction model is finding meaningful
and nontrivial invariants (e.g., invariants of deformation gradient, strain-
rate and vorticity gradient tensors, Lie symmetries, and gauge
transformations).

_Using physics and domain knowledge to better frame ML solutions:_ How should
we inject physics and domain knowledge into AI models? Our recent work in this
direction pertains to introducing a physics guided ML approach based on
augmenting reason-based and principled models into the neural network
architectures. To emphasize on their physical importance, our architecture
consists of adding certain features at intermediate layers rather than in the
input layer. Such a feature enhancement approach can significantly improve the
prediction accuracy with better uncertainty confidence characteristics, and it
can be effectively used in many scientific ML applications, especially for the
systems where we can use a theoretical, empirical, or simplified model to
guide the learning module. In certain applications, like offshore wind energy
predictions, we can create a huge benefit of improving the prediction system
if the domain knowledge is utilized for feature selection and feature
engineering before using a data-driven approach.

_Improving interpretability and explainability of ML algorithms:_ Although the
universal approximation theorem states that a deep neural network can
approximate any complex function, it is imperative to make the neural networks
more interpretable for a risk free adoption of the approaches in safety
critical applications. A first step to interpreting these networks is to
develop alternative representations that allow for further analysis. We have
provided a proof-of-concept as a first step for computing piecewise affine
form of any fully connected networks. We plan to extend our analysis to modern
networks with more complex branching architectures. Using our dissecting
method together with dimensionality reduction techniques is an approach that
shows great promise for the study of complex systems that resist analysis.

_Communication cost reducing algorithms._ Probably, another important aspect
is the design of communication-cost reducing algorithms toward exascale
computing with emerging heterogeneous architectures. Since the chip
manufacturing sector reaches the limits of the atomic scale, unless we see a
new transistor technology to replace current metal-oxide semiconductor
technologies, a more effective design of algorithms becomes crucial as the
number of processing elements increases and/or the number of grid points
decreases within a processing element. This leads to more effective use of
transistors through more efficient architectures in an ecosystem of extreme
heterogeneity. The HPC community has moved forward to incorporate GPU based
accelerators and beyond (e.g., TPUs) for not only graphics rendering but also
scientific computing applications in a post Moore’s Law world. This
heterogeneity shift will become even more crucial in the future since there is
a rapid increase in the usage of high-productivity programming languages
(e.g., MATLAB, R, Python, Julia) among engineers and scientists. Beyond the
obvious use of GPUs in tensor based methods, the integration of traditional
HPC practices with ML becomes crucial to obtain major performance
improvements.

_Integrating ML into large scale computations:_ Where is it valuable and how
should we do this? Combining ML methodologies with solvers, a chief research
focus can be tackling a series of open cyberinfrastructure questions and
challenges that the HAM paradigm presents. We advocate further to exploit
techniques that are more common in “ML for HPC” than “HPC for ML” and
investigate the benefit of this approach. This requires a seamless approach of
both horizontal (many tasks) and vertical (hierarchical) parallelisms. For
example, sampling techniques today offer an unbiased way of quantifying
uncertainty. Using ML for pattern recognition in combination with such
techniques is a promising option for reliable uncertainty quantification for
data sets that are spatially limited and have limited resolution. Moreover,
most data-driven approaches involve matrix operations, which can be very
efficiently parallelized on affordable graphical processing units and tensor
processing units giving several orders of magnitude speedup as required in
real-time applications. Although “ML for quantum computers” is largely an open
area, utilizing quantum computers to potentially enhance conventional machine
learning algorithms has been discussed recently in the context of generating
high-quality images 370.

_Toward asynchrony-tolerant computations:_ Increasing the scalability of the
codes is a direct consequence of relaxing the data dependence, since the
requirement of such global synchronization throughout the domain leads to poor
scaling performance with increasing system size, especially for more realistic
direct numerical simulations. It is thus imperative to develop computational
strategies and stable algorithms that tolerate asynchrony between processing
elements. To mitigate the effect of asynchrony, one can exploit the
feasibility of a ML based proxy-equation paradigm for asynchrony-tolerant
computations. This approach might introduce a data-driven delay correction
factor by modifying the time scale near processing element boundaries (i.e.,
learning time index, adding inertia to the system to reduce the delay error
and improve accuracy).

Overall, the unifying theme in these research thrusts is to develop a HAM
paradigm that combines physics-based modeling with the versatility of data-
driven approaches. The HAM concept consists of a wide spectrum from
fundamental research, to enabling technology development, to the system
integration phase, and thus offers a compelling theme for center-wide projects
as well as university-industry partnership calls. The interactions between
domain specific scientists and data scientists have increased exponentially in
recent times. Hence we may witness many novel approaches or even paradigm
shifts in the near future to tackle some of the fundamental challenges not
addressed by evolutionary segregated modeling practices. We believe that there
is an urgent need to focus on online real-time compression and extrapolatory
predictive modeling for safety-critical applications in engineering and
applied sciences. Through the development of such a profound HAM computational
pipeline, we believe that it is possible to advance predictive tools for
simulating and characterizing complex multi-x systems with quantified
uncertainties and make them more trustworthy, generalizable, and explainable
for real life applications.

## Acknowledgments

Omer San would like to acknowledge support from the U.S. Department of Energy
under the Advanced Scientific Computing Research program (grant DE-SC0019290),
the National Science Foundation under the Computational Mathematics program
(grant DMS-2012255). Adil Rasheed and Trond Kvamsdal gratefully acknowledge
the funding received by SINTEF Digital in the SEP Digital Twin project
(102015968-22). OPWIND: Operational Control for Wind Power Plants (Grant No.:
268044/E20) project funded by the Norwegian Research Council and its
industrial partners (Equinor, Vestas, Vattenfall) is also acknowledged.

### Conflict of interest

The authors declare no potential conflict of interests.

## References

  * 1 Peter Shaver, The rise of science: from prehistory to the far future, Springer, Berlin, 2018.
  * 2 Petter E Bjørstad, Olof B Widlund, SIAM Journal on Numerical Analysis 1986, 23 (6), 1097–1120.
  * 3 Tony F Chan, Roland Glowinski, Jacques Périaux, Olof B Wildlund, in Domain decomposition methods: proceedings of the Second International Symposium on Domain Decomposition Methods, Los Angeles, California, January 14-16, 1988, 1989.
  * 4 Charbel Farhat, Francois-Xavier Roux, International Journal for Numerical Methods in Engineering 1991, 32 (6), 1205–1227.
  * 5 Tony F Chan, Weinan E, Jiachang Sun, Applied Numerical Mathematics 1991, 8 (4-5), 317–331.
  * 6 William D Gropp, David E Keyes, International Journal for Numerical Methods in Fluids 1992, 14 (2), 147–165.
  * 7 Tony F Chan, Tarek P Mathew, Acta Numerica 1994, 3 (1), 61–143.
  * 8 Jing Li, Olof B Widlund, International Journal for Numerical Methods in Engineering 2006, 66 (2), 250–271.
  * 9 Andrea Toselli, Olof Widlund, Domain decomposition methods-algorithms and theory, Vol. 34, Springer Science & Business Media, 2006.
  * 10 Martin Jakob Gander, Electronic Transactions on Numerical Analysis 2008, 31, 228–255.
  * 11 Luca F Pavarino, Andrea Toselli, Recent developments in domain decomposition methods, Vol. 23, Springer Science & Business Media, 2012.
  * 12 Xuemin Tu, Jing Li, SIAM Journal on Numerical Analysis 2015, 53 (2), 720–742.
  * 13 Victorita Dolean, Pierre Jolivet, Frédéric Nataf, An introduction to domain decomposition methods: algorithms, theory, and parallel implementation, SIAM, 2015.
  * 14 Michael Pedneault, Catalin Turc, Yassine Boubendir, IMA Journal of Applied Mathematics 2017, 82 (5), 1104–1134.
  * 15 G Houzeaux, JC Cajas, Marco Discacciati, B Eguzkitza, A Gargallo-Peiró, M Rivero, M Vázquez, Archives of Computational Methods in Engineering 2017, 24 (4), 1033–1070.
  * 16 HS Tang, RD Haynes, G Houzeaux, Archives of Computational Methods in Engineering 2020, 1–33.
  * 17 Thomas J.R. Hughes, Computer Methods in Applied Mechanics and Engineering 1995, 127 (1-4), 387–401.
  * 18 Thomas J.R. Hughes, Gonzalo R. Feijóo, Luca Mazzei, Jean-Baptiste Quincy, Computer Methods in Applied Mechanics and Engineering 1998, 166 (1-2), 3–24.
  * 19 Weinan E, Bjorn Engquist, Xiantao Li, Weiqing Ren, Eric Vanden-Eijnden, Communications in Computational Physics 2007, 2 (3), 367–450.
  * 20 Grigoris Pavliotis, Andrew Stuart, Multiscale methods: averaging and homogenization, Springer Science & Business Media, 2008.
  * 21 Jacob Fish, Multiscale methods: bridging the scales in science and engineering, Oxford University Press, 2010.
  * 22 Karel Matouš, Marc GD Geers, Varvara G Kouznetsova, Andrew Gillman, Journal of Computational Physics 2017, 330, 192–220.
  * 23 Marc GD Geers, Varvara G Kouznetsova, Karel Matouš, Julien Yvonnet, Encyclopedia of Computational Mechanics Second Edition 2017, 1–34.
  * 24 Ana Maria Soane, Andrei Draganescu, SIAM Journal on Numerical Analysis 2019, 57 (3), 1494–1523.
  * 25 Huaiqian You, Yue Yu, David Kamensky, Computer Methods in Applied Mechanics and Engineering 2020, 366, 113038.
  * 26 Mary F Wheeler, Ivan Yotov, Contemporary Mathematics 1998, 218, 217–228.
  * 27 Yanzhao Cao, Max Gunzburger, Xiaoming He, Xiaoming Wang, Mathematics of Computation 2014, 83 (288), 1617–1644.
  * 28 Mark Alber, Adrian Buganza Tepole, William R Cannon, Suvranu De, Salvador Dura-Bernal, Krishna Garikipati, George Karniadakis, William W Lytton, Paris Perdikaris, Linda Petzold, NPJ Digital Medicine 2019, 2 (1), 1–11.
  * 29 David E Keyes, Lois C McInnes, Carol Woodward, William Gropp, Eric Myra, Michael Pernice, John Bell, Jed Brown, Alain Clo, Jeffrey Connors, The International Journal of High Performance Computing Applications 2013, 27 (1), 4–83.
  * 30 Pierre Gosselet, Vincent Chiaruttini, Christian Rey, Frédéric Feyel, Revue Européenne des Éléments 2004, 13 (5-7), 523–534.
  * 31 Andrew T Barker, Xiao-Chuan Cai, Journal of Computational Physics 2010, 229 (3), 642–659.
  * 32 Michael W Gee, Ulrich Küttler, Wolfgang A Wall, International Journal for Numerical Methods in Engineering 2011, 85 (8), 987–1016.
  * 33 Ulrich Langer, Huidong Yang, Fluid-Structure Interactions. Modeling, Adaptive Discretization and Solvers, Radon Series on Computational and Applied Mathematics 2016, 20, 169–192.
  * 34 Roger A Sauer, Tobias Luginsland, Computer Methods in Applied Mechanics and Engineering 2018, 341, 1–31.
  * 35 Alexander Heinlein, Christian Hochmuth, Axel Klawonn, SIAM Journal on Scientific Computing 2019, 41 (4), C291–C316.
  * 36 Carlos A. Felippa, Kwang-Chun Park, Computer Methods in Applied Mechanics and Engineering 1980, 24 (1), 61–111.
  * 37 Alfio Quarteroni, Alberto Valli, Domain decomposition methods for partial differential equations, Oxford University Press, 1999.
  * 38 Carlos A. Felippa, Kwung-Chun Park, Charbel Farhat, Computer Methods in Applied Mechanics and Engineering 2001, 190 (24-25), 3247–3270.
  * 39 Tayfun E Tezduyar, Sunil Sathe, International Journal for Numerical Methods in Fluids 2007, 54 (6-8), 855–900.
  * 40 Ulrich Küttler, Wolfgang A Wall, Computational Mechanics 2008, 43 (1), 61–72.
  * 41 Sang Truong Ha, Long Cu Ngo, Muhammad Saeed, Byoung Jin Jeon, Hyounggwon Choi, Journal of Mechanical Science and Technology 2017, 31 (1), 281–287.
  * 42 Alfio Quarteroni in Numerical Models for Differential Problems, Springer, 2017, pp. 555–612.
  * 43 Pierre-Louis Lions, in First international symposium on domain decomposition methods for partial differential equations, Paris, France, 1988, p. 42.
  * 44 Pierre-Louis Lions, Domain decomposition methods 1989, 628, 47–70.
  * 45 Pierre-Louis Lions, in Third international symposium on domain decomposition methods for partial differential equations, SIAM Philadelphia, PA, 1990, pp. 202–223.
  * 46 KH Tan, MJA Borsboom, in Domain Decomposition Methods in Scientific and Engineering Computing: Proceedings of the Seventh International Conference on Domain Decomposition, October 27-30, 1993, the Pennsylvania State University, American Mathematical Soc., 1994, p. 125.
  * 47 Bjorn Engquist, Hong-Kai Zhao, Applied Numerical Mathematics 1998, 27 (4), 341–365.
  * 48 Caroline Japhet, in Proceedings of the 9th international conference on domain decomposition methods, 1998, pp. 382–389.
  * 49 Marco Discacciati, Paola Gervasio, Alfio Quarteroni, SIAM Journal on Control and Optimization 2013, 51 (5), 3434–3458.
  * 50 Martin J Gander, Olivier Dubois, Numerical Algorithms 2015, 69 (1), 109–144.
  * 51 Eric Blayo, David Cherel, Antoine Rousseau, Journal of Scientific Computing 2016, 66 (1), 275–295.
  * 52 Martin J Gander, Tommaso Vanzan, SIAM Journal on Scientific Computing 2019, 41 (4), A2329–A2354.
  * 53 HS Tang, L Li, M Grossberg, YJ Liu, YM Jia, SS Li, WB Dong, Communications in Nonlinear Science and Numerical Simulation 2021, 97, 105729\.
  * 54 Ke Li, Kejun Tang, Tianfan Wu, Qifeng Liao, IEEE Access 2019, 8, 5283–5294.
  * 55 Alexander Heinlein, Axel Klawonn, Martin Lanser, Janine Weber, SIAM Journal on Scientific Computing 2019, 41 (6), A3887–A3912.
  * 56 Shady E Ahmed, Omer San, Kursat Kara, Rami Younis, Adil Rasheed, Physical Review E 2020, 102 (5), 053304.
  * 57 Shady E Ahmed, Omer San, Kursat Kara, Rami Younis, Adil Rasheed, PLOS ONE 2021, 16 (2), e0246092.
  * 58 Shady E Ahmed, Sk Rahman, Omer San, Adil Rasheed, Ionel M Navon, arXiv preprint arXiv:1910.07649 2019.
  * 59 Suraj Pawar, SM Rahman, H Vaddireddy, Omer San, Adil Rasheed, Prakash Vedula, Physics of Fluids 2019, 31 (8), 085101.
  * 60 Adil Rasheed, Omer San, Trond Kvamsdal, IEEE Access 2020, 8, 21980 – 22012.
  * 61 K Kashinath, M Mustafa, A Albert, et al., Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 2021, 379, 20200093.
  * 62 Peter Bauer, Bjorn Stevens, Wilco Hazeleger, Nature Climate Change 2021, 11, 80–83.
  * 63 AJ Geer, Philosophical Transactions of the Royal Society A 2021, 379 (2194), 20200089.
  * 64 Steven L Brunton, Bernd R Noack, Petros Koumoutsakos, Annual Review of Fluid Mechanics 2020, 52, 477–508.
  * 65 Kunihiko Taira, Maziar S Hemati, Steven L Brunton, Yiyang Sun, Karthik Duraisamy, Shervin Bagheri, Scott TM Dawson, Chi-An Yeh, AIAA journal 2020, 58 (3), 998–1022.
  * 66 Michael Frank, Dimitris Drikakis, Vassilis Charissis, Computation 2020, 8 (1), 15.
  * 67 Senlin Zhu, Adam P Piotrowski, Acta Geophysica 2020, 1–10.
  * 68 Pejman Tahmasebi, Serveh Kamrava, Tao Bai, Muhammad Sahimi, Advances in Water Resources 2020, 142, 103619.
  * 69 Jared Willard, Xiaowei Jia, Shaoming Xu, Michael Steinbach, Vipin Kumar, arXiv preprint arXiv:2003.04919 2020.
  * 70 Gonçalo Mendonça, Frederico Afonso, Fernando Lau, Proceedings of the Institution of Mechanical Engineers, Part G: Journal of Aerospace Engineering 2019, 233 (15), 5816–5836.
  * 71 Jian Yu, Chao Yan, Mengwu Guo, Proceedings of the Institution of Mechanical Engineers, Part G: Journal of Aerospace Engineering 2019, 233 (16), 5896–5912.
  * 72 MP Brenner, JD Eldredge, JB Freund, Physical Review Fluids 2019, 4 (10), 100501.
  * 73 Raul Yondo, Kamil Bobrowski, Esther Andrés, Eusebio Valero in Advances in evolutionary and deterministic methods for design, optimization and control in engineering and sciences, Springer, 2019, pp. 19–33.
  * 74 Kunihiko Taira, Steven L Brunton, Scott TM Dawson, Clarence W Rowley, Tim Colonius, Beverley J McKeon, Oliver T Schmidt, Stanislav Gordeyev, Vassilios Theofilis, Lawrence S Ukeiley, AIAA Journal 2017, 4013–4041.
  * 75 Clarence W Rowley, Scott TM Dawson, Annual Review of Fluid Mechanics 2017, 49, 387–417.
  * 76 Peter Benner, Serkan Gugercin, Karen Willcox, SIAM Review 2015, 57 (4), 483–531.
  * 77 Steven L Brunton, Bernd R Noack, Applied Mechanics Reviews 2015, 67 (5), 050801.
  * 78 Michael J Asher, Barry FW Croke, Anthony J Jakeman, Luk JM Peeters, Water Resources Research 2015, 51 (8), 5957–5973.
  * 79 Toni Lassila, Andrea Manzoni, Alfio Quarteroni, Gianluigi Rozza in Reduced Order Methods for modeling and computational reduction, Springer, 2014, pp. 235–273.
  * 80 Vasileios Machairas, Aris Tsangrassoulis, Kleo Axarli, Renewable and sustainable energy reviews 2014, 31, 101–112.
  * 81 Igor Mezić, Annual Review of Fluid Mechanics 2013, 45, 357–378.
  * 82 Marc P Mignolet, Adam Przekop, Stephen A Rizzi, S Michael Spottswood, Journal of Sound and Vibration 2013, 332 (10), 2437–2460.
  * 83 Mohammad Abid Bazaz, S Janardhanan, et al., in 2012 IEEE International Conference on Signal Processing, Computing and Control, IEEE, 2012, pp. 1–6.
  * 84 Saman Razavi, Bryan A Tolson, Donald H Burn, Water Resources Research 2012, 48 (7).
  * 85 Constantinos Theodoropoulos in Coping with Complexity: Model Reduction and Data Analysis, Springer, 2011, pp. 37–61.
  * 86 Francisco Chinesta, Pierre Ladeveze, Elias Cueto, Archives of Computational Methods in Engineering 2011, 18 (4), 395.
  * 87 Matthew B Wagner, Amir Younan, Paul Allaire, Randy Cogill, International Journal of Rotating Machinery 2010, 2010.
  * 88 Nicola Massarotti, Perumal Nithiarasu, Emad Samadiani, Yogendra Joshi, International Journal of Numerical Methods for Heat & Fluid Flow 2010, 20 (5), 529–550.
  * 89 Jack PC Kleijnen, European Journal of Operational Research 2009, 192 (3), 707–716.
  * 90 René Pinnau in Model order reduction: theory, research aspects and applications, Springer, Berlin, 2008, pp. 95–109.
  * 91 David J Lucia, Philip S Beran, Walter A Silva, Progress in Aerospace Sciences 2004, 40 (1-2), 51–117.
  * 92 Yew S Ong, Prasanth B Nair, Andrew J Keane, AIAA Journal 2003, 41 (4), 687–696.
  * 93 Roland W Freund, Acta Numerica 2003, 12, 267–319.
  * 94 Zhaojun Bai, Applied Numerical Mathematics 2002, 43 (1-2), 9–44.
  * 95 Anindya Chatterjee, Current Science 2000, 808–817.
  * 96 Gal Berkooz, Philip Holmes, John L Lumley, Annual Review of Fluid Mechanics 1993, 25 (1), 539–575.
  * 97 D Bonvin, DA Mellichamp, International Journal of Control 1982, 35 (5), 829–848.
  * 98 Z Elrazaz, NK Sinha, Canadian Electrical Engineering Journal 1981, 6 (1), 34–40.
  * 99 Kurt Hornik, Maxwell Stinchcombe, Halbert White, Neural Networks 1989, 2 (5), 359–366.
  * 100 Martin T Hagan, Mohammad B Menhaj, IEEE Transactions on Neural Networks 1994, 5 (6), 989–993.
  * 101 Martin T Hagan, Howard B Demuth, Mark Beale, Neural network design, PWS Publishing Co., 1997.
  * 102 Rene Vidal, Joan Bruna, Raja Giryes, Stefano Soatto, arXiv preprint arXiv:1712.04741 2017.
  * 103 Jeong-Woo Lee, Jun-Ho Oh, Neural Computation 1997, 9 (5), 937–958.
  * 104 Stephen Marsland, Jonathan Shapiro, Ulrich Nehmzow, Neural Networks 2002, 15 (8-9), 1041–1058.
  * 105 Maziar Raissi, Paris Perdikaris, George E Karniadakis, Journal of Computational Physics 2019, 378, 686–707.
  * 106 Yinhao Zhu, Nicholas Zabaras, Phaedon-Stelios Koutsourelakis, Paris Perdikaris, Journal of Computational Physics 2019, 394, 56–81.
  * 107 Shaowu Pan, Karthik Duraisamy, SIAM Journal on Applied Dynamical Systems 2020, 19 (1), 480–509.
  * 108 Arvind T Mohan, Nicholas Lubbers, Daniel Livescu, Michael Chertkov, arXiv preprint arXiv:2002.00021 2020.
  * 109 Christian Leibig, Vaneeda Allken, Murat Seçkin Ayhan, Philipp Berens, Siegfried Wahl, Scientific Reports 2017, 7 (1), 1–14.
  * 110 Dominik Sacha, Michael Sedlmair, Leishi Zhang, John A Lee, Jaakko Peltonen, Daniel Weiskopf, Stephen C North, Daniel A Keim, Neurocomputing 2017, 268, 164–175.
  * 111 Tyrus Berry, Dimitrios Giannakis, John Harlim, arXiv preprint arXiv:2002.07928 2020.
  * 112 Gene H Golub, Michael W Mahoney, Petros Drineas, Lek-Heng Lim, SIAM News 2006, 39 (8), 1–3.
  * 113 Dan Ciresan, Ueli Meier, Juergen Schmidhuber, in 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012, pp. 3642–3649.
  * 114 Geert J. S. Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen A. W. M. van der Laak, Bram van Ginneken, Clara I. Sánchez, Medical Image Analysis 2017, 42, 60–88.
  * 115 Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, Alexander A Alemi, in The Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, California, USA, February 4–-9, 2017.
  * 116 Kai Li, Jiaqing Kou, Weiwei Zhang, Nonlinear Dynamics 2019, 96 (3), 2157–2177.
  * 117 G. E. Hinton, R. R. Salakhutdinov, Science 2006, 313 (5786), 504–507.
  * 118 Dinggang Shen, Guorong Wu, Heung-Il Suk, Annual Review of Biomedical Engineering 2017, 19, 221–248.
  * 119 Yiwen Xu, Ahmed Hosny, Roman Zeleznik, Chintan Parmar, Thibaud Coroller, Idalid Franco, Raymond H. Mak, Hugo J.W.L. Aerts, Clinical Cancer Research 2019, 25 (11), 3266–3275.
  * 120 Zhuo Liu, Chenhui Yao, Hang Yu, Taihua Wu, Future Generation Computer Systems 2019, 97, 1 – 9.
  * 121 Harbir Antil, Zichao Wendy Di, Ratna Khatri, Inverse Problems 2020, 36 (6), 064001.
  * 122 Fanyu Bu, Xin Wang, Future Generation Computer Systems 2019, 99, 500 – 507.
  * 123 Markus Reichstein, Gustau Camps-Valls, Bjorn Stevens, Martin Jung, Joachim Denzler, Nuno Carvalhais, et al., Nature 2019, 566 (7743), 195–204.
  * 124 Jonathan Schmidt, Mário RG Marques, Silvana Botti, Miguel AL Marques, Computational Materials 2019, 5 (1), 1–36.
  * 125 Naveed Akhtar, Ajmal Mian, IEEE Access 2018, 6, 14410–14430.
  * 126 Xiaoyong Yuan, Pan He, Qile Zhu, Xiaolin Li, IEEE Transactions on Neural Networks and Learning Systems 2019, 30 (9), 2805–2824.
  * 127 Han Xu Yao Ma Hao-Chen, Liu Debayan Deb, Hui Liu Ji-Liang Tang Anil, K Jain, International Journal of Automation and Computing 2020, 17 (2), 151–178.
  * 128 Vladimir M Krasnopolsky, Michael S Fox-Rabinovitz, Ecological Modelling 2006, 191 (1), 5–18.
  * 129 Vladimir M Krasnopolsky, Michael S Fox-Rabinovitz, Neural Networks 2006, 19 (2), 122–134.
  * 130 William D Collins, Journal of the Atmospheric Sciences 2001, 58 (21), 3224–3242.
  * 131 Alexei Belochitski, Peter Binev, Ronald DeVore, Michael Fox-Rabinovitz, Vladimir Krasnopolsky, Philipp Lamby, Journal of Computational and Applied Mathematics 2011, 236 (4), 447–460.
  * 132 Suraj Pawar, Shady E Ahmed, Omer San, Adil Rasheed, Mathematics 2020, 8 (4), 570.
  * 133 Suraj Pawar, Shady E Ahmed, Omer San, Adil Rasheed, Physics of Fluids 2020, 32 (3), 036602.
  * 134 Sk Rahman, Omer San, Adil Rasheed, Fluids 2018, 3 (4), 86.
  * 135 Omer San, Romit Maulik, Advances in Computational Mathematics 2018, 44 (6), 1717–1750.
  * 136 Romit Maulik, Omer San, Jamey D Jacob, Christopher Crick, Journal of Fluid Mechanics 2019, 870, 784–812.
  * 137 Jaideep Pathak, Alexander Wikner, Rebeckah Fussell, Sarthak Chandra, Brian R Hunt, Michelle Girvan, Edward Ott, Chaos: An Interdisciplinary Journal of Nonlinear Science 2018, 28 (4), 041101.
  * 138 Pantelis R Vlachas, Wonmin Byeon, Zhong Y Wan, Themistoklis P Sapsis, Petros Koumoutsakos, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 2018, 474 (2213), 20170844.
  * 139 Zhong Yi Wan, Pantelis Vlachas, Petros Koumoutsakos, Themistoklis Sapsis, PloS one 2018, 13 (5).
  * 140 Peter AG Watson, Journal of Advances in Modeling Earth Systems 2019, 11 (5), 1402–1417.
  * 141 Alexander Wikner, Jaideep Pathak, Brian Hunt, Michelle Girvan, Troy Arcomano, Istvan Szunyogh, Andrew Pomerance, Edward Ott, Chaos: An Interdisciplinary Journal of Nonlinear Science 2020, 30 (5), 053111\.
  * 142 Franz Hamilton, Alun L Lloyd, Kevin B Flores, PLoS Computational Biology 2017, 13 (7), e1005655.
  * 143 Jin-Long Wu, Heng Xiao, Eric Paterson, Physical Review Fluids 2018, 3 (7), 074602.
  * 144 Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ali Ramadhan, arXiv preprint arXiv:2001.04385 2020.
  * 145 Sebastian Scher, Gabriele Messori, Geoscientific Model Development 2019, 12 (7), 2797–2809.
  * 146 William W Hsieh, Benyang Tang, Bulletin of the American Meteorological Society 1998, 79 (9), 1855–1870.
  * 147 Youmin Tang, William W Hsieh, Monthly Weather Review 2001, 129 (4), 818–834.
  * 148 F Chevallier, F Chéruy, NA Scott, A Chédin, Journal of Applied Meteorology 1998, 37 (11), 1385–1397.
  * 149 Vladimir M Krasnopolsky, Dmitry V Chalikov, Hendrik L Tolman, Ocean Modelling 2002, 4 (3-4), 363–383.
  * 150 Vladimir M Krasnopolsky, Michael S Fox-Rabinovitz, Dmitry V Chalikov, Monthly Weather Review 2005, 133 (5), 1370–1383.
  * 151 Stanley G Benjamin, Stephen S Weygandt, John M Brown, Ming Hu, Curtis R Alexander, Tatiana G Smirnova, Joseph B Olson, Eric P James, David C Dowell, Georg A Grell, et al., Monthly Weather Review 2016, 144 (4), 1669–1694.
  * 152 Knut Nordanger, Runar Holdahl, Trond Kvamsdal, Arne Morten Kvarving, Adil Rasheed, Computer Methods in Applied Mechanics and Engineering 2015, 290, 183 – 208.
  * 153 Knut Nordanger, Runar Holdahl, Arne Morten Kvarving, Adil Rasheed, Trond Kvamsdal, Computer Methods in Applied Mechanics and Engineering 2015, 284, 664 – 688, Isogeometric Analysis Special Issue.
  * 154 Knut Nordanger, Adil Rasheed, Knut Morten Okstad, Arne Morten Kvarving, Runar Holdahl, Trond Kvamsdal, Ocean Engineering 2016, 124, 324 – 339.
  * 155 Bo O. Almroth, Perry Stern, Francis A. Brogan, AIAA Journal 1978, 16 (5), 525–528.
  * 156 Ahmed K. Noor, Jeanne M. Peters, AIAA Journal 1980, 18 (4), 455–462.
  * 157 Bruce C. Moore, IEEE Transactions on Automatic Control 1981, 26 (1), 17–32.
  * 158 Amine Ammar, Béchir Mokdad, Francisco Chinesta, Roland Keunings, Journal of Non-Newtonian Fluid Mechanics 2007, 144 (2-3), 98–121.
  * 159 Gianluigi Rozza, D. B. Phuong Huynh, Anthony T. Patera, Archives of Computational Methods in Engineering 2008, 15 (3), 229–275.
  * 160 Irina Kalashnikova, Matthew F. Barone, International Journal for Numerical Methods in Engineering 2010, 83 (10), 1345–1375.
  * 161 Francisco Chinesta, Pierre Ladeveze, Elías Cueto, Archives of Computational Methods in Engineering 2011, 18 (4), 395–404.
  * 162 Alfio Quarteroni, Gianluigi Rozza, Reduced order methods for modeling and computational reduction, Vol. 9, Springer, Switzerland, 2014.
  * 163 Jonas Ballani, D.B. Phuong Huynh, David J. Knezevic, Loi Nguyen, Anthony T. Patera, Computer Methods in Applied Mechanics and Engineering 2018, 329, 498–531.
  * 164 Giovanni Stabile, Francesco Ballarin, Giacomo Zuccarino, Gianluigi Rozza, Advances in Computational Mathematics 2019, 45 (5-6), 2349–2368.
  * 165 Eivind Fonn, Harald van Brummelen, Trond Kvamsdal, Adil Rasheed, Computer Methods in Applied Mechanics and Engineering 2019, 346, 486–512.
  * 166 Ivo Babuška, Werner C. Rheinboldt, SIAM Journal on Numerical Analysis 1978, 15, 736–754.
  * 167 Ivo Babuška, Werner C. Rheinboldt, International Journal for Numerical Methods in Engineering 1978, 12, 1597–1615.
  * 168 Pierre Ladevèze, Dominique Leguillon, SIAM Journal on Numerical Analysis 1983, 20, 485–509.
  * 169 Olgierd C. Zienkiewicz, Jiangzhong Z. Zhu, International Journal for Numerical Methods in Engineering 1987, 24 (2), 337–357.
  * 170 Mark Ainsworth, Jiangzhong Z. Zhu, Alan W. Craig, Olgierd C. Zienkiewicz, International Journal for Numerical Methods in Engineering 1989, 28 (9), 2161–2174.
  * 171 Mark Ainsworth, Alan W. Craig, Numerische Mathematik 1992, 60 (1), 429–463.
  * 172 Olgierd C. Zienkiewicz, Jianzhong Z. Zhu, International Journal for Numerical Methods in Engineering 1992, 33 (7), 1331–1364.
  * 173 Olgierd C. Zienkiewicz, Jiangzhong Z. Zhu, International Journal for Numerical Methods in Engineering 1992, 33 (7), 1365–1382.
  * 174 Claes G. Johnson, Peter Hansbo, Computer Methods in Applied Mechanics and Engineering 1992, 101 (1-3), 143–181.
  * 175 Marius Paraschivoiu, Jaime Peraire, Anthony T. Patera, Computer Methods in Applied Mechanics and Engineering 1997, 150 (1-4), 289–312.
  * 176 Trond Kvamsdal, Knut Morten Okstad, International Journal for Numerical Methods in Engineering 1998, 42 (3), 443–472.
  * 177 Knut Morten Okstad, Trond Kvamsdal, Kjell Magne Mathisen, International Journal for Numerical Methods in Engineering 1999, 44 (5), 697–727.
  * 178 Mark Ainsworth, J Tinsley Oden, A posteriori error estimation in finite element analysis, John Wiley & Sons, 2000.
  * 179 Ivo Babuška, Theofanis Strouboulis, The Finite Element Method and Its Reliability, Clarendon Press, 2001.
  * 180 Hallgeir Melbø, Trond Kvamsdal, Computer Methods in Applied Mechanics and Engineering 2003, 192 (5-6), 613–633.
  * 181 Wolfgang Bangerth, Rolf Rannacher, Adaptive Finite Element Methods for Differential Equations, Birkhäuser Basel, 2013.
  * 182 Kjetil André Johannessen, Trond Kvamsdal, Tor Dokken, Computer Methods in Applied Mechanics and Engineering 2014, 269, 471–514.
  * 183 Mukesh Kumar, Trond Kvamsdal, Kjetil André Johannessen, Computers & Mathematics with Applications 2015, 70 (7), 1555–1582.
  * 184 Mukesh Kumar, Trond Kvamsdal, Kjetil André Johannessen, Computer Methods in Applied Mechanics and Engineering 2017, 316, 1086–1156.
  * 185 Thomas A. Porsching, Mathematics of Computation 1985, 45 (172), 487–487.
  * 186 Luc Machiels, Yvon Maday, Anthony T. Patera, Computer Methods in Applied Mechanics and Engineering 2001, 190 (26-27), 3413–3426.
  * 187 Jens L. Eftang, David J. Knezevic, Anthony T. Patera, Mathematical and Computer Modelling of Dynamical Systems 2011, 17 (4), 395–422.
  * 188 Jens L. Eftang, Dinh B. P. Huynh, David J. Knezevic, Anthony T. Patera, Journal of Scientific Computing 2011, 51 (1), 28–58.
  * 189 Bernard Haasdonk, Mario Ohlberger, Mathematical and Computer Modelling of Dynamical Systems 2011, 17 (2), 145–161.
  * 190 David J. Knezevic, Ngoc-Cuong Nguyen, Anthony T. Patera, Mathematical Models and Methods in Applied Sciences 2011, 21 (07), 1415–1442.
  * 191 D. B. Phuong Huynh, David J. Knezevic, John W. Peterson, Anthony T. Patera, Computers & Fluids 2011, 43 (1), 74–81.
  * 192 D.B. Phuong Huynh, David J. Knezevic, Anthony T. Patera, Computer Methods in Applied Mechanics and Engineering 2012, 201-204, 13–24.
  * 193 Dick P Dee, Arlindo M Da Silva, Quarterly Journal of the Royal Meteorological Society 1998, 124 (545), 269–295.
  * 194 Dick P Dee, Ricardo Todling, Monthly Weather Review 2000, 128 (9), 3268–3282.
  * 195 Francisco Chinesta, Elias Cueto, Emmanuelle Abisset-Chavanne, Jean Louis Duval, Fouad El Khaldi, Archives of Computational Methods in Engineering 2018, 1–30.
  * 196 Alberto Badías, Icíar Alfaro, David González, Francisco Chinesta, Elías Cueto, Computer Methods in Applied Mechanics and Engineering 2018, 341, 53–70.
  * 197 José V Aguado, Antonio Huerta, Francisco Chinesta, Elías Cueto, International Journal for Numerical Methods in Engineering 2015, 102 (5), 991–1017.
  * 198 Jose Vicente Aguado, Domenico Borzacchiello, Chady Ghnatios, François Lebel, Ram Upadhyay, Christophe Binetruy, Francisco Chinesta, Advanced Modeling and Simulation in Engineering Sciences 2017, 4 (1), 1\.
  * 199 Siamak Niroomandi, Iciar Alfaro, Elías Cueto, Francisco Chinesta, Computer Methods and Programs in Biomedicine 2012, 105 (1), 1–12.
  * 200 Shady Ahmed, Suraj Pawar, Omer San, Adil Rasheed, Mandar Tabib, Computers & Fluids 2021, 221, 104895.
  * 201 Romit Maulik, Omer San, Adil Rasheed, Prakash Vedula, Physics of Fluids 2018, 30 (12), 125109.
  * 202 Suraj Pawar, Omer San, Adil Rasheed, Prakash Vedula, Theoretical and Computational Fluid Dynamics 2020, 1–27.
  * 203 William Layton, Michael Schneier, Results in Applied Mathematics 2020, 8, 100099.
  * 204 Alex Gittens, Aditya Devarakonda, Evan Racah, Michael Ringenburg, Lisa Gerhardt, Jey Kottalam, Jialin Liu, Kristyn Maschhoff, Shane Canon, Jatin Chhugani, Pramod Sharma, Jiyan Yang, James Demmel, Jim Harrell, Venkat Krishnamurthy, Michael W. Mahoney, Prabhat, in 2016 IEEE International Conference on Big Data (Big Data), 2016, pp. 204–213.
  * 205 Daniel J. Tait, Theodoros Damoulas, arXiv preprint arXiv:2006.15641 2020.
  * 206 Mauro Maggioni, Stanislav Minsker, Nate Strawn, Journal of Machine Learning Research 2016, 17 (1), 43–93.
  * 207 Fei Lu, Ming Zhong, Sui Tang, Mauro Maggioni, Proceedings of the National Academy of Sciences of the United States of America 2019, 116 (29), 14424–14433.
  * 208 Bo Zhu, Jeremiah Z. Liu, Stephen F. Cauley, Bruce R. Rosen, Matthew S. Rosen, Nature 2018, 555 (7697), 487–492.
  * 209 Geoffrey E. Hinton, Sam T. Roweis, in Advances in Neural Information Processing Systems 15, 2002, pp. 857–864.
  * 210 Laurens van der Maaten, Geoffrey Hinton, Journal of Machine Learning Research 2008, 9, 2579–2605.
  * 211 George C. Linderman, Stefan Steinerberger, SIAM Journal on Mathematics of Data Science 2019, 1 (2), 313–332.
  * 212 Shuixia Guo, Anil K. Seth, Keith M. Kendrick, Cong Zhou, Jianfeng Feng, Journal of Neuroscience Methods 2008, 172 (1), 79–93.
  * 213 Sk M Rahman, Suraj Pawar, Omer San, Adil Rasheed, Traian Iliescu, Physical Review E 2019, 100 (5), 053306.
  * 214 O San, J Borggaard, International Journal for Numerical Methods in Fluids 2015, 78 (1), 37–62.
  * 215 Mansoor Ahmed, Omer San, Computational and Applied Mathematics 2018, 37 (5), 6870–6902.
  * 216 Shady E Ahmed, Omer San, Adil Rasheed, Traian Iliescu, Physica D: Nonlinear Phenomena 2020, 409, 132471.
  * 217 Christopher J. Roy, Journal of Computational Physics 2005, 205 (1), 131 – 156.
  * 218 Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio, in Advances in neural information processing systems, 2014, pp. 2672–2680.
  * 219 YC Liang, HP Lee, SP Lim, WZ Lin, KH Lee, CG Wu, Journal of Sound and Vibration 2002, 252 (3), 527–544.
  * 220 Julius Reiss, Philipp Schulze, Jörn Sesterhenn, Volker Mehrmann, SIAM Journal on Scientific Computing 2018, 40 (3), A1322–A1344.
  * 221 Shawn C Shadden, Francois Lekien, Jerrold E Marsden, Physica D: Nonlinear Phenomena 2005, 212 (3-4), 271–304.
  * 222 George Haller, Annual Review of Fluid Mechanics 2015, 47, 137–162.
  * 223 M Ani Hsieh, Hadi Hajieghrary, Dhanushka Kularatne, Christoffer R Heckman, Eric Forgoston, Ira B Schwartz, Philip A Yecko in Robotics Research, Springer, 2018, pp. 387–402.
  * 224 Fabian Waleffe, Journal of Fluid Mechanics 2001, 435, 93.
  * 225 Bruno Eckhardt, Stefan Zammert, Nonlinearity 2018, 31 (2), R66.
  * 226 Cédric Beaume, Edgar Knobloch, Gregory P Chini, Keith Julien, Fluid Dynamics Research 2014, 47 (1), 015504.
  * 227 Balachandra Suri, Jeffrey Tithof, Roman O Grigoriev, Michael F Schatz, Physical Review Letters 2017, 118 (11), 114501.
  * 228 SA Thorpe, Jonathan Malarkey, Gunnar Voet, MH Alford, JB Girton, GS Carter, Journal of Fluid Mechanics 2018, 834, 125–148.
  * 229 Marco Restelli, Francis X Giraldo, SIAM Journal on Scientific Computing 2009, 31 (3), 2231–2257.
  * 230 James D Doyle, Qingfang Jiang, P Alexander Reinecke in Aviation Turbulence, Springer, 2016, pp. 357–384.
  * 231 JD Mireles James, Journal of Nonlinear Science 2013, 23 (4), 585–615.
  * 232 Sevdzhan Hakkaev, Milena Stanislavova, Atanas Stefanov, SIAM Journal on Mathematical Analysis 2017, 49 (1), 674–698.
  * 233 Blake Barker, JD Mireles James, Jalen Morgan, SIAM Journal on Applied Dynamical Systems 2020, 19 (3), 1758–1797.
  * 234 Kai Cieliebak, Viktor L Ginzburg, Ely Kerman, Commentarii Mathematici Helvetici 2004, 79 (3), 554–581.
  * 235 Romit Maulik, Romain Egele, Bethany Lusch, Prasanna Balaprakash, arXiv preprint arXiv:2004.10928 2020.
  * 236 Haakon Robinson, Adil Rasheed, Omer San, arXiv e-prints 2019, arXiv:1910.03879.
  * 237 Shady E Ahmed, Suraj Pawar, Omer San, Fluids 2020, 5 (4), 225\.
  * 238 Richard A Anthes, Journal of the Atmospheric Sciences 1974, 31 (3), 702–719.
  * 239 Lili Lei, Joshua P Hacker, Monthly Weather Review 2015, 143 (7), 2600–2610.
  * 240 David R Stauffer, Jian-Wen Bao, Tellus A 1993, 45 (5), 358–369.
  * 241 S. E. Ahmed, S Pawar, Omer San, Adil Rasheed, 2020 AIAA Aviation Forum and Exposition, Virtual Event, June 15–19, 2020.
  * 242 X Zou, I M Navon, FX Le Dimet, Quarterly Journal of the Royal Meteorological Society 1992, 118 (508), 1163–1186.
  * 243 PA Vidard, FX Le Dimet, A Piacentini, Tellus A: Dynamic Meteorology and Oceanography 2003, 55 (1), 1–15.
  * 244 Didier Auroux, Jacques Blum, Comptes Rendus Mathematique 2005, 340 (12), 873–878.
  * 245 S Lakshmivarahan, John M Lewis in Data Assimilation for Atmospheric, Oceanic and Hydrologic Applications (Vol. II), Springer, 2013, pp. 27–57.
  * 246 Suraj Pawar, Shady E Ahmed, Omer San, Adil Rasheed, Ionel M Navon, Physics of Fluids 2020, 32, 076606.
  * 247 Edward N Lorenz, in Proc. Seminar on Predictability, 1996.
  * 248 Pavel Sakov, Peter R Oke, Tellus A: Dynamic Meteorology and Oceanography 2008, 60 (2), 361–371.
  * 249 Suraj A Pawar, Omer San, in AIAA Scitech 2021 Forum, 2021, p. 1749.
  * 250 Alan J Geer, ECMWF Technical Memo 2020, 863.
  * 251 Henry DI Abarbanel, Paul J Rozdeba, Sasha Shirman, Neural Computation 2018, 30 (8), 2025–2055.
  * 252 Eivind Meyer, Haakon Robinson, Adil Rasheed, Omer San, IEEE Access 2020, 8, 41466–41481.
  * 253 Simen Theie Havenstrøm, Adil Rasheed, Omer San, arXiv 2020, arXiv–2006.
  * 254 Eivind Meyer, Amalie Heiberg, Adil Rasheed, Omer San, arXiv preprint arXiv:2006.09540 2020.
  * 255 Harsha Vaddireddy, Adil Rasheed, Anne E Staples, Omer San, Physics of Fluids 2020, 32, 015113.
  * 256 Michael Schmidt, Hod Lipson, Science 2009, 324 (5923), 81–85.
  * 257 Miles Cranmer, Alvaro Sanchez-Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, Shirley Ho, arXiv preprint arXiv:2006.11287 2020.
  * 258 Ioannis G Kevrekidis, C William Gear, James M Hyman, Panagiotis G Kevrekidid, Olof Runborg, Constantinos Theodoropoulos, Communications in Mathematical Sciences 2003, 1 (4), 715–762.
  * 259 Sirod Sirisup, George Em Karniadakis, Dongbin Xiu, Ioannis G Kevrekidis, Journal of Computational Physics 2005, 207 (2), 568–587.
  * 260 Ioannis G Kevrekidis, Giovanni Samaey, Annual Review of Physical Chemistry 2009, 60, 321–344.
  * 261 Thierry Crestaux, Olivier Le Maıtre, Jean-Marc Martinez, Reliability Engineering & System Safety 2009, 94 (7), 1161–1172.
  * 262 Frédéric Chevallier, Jean-François Mahfouf, Journal of Applied Meteorology 2001, 40 (8), 1445–1461.
  * 263 Filipe Aires, Michel Schmitt, Alain Chedin, Noëlle Scott, IEEE Transactions on Neural Networks 1999, 10 (6), 1502–1510.
  * 264 Romit Maulik, Omer San, Journal of Fluid Mechanics 2017, 831, 151–181.
  * 265 Suraj Pawar, Omer San, Burak Aksoylu, Adil Rasheed, Trond Kvamsdal, Physics of Fluids 2021, 33 (1), 011701.
  * 266 Ge-Cheng Zha, Bruce F Carroll, Craig D Paxton, Clark A Conley, Adam Wells, AIAA Journal 2007, 45 (8), 2087–2090.
  * 267 Patrick LeGresley, Juan Alonso, in Fluids 2000 conference and exhibit, 2000, p. 2545.
  * 268 Yao Zhang, Woong Je Sung, Dimitri N Mavris, in 2018 AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, 2018, p. 1903.
  * 269 Saakaar Bhatnagar, Yaser Afshar, Shaowu Pan, Karthik Duraisamy, Shailendra Kaushik, Computational Mechanics 2019, 64, 525–545.
  * 270 Weinan E, Bjorn Engquist, Zhongyi Huang, Physical Review B 2003, 67 (9), 092101.
  * 271 Alfio Quarteroni, Alessandro Veneziani, Christian Vergara, Computer Methods in Applied Mechanics and Engineering 2016, 302, 193–252.
  * 272 Alfio Quarteroni, Alessandro Veneziani, Multiscale Modeling & Simulation 2003, 1 (2), 173–195.
  * 273 Thomas JR Hughes, Luca Mazzei, Kenneth E Jansen, Computing and Visualization in Science 2000, 3 (1-2), 47–59.
  * 274 Thomas JR Hughes, Luca Mazzei, Assad A Oberai, Alan A Wray, Physics of Fluids 2001, 13, 505–512.
  * 275 EH Van Brummelen, International Journal for Numerical Methods in Fluids 2011, 65 (1-3), 3–27.
  * 276 Sriram Shankaran, Juan Alonso, May-Fun Liou, Nan-Suey Liu, Roger Davis, in 39th Aerospace Sciences Meeting and Exhibit, 2001, p. 974.
  * 277 L. W. Cook, A. A. Mishra, J. P. Jarrett, K. E. Willcox, G. Iaccarino, Physics of Fluids 2019, 31 (10), 105111.
  * 278 Jiayang Xu, Cheng Huang, Karthik Duraisamy, AIAA Journal 2020, 58 (2), 618–632.
  * 279 Knut Steinar Bjørkevoll, in The 2nd IFAC Workshop on Automatic Control in Offshore Oil and Gas Production, 2015.
  * 280 John Macpherson, Journal of Petroleum Technology 2015, 67 (09), 136–136.
  * 281 M Salman Siddiqui, Adil Rasheed, Mandar Tabib, Trond Kvamsdal, Renewable Energy 2019, 132, 1058–1075.
  * 282 Martin J Gander, Laurence Halpern in Domain Decomposition Methods in Science and Engineering XX, Springer, 2013, pp. 377–385.