# An Efficient Approach for Super and Nested Term Indexing and Retrieval

Md Faisal Mahbub Chowdhury
IBM Research AI
<EMAIL_ADDRESS>
Robert Farrell
IBM Research AI
<EMAIL_ADDRESS>

###### Abstract

This paper describes a new approach, called _Terminological Bucket Indexing
(TBI)_ , for efficient indexing and retrieval of both nested and super terms
using a single method. We propose a hybrid data structure for facilitating
faster indexing building. An evaluation of our approach with respect to widely
used existing approaches on several publicly available dataset is provided.
Compared to Trie based approaches, TBI provides comparable performance on
nested term retrieval and far superior performance on super term retrieval.
Compared to traditional hash table, TBI needs 80% less time for indexing.

##  1 Introduction

A super term is a term that contains another term, e.g. “Google LLC” is a
super term of the two corresponding nested terms (aka sub-terms) “Google” and
“LLC”. Super and nested terms are used in many Natural Language Processing
(NLP) applications/tasks, e.g. terminology extraction Frantzi et al. (1998),
taxonomy/ontology population Basili et al. (2003); Lassoued and Deleris
(2016), query expansion/re-formulation Polozov et al. (2012), query
completion, etc.

The task of finding super and nested terms can be defined as follows. Given an
input term, retrieve all corresponding super or nested terms from a list of
terms (i.e. vocabulary), typically extracted from document corpora. To
facilitate such retrieval, one typically indexes the aforementioned vocabulary
in a data structure. If the indexing algorithm is inefficient, it can result
in substantially more time to build an index for a large vocabulary and can be
a bottleneck for NLP applications.

The most commonly used data structure for term indexing is a key-value store,
aka hash table Luhn (1953); Knuth (1998), where key is a term and value is a
set of super/nested terms. For example, Zhang et al. (2016) used such an
implementation111https://github.com/ziqizhang/jate. The indexing time
complexity is $O(v^{2})$ where _v_ is the size of the vocabulary. Assuming the
values in the hash table are super terms (same holds for nested terms), the
space complexity is $O(v*s)$ where s is the average number of super terms per
term and $v\gg s$. Search time complexities for retrieval of super terms and
nested terms (from same hash table) are $O(1)$ and $O(m^{2})$ correspondingly,
where $m$ is total tokens in the input term, $O(m^{2})$ is time complexity for
all possible n-gram generation from input term and $v\gg m$.222Note, the
meanings of symbols ($m,v,t,s$ and $L$) used inside the Big O notation are
consistent across this paper.

Lu and Browne (2012) described an approach for nested term finding where the
indexing is done by loading the vocabulary in a Trie De La Briandais (1959)
where words are nodes. The indexing complexity is $O(v*t)$, where $t$ is the
average number of words for a term in the vocabulary and $v\gg t$.333See
https://tinyurl.com/yask8u7x for step-by-step in details of Lu and Browne
(2012). The search complexity for nested terms is $O(m^{2})$. However, search
complexity for super terms is $O(v*t)$, which is very slow. So, realistically,
this approach can be only used for nested term retrieval.

The Aho and Corasick (1975), a string search algorithm based on Trie, can be
used for nested term finding. The algorithm is widely used in intrusion
detection and anti-virus systems, and many other applications that require
fast matching against a pre-defined set of strings/terms. It generalizes the
Matiyasevich (1973) and Knuth et al. (1977) algorithms for multiple exact sub-
string (i.e. nested term) matching. Time complexity of nested term retrieval
is $O(m+L)$ where $L$ is total length of all terms in vocabulary. But it is
very slow for super term retrieval (like Lu and Browne (2012)) as it is based
on Trie.444Other string search algorithms (e.g. Karp–Rabin algorithm Karp and
Rabin (1987)) have similar or worse search complexity.

A somewhat related work by Sun et al. (2008) addressed chemical name indexing
and search. For complex chemical names, e.g. “methylethyl”, they proposed an
algorithm to identify substrings, e.g. “methyl” and “ethyl”, that have higher
frequency. They proposed to index these substrings instead of all possible
substrings. These substrings are similar to nested terms in the chemical
domain. The authors did not discuss what algorithm or data structure they used
to index the substrings nor whether their algorithm could be used to
efficiently retrieve super terms.

In the remaining of the paper, we describe our proposed approach and empirical
observations. The contributions of this paper are following – (i) a new
indexing algorithm, _Terminological Bucket Indexing (TBI)_ , that uses a
hybrid data structure and can be used for both fast super term and nested term
retrieval, (ii) empirical results demonstrating that TBI reduces indexing time
by a minimum 81% when compared with hash table, and (iii) complexity analysis
showing that the search time for super and nested term retrieval using the
hybrid data structure is the same as a hash table and better than Trie based
algorithms.

Data | Total | Avg. no. | Avg. no.
---|---|---|---
|  |  | of
| unique | of words | characters
| terms | per term | per term
Brown | 95,241 | 2.23 | 15.21
WSJ | 92,690 | 2.49 | 17.11
SWBD | 28,429 | 2.21 | 13.23
NYT | 16,242 | 2.01 | 13.27
Google w2v | 3 millions | 1.96 | 13.74
Table 1: Corpora used for the experiments.

##  2 Proposed Approach

Index Building: Proposed _TBI_ uses a hybrid data structure (henceforth, term
store) composed of two hash tables. $Table\\_SuperTerms$ stores terms as keys
and set of corresponding super-terms as values. Its space complexity is
$O(v*s)$. Here, $v\gg s$. $Table\\_Bucket$ has 4 levels as shown below; e.g.,
k1=21, k2=4, k3=24 and k4=7 for term “united states of america”. 555The
ordering, selection and combination of the 4 types of bucket keys were
resulted from through checking output accuracy and runtime performance among a
wide variety of keys. One can verify accuracy by simply comparing the list of
super terms (for every term of a dataset) indexed using the traditional hash
table indexing with that inside $Table\\_SuperTerms$ using the proposed TBI.

The space complexity of $Table\\_Bucket$ is $O(v)$ since a term is inserted
only once. So, complexity for proposed hybrid term store is $O(v*s+v)$. This
is a bit higher but still comparable to $O(v*s)$ space complexity of hash
table. $Table\\_Bucket$ can be discarded after all terms in vocabulary are
indexed.

| For nested term | For super term
---|---|---
Data | Aho–Corasick | Lu-Browne | Ours | Aho–Corasick | Lu-Browne | Ours
Brown | 15.21 | 2.23 | 3.45 | 42433.99 | 33629.19 | 1
WSJ | 17.11 | 2.48 | 4.09 | 29170.87 | 22657.57 | 1
SWBD | 13.23 | 2.21 | 3.39 | 9556.54 | 12172.2 | 1
NYT | 13.27 | 2.01 | 3.01 | 8673.56 | 6496.11 | 1
Table 2: Average number of node comparisons needed for term retrieval for
different approaches.

Algorithm 1 describes proposed TBI. First, the vocabulary is sorted by term
length in ascending order (line no. 2). The complexity of this sorting step is
$O(v*log\\_v)$ which is very fast. The goal is – for each (yet to be indexed)
new_term from the vocabulary, efficiently find nested terms from terms that
were already indexed (from this sorted vocabulary). Because they are pre-
sorted, none of the already indexed terms can be super term of new_term (as
they are shorter or equal). To populate the nested terms for new_term, the
algorithm first looks for the buckets that have terms with lower ‘number of
non-space characters‘ (line 25). Then, it further reduces the search space by
identifying buckets containing terms with lower ‘number of tokens‘ (line 27),
which is followed by locating buckets having terms with lower ‘term length‘
(line 29), and finally, finding buckets having terms with smaller or equal
‘largest token length‘ (line 31).

Super and Nested Term Retrieval: _TBI_ has $O(1)$ time complexity for super
term retrieval (from $Table\\_SuperTerms$). The nested term retrieval
complexity is same as that of the hash table (i.e. $O(m^{2})$). To retrieve
nested terms, candidate nested terms are generated from all possible n-grams
of the input term. If a candidate term is not present as a key in
$Table\\_SuperTerms$, it is discarded. Remaining terms are returned as result.

##  3 Data

We did experiments on 4 different corpora and 1 big dataset (Table 1). Three
of them are from the Penn Treebank-3 dataset Marcus et al. (1999) – Brown (1
million words of American English from 1961), WSJ (1 million words from the
1989 Wall Street Journal) and SWBD (transcribed spontaneous conversations from
American English dialects). The NYT corpus (news from New York Times) is
obtained the Gigaword corpus Graff and Cieri (2003). For these 4 corpora, we
used noun phrases (NPs) as terms. They were extracted from using a popular
open source NLP tool666https://spacy.io/.

The big dataset is Google w2v model Mikolov et al. (2013) containing 3 million
terms from a Google News dataset of about 100 billion words.

##  4 Empirical Evaluation and Conclusion

Term Retrieval: The first set of experiments show that _TBI_ is preferable for
super and nested term retrieval than Trie based state-of-the-art methods. We
compare with Lu and Browne (2012) and Aho and Corasick (1975). An open source
implementation777https://github.com/robert-bor/aho-corasick is used for the
former, and UMLS Sub-Term Mapping Tools888https://specialist.nlm.nih.gov/stmt/
for the latter.

These (Trie) implementations allow only nested term retrieval. So, we enhanced
their codes to retrieve super terms. We added an attribute for the nodes in
the Trie that keeps track of the maximum depth of the sub-tree for any given
node. This allows the system(s) to ignore traversing the sub-tree of a node
(for a super term match) when number of remaining words (characters, for
Aho–Corasick) of the input term, yet to be matched, is bigger than the maximum
depth of that sub-tree.

Table 2 shows the average number of node comparisons required per term for the
different approaches. Note, a hash table is basically a graph. For nested term
retrieval, our approach performs almost as well as Lu and Browne (2012).
However, for super term retrieval, our approach is vastly superior to either
of the other approaches. We skipped these retrieval experiments on the big w2v
dataset since the expected results are obvious from the results of other 4
datasets.

Indexing: A second set of experiments compare indexing performance of TBI with
a traditional hash table implementation done by Zhang et al. (2016).999All
experiments were done on a macOS machine with 2.5 GHz Intel Core i7 processor
and 16 GB RAM. Table 3 shows in each of these dataset, TBI requires
substantially, at least 81%, less time with respect to that of the traditional
hash table indexing approach.

Conclusion: In this paper, we presented a new very fast approach for nested
and super term indexing. Unlike commonly used Trie based algorithms, proposed
TBI can be used for efficiently retrieving both super and nested terms. We
were able to significantly reduce runtime for NLP applications such as
terminology and taxonomy extraction using proposed _Terminological Bucket
Indexing_ and hope the NLP community will benefit from it, too.

| Total runtime
---|---
| Vanilla hash |
Data | table indexing | Proposed TBI
Brown | 37.18 min |  6.97 min (81% less)
WSJ | 35.18 min |  6.4 min (82% less)
SWBD | 3.57 min |  0.55 min (85% less)
NYT | 1.13 min |  0.18 min (84% less)
Google w2v | 453.02 hours |  65.33 hours (86% less)
Table 3: Comparisons of performance between traditional hash table indexing
and proposed TBI. Algorithm 1 Proposed indexing algorithm to index a
vocabulary inside the proposed term store

1:function $build\\_index$($list\\_of\\_terms$)

2: Let $sorted\\_list\\_of\\_terms$ := terms in $list\\_of\\_terms$ sorted by
term length in ascending order

3: for Each $new\\_term$ in $sorted\\_list\\_of\\_terms$ do

4: if $new\\_term$ is not added as a key $Table\\_SuperTerms$ then

5: Let $len$ := number of characters (i.e. term length) in $new\\_term$

6: Let $tot\\_tok$ := number of tokens in $new\\_term$

7: Let $lt\\_len$ := number of characters in the largest token in $new\\_term$

8: Let $tot\\_nsp\\_char$ := $len$ \- $tot\\_tok$ \+ 1

9: Add a new key $new\\_term$ in $Table\\_SuperTerms$ with an empty set as
value

10:
$insert\\_term\\_in\\_bucket(new\\_term,Table\\_Bucket,tot\\_nsp\\_char,tot\\_tok,len,lt\\_len)$

11:
$update\\_nested\\_terms(new\\_term,Table\\_Bucket,tot\\_nsp\\_char,tot\\_tok,len,lt\\_len,$
$Table\\_SuperTerms)$

12: end if

13: end for

14:end function

15:

16:function
$insert\\_term\\_in\\_bucket$($new\\_term,Table\\_Bucket,tot\\_nsp\\_char,tot\\_tok,len,lt\\_len$)

17: $bucket\\_for\\_tot\\_nsp\\_char$ :=
$Table\\_Bucket.get\\_or\\_initialize\\_value$(key=$tot\\_nsp\\_char$)

18: $bucket\\_for\\_tot\\_tok$ :=
$bucket\\_for\\_tot\\_nsp\\_char.get\\_or\\_initialize\\_value$(key=$tot\\_tok$)

19: $bucket\\_for\\_term\\_len$ :=
$bucket\\_for\\_tot\\_nsp\\_char.get\\_or\\_initialize\\_value$(key=$len$)

20: $set\\_of\\_terms$ =
$bucket\\_for\\_tot\\_nsp\\_char.get\\_or\\_initialize\\_value$(key=$lt\\_len$)

21: $set\\_of\\_terms.add$($new\\_term$)

22:end function

23:

24:function
$update\\_nested\\_terms$($new\\_term,Table\\_Bucket,tot\\_nsp\\_char,tot\\_tok,len,lt\\_len,$
$Table\\_SuperTerms$)

25: for $k1$ in $Table\\_Bucket$.keys() where $k1<tot\\_nsp\\_char$ do

26: $bucket\\_for\\_k1$ := $Table\\_Bucket.get\\_value$(key=$k1$)

27: for $k2$ in $bucket\\_for\\_k1$.keys() where $k2<tot\\_tok$ do

28: $bucket\\_for\\_k2$ := $bucket\\_for\\_k1.get\\_value$(key=$k2$)

29: for $k3$ in $bucket\\_for\\_k2$.keys() where $k3<len$ do

30: $bucket\\_for\\_k3$ := $bucket\\_for\\_k2.get\\_value$(key=$k3$)

31: for $k4$ in $bucket\\_for\\_k3$.keys() where $k4\leq lt\\_len$ do

32: $candidate\\_nested\\_terms$ := $bucket\\_for\\_k3.get\\_value$(key=$k4$)

33: for Each $t$ in $candidate\\_nested\\_terms$ where $new\\_term$ is super
term of $t$ do

34: $Table\\_SuperTerms.add\\_value$(key_nested_term=$t$,
value_super_term=$new\\_term$)

35: end for

36: end for

37: end for

38: end for

39: end for

40:end function

## References

  * Aho and Corasick (1975) Alfred V. Aho and Margaret J. Corasick. 1975.  Efficient string matching: An aid to bibliographic search.  _Communications of the ACM_ , 18(6):333–340.
  * Basili et al. (2003) Roberto Basili, Michele Vindigni, and Fabio M. Zanzotto. 2003.  Integrating ontological and linguistic knowledge for conceptual information extraction.  In _Proceedings of the IEEE/WIC International Conference on Web Intelligence (WI 2003)_. IEEE.
  * De La Briandais (1959) Rene De La Briandais. 1959.  File searching using variable length keys.  In _Proceedings of the Western Joint Computer Conference_ , IRE-AIEE-ACM ’59 (Western), pages 295–298, New York, NY, USA. ACM.
  * Frantzi et al. (1998) Katerina T. Frantzi, Sophia Ananiadou, and Jun-ichi Tsujii. 1998.  The c-value/nc-value method of automatic recognition for multi-word terms.  In _Proceedings of the 2nd European Conference on Research and Advanced Technology for Digital Libraries_ , ECDL ’98, pages 585–604, London, UK, UK. Springer-Verlag.
  * Graff and Cieri (2003) David Graff and Christopher Cieri. 2003.  English Gigaword LDC2003T05.  Web Download. Philadelphia: Linguistic Data Consortium.
  * Karp and Rabin (1987) Richard M. Karp and Michael O. Rabin. 1987.  Efficient Randomized Pattern-matching Algorithms.  _IBM Journal of Research and Development._ , 31(2):249–260.
  * Knuth (1998) Donald E. Knuth. 1998.  _The Art of Computer Programming, Volume 3: (2Nd Ed.) Sorting and Searching_.  Addison Wesley Longman Publishing Co., Inc., Redwood City, CA, USA.
  * Knuth et al. (1977) Donald E. Knuth, James H. Morris, and Vaughan R. Pratt. 1977.  Fast Pattern Matching in Strings.  _SIAM Journal on Computing_ , 6(2):323–350.
  * Lassoued and Deleris (2016) Yassine Lassoued and Lea Deleris. 2016.  Thesaurus-based hierarchical semantic grouping of medical terms in information extraction.  In _Studies in health technology and informatics_ , volume 228, pages 446–50. European Federation for Medical Informatics (EFMI) and IOS Press.
  * Lu and Browne (2012) Chris J. Lu and Allen C. Browne. 2012.  Development of sub-term mapping tools (stmt).  In _Proceedings of the AMIA 2012 Annual Symposium_.
  * Luhn (1953) H. P. Luhn. 1953.  Internal IBM Memorandum.
  * Marcus et al. (1999) Mitchell P. Marcus, Beatrice Santorini, Mary Ann Marcinkiewicz, and Ann Taylor. 1999\.  Treebank-3 LDC99T42.  Web Download. Philadelphia: Linguistic Data Consortium.
  * Matiyasevich (1973) Yuri Matiyasevich. 1973.  Real-time recognition of the inclusion relation.  _Journal of Soviet Mathematics_ , 6:64–70.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Distributed representations of words and phrases and their compositionality.  In _Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2_ , NIPS’13, pages 3111–3119, USA. Curran Associates Inc.
  * Polozov et al. (2012) Oleksandr Polozov, Sumit Gulwani, and Sriram Rajaman. 2012.  Structure and term prediction for mathematical text.  Technical Report MSR-TR-2012-7, Microsoft Research.
  * Sun et al. (2008) Bingjun Sun, Prasenjit Mitra, and C. Lee Giles. 2008.  Mining, indexing, and searching for textual chemical molecule information on the web.  In _Proceedings of the 17th International Conference on World Wide Web (WWW ’08)_ , WWW ’08, pages 735–744, New York, NY, USA. ACM.
  * Zhang et al. (2016) Ziqi Zhang, Jie Gao, and Fabio Ciravegna. 2016.  JATE 2.0: Java Automatic Term Extraction with Apache Solr.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)_ , Paris, France. European Language Resources Association (ELRA).


# Physics-informed Localized Learning for Advection-Diffusion-Reaction Systems

Surya T. Sathujoda  Soham M. Sheth

###### Abstract

The global push to advance Carbon Capture and Sequestration initiatives and
green energy solutions, such as geothermal, have thrust new demands upon the
current state-of-the-art subsurface fluid simulators. The requirement to be
able to simulate a large order of reservoir states simultaneously, in a short
period of time, has opened the door of opportunity for the application of
machine learning techniques for surrogate modelling. We propose a novel
physics-informed and boundary condition-aware Localized Learning method which
extends the Embed-to-Control (E2C) and Embed-to-Control and Observe (E2CO)
models to learn local representations of global state variables in an
Advection-Diffusion Reaction system. Trained on reservoir simulation data, we
show that our model is able to predict future states of the system, for a
given set of controls, to a great deal of accuracy with only a fraction of the
available information. It hence reduces training times significantly compared
to the original E2C and E2CO models, lending to its benefit in application to
optimal control problems.

Deep Learning, Embed to Control, Reservoir surrogate modelling, Localised
Learning, Physics-informed Machine Learning

##  1 Introduction

Subsurface fluid simulators play an important role in modelling various modern
geological processes. They enable the efficient and sustainable utilization of
geothermal energy by modelling heat and mass transfer processes; they play a
vital role in modern Carbon Capture and Sequestration (CCS) initiatives by
assessing optimal locations to inject $CO_{2}$ into geological formations, and
they facilitate the efficient extraction of natural resources to reduce the
risks of exploration. Developments in these areas have produced novel
challenges to the production workflow of subsurface fluid modelling. The task
of optimizing the location and volume of $CO_{2}$ injection into a porous
medium, for instance, requires vast amounts of simulation data and compute
power for each individual control state and time step of the system to
determine the maximal storage case. This is especially problematic for
applications in which we require real-time decision making. This has hence
given birth to a new class of subsurface surrogate modelling techniques using
machine learning.

The governing equation of subsurface flow, derived from the law of mass-
conservation and Darcy’s law, relates the fluid flow and flow potential
gradients of a multi-phase porous-medium as given below

$\nabla\cdot\biggl{[}\alpha\textbf{k}\,\frac{k_{r,m}(S_{m})}{\mu_{m}B_{m}}(\nabla
p_{m}-\gamma_{m}\nabla z)\biggr{]}-\beta\frac{\partial}{\partial
t}\biggl{(}\frac{\phi
S_{m}}{B_{m}}\biggr{)}+\sum_{w}\frac{q_{sc,m}^{w}}{V_{b}}=0$

where k denotes the permeability tensor, $k_{r}$ the relative permeability,
$\mu$ the viscosity, $B$ the formation volume factor, $p$ the pressure, $S$
the saturation, $\phi$ the porosity, $t$ the time, $\gamma$ the specific
weight, $q$ the source/sink terms, $z$ the depth and $V_{b}$ the bulk volume.
The subscript $sc$ represents standard conditions and $\alpha$ and $\beta$ are
unit field constants. More generally, this equation is an instance of an
Advection-Diffusion-Reaction (ADR) partial differential equation (PDE) which
describes the transport and transformation of scalar quantities (such as
pressure and saturation in this case) in a medium due to advection, diffusion
and reaction, as the name suggests. The ADR equation arises in many other
areas of science and engineering such as environmental engineering, where it
can be applied to help understand the spread of contaminants in groundwater,
biomedical engineering, where it can be applied to model the transport of
drugs in biological tissues, and in chemical engineering, where it can be used
to help optimize reactor design and operation. Although our proposed model
focuses on subsurface fluid dynamics, it easily generalises to all the above
areas with the utilisation of the appropriate domain specific knowledge.

The above equation of subsurface flow describes a highly non-linear system
that even state-of-the-art numerical solvers using Newton’s method require
significant compute power to converge to a solution. For this reason, various
reduced-order-models (ROMs) have been developed which approximate the
evolution of state variables in time to be linear. The most common such ROM is
the Proper Orthogonal Decomposition Trajectory Piece-wise Linearization
Reduced Order Model (POD-TPWL) (Cardoso & Durlofsky, 2010; He et al., 2011; He
& Durlofsky, 2013; Rousset et al., 2014; TPW, 2015; He & Durlofsky, 2015; Jin
& Durlofsky, 2018) which reduces the size of the system-state variables via
the POD step before approximating their evolution linearly for each time step
of the control. Although this method has played an important role in speeding
up calculations, in recent years machine learning methods have been developed
which, with a one time overhead training cost, promise an even faster
inference of the next system state.

Early on, some general machine learning models were first applied to this
problem by treating the initial field values at each point as the input to the
model and the numerical solution to the PDE as the output for time-independent
cases and the next step field values as the output for time dependent cases.
These were purely data-driven approaches which were agnostic to the form or
the physics of the underlying PDEs governing the system. Some of these early
methods, which now serve as a benchmark for which to compare more advanced
methods to, are: NN: a simple point-wise feedforward neural network. RBM: the
classical Reduced Basis Method (using POD basis). FCN: a state-of-the-art
neural network architecture based on Fully Convolution Networks (Zhu &
Zabaras, 2018). PCANN: an operator method using PCA as an auto-encoder on both
the input and output data and interpolating the latent spaces with a neural
network (Bhattacharya et al., 2021). For time dependent tasks, popular models
in other time dependent regression tasks were applied such as: ResNet: a
residual learning framework to ease the training of networks that are
substantially deeper than those used previously (He et al., 2015). U-Net: A
popular choice for image-to-image regression tasks consisting of four blocks
with 2-d convolutions and deconvolutions (Ronneberger et al., 2015).

As developments continued, one of the main directions of success to model non-
linear control dynamics came from the Embed-to-Control (E2C) model (Watter et
al., 2015). The E2C model consists of a variational auto-encoder and a
transitional block which learns to generate image trajectories from a latent
space in which the dynamics are constrained to be locally linear. This work
was subsequently modified by replacing the variational auto-encoder with an
ordinary encoder-decoder structure and then applied to the problem of 2D
reservoir surrogate modelling by (Jin et al., 2020). The results produced by
this model were extremely accurate compared to previous methods and formed the
bed rock for the expansion of the model to predict well outputs also for
resource extraction in (Coutinho et al., 2021). The thus proposed Embed to
Control and Observe (E2CO) model added an additional parameterized network
flow in the transitional block to predict well outputs such as flow rates at
source/sink points. Both the E2C and the E2CO models were naturally expanded
to 3D grids by (Atadeger et al., 2022) by remodelling the architecture to
involve 3D convolution blocks and transposes in the encoder and decoder
respectively to more reflect real-world systems in the area. The main draw
back of these current models however is that, although they produce promising
predictions for future time steps, the training times grow rapidly as grid
sizes increase. Training time for a 3D case of $5\times 10^{5}$ cells, which
is on the lower end of a real-world case, takes upwards of 47 hours on a
NVIDIA Tesla V100 GPU. Various new architectures have been proposed to tackle
this problem, such as Neural Operators (Li et al., 2020b, a, 2021, 2022;
Kovachki et al., 2023) and DeepONet (Lu et al., 2021), but in their current
state, they do not rival the accuracy of convolution-based methods.

We hence propose Localized Learning for Embed to Control (LL-E2C) and
Localized Learning for Embed to Control and Observe (LL-E2CO), a method that
learns on a random set of sub-grids of the original full grid of simulation,
using physics-informed losses, and reconstructs the next state using the
locally learned model. The intuition behind this being that physics at a local
level is constant everywhere when boundary conditions are accounted for. We
show that we achieve similar levels of accuracy to E2C and E2CO with our model
while drastically reducing the training times.

##  2 Preliminaries

Figure 1: 2D architecture for Embed-to-control applied to the reservoir
simulation problem where $x^{s}_{t}$ and $x^{p}_{t}$ represent the saturation
and pressure values respectively and $u_{t}$ the well controls.

The task that we apply our model to is that of a reservoir simulation. The
reservoir is discretized into a regular grid and different geological
structures are modelled by spatially varying but temporally static
permeabilities across the grid. At each point in the grid we have a certain
pressure and saturation value and at various points in the reservoir we have
sources/sinks in the form of injection/production wells. We can control these
wells to affect the source an sink terms in the governing equation which in
turn aids an important application of reservoir simulation, that is the
deduction of optimal control and well placement for a given subsurface which
maximises well production and storage rates.

###  2.1 Problem Setting

Let a fluid dynamical system be discretized on a regular grid with dimensions
$n=N_{x}\times N_{y}\times N_{z}$. The state of the system at time $t$ is
represented by $\textbf{x}_{t}\in\mathcal{G}$, where
$\mathcal{G}\subset\mathbb{R}^{n\times d_{x}}$. Given an applied control
$\textbf{u}_{t}\in\mathcal{U}$, where $\mathcal{U}\subset\mathbb{R}^{d_{u}}$,
we aim to model the arbitrary, smooth, system dynamics function $f$ in the
time evolution equation $\textbf{x}_{t+1}=f(\textbf{x}_{t},\textbf{u}_{t})$
and the function $g$ determining the well outputs
$\textbf{y}_{t+1}\in\mathcal{Y}$ where $\mathcal{Y}\subset\mathbb{R}^{d_{u}}$,
given by $\textbf{y}_{t+1}=\bf{g}(x_{t},u_{t})$.

###  2.2 Data

Training and testing data is generated by a proprietary high-fidelity
reservoir simulator (HFS). We consider two separate cases to evaluate our
model, a 2D ($60\times 60$) case and a larger 3D ($60\times 220\times 40$)
case. In the 2D case, each train/test sample consists of pressure
$\textbf{p}_{t}\in\mathbb{R}^{n}$, saturation
$\textbf{S}_{t}\in\mathbb{R}^{n}$ and control $\textbf{u}_{t}$ which
corresponds to the source/sink terms for $T=24$ times steps. We also have well
output values for the next time step $\textbf{y}_{t+1}$ in the case of E2CO.
The input to the model is constructed such that
$\textbf{x}_{t}=[\textbf{p}_{t},\textbf{S}_{t}]$. Permeability and source/sink
location information, which is static with time, are available where required.
The same data is simulated for the 3D case but with $T=20$. We generate 400
such samples and utilise a 3:1 split for training and testing.

##  3 Methodology

Figure 2: Three different cases of inducing boundary condition awareness in
the model. The solid line indicates the sector which is used in training and
inference and the cells inside the dotted lines indicate the part of the
sector which are ‘stitched’ together to recreate the full next state. Left.
The stitched sub-sector shares hard boundaries on the top and left with the
actual grid, teaching the model that flow cannot continue in that direction.
Middle. The stitched sub-sector shares no hard boundaries allowing the model
to learn that fluid can flow out of the sub-sector in all directions. Right.
The sub-sector shares the same hard boundaries on the left and bottom as the
grid, imposing Dirichlet boundary conditions, and Von Neumann on the right and
top.

###  3.1 E2C Reduced Order Model

Figure 3: Top. Boundary condition-aware mechanism to train and infer on the
sub-grid within the solid white line but stitch up using the sub-grid within
the dotted white line. Middle. Model architecture of the 3D LL-E2C with the
equation in red showing the additional flow for LL-E2CO model. Bottom.
Representative next state prediction using individual stitching of sectors to
reconstruct the full image.

The E2C model architecture, figure 1, consists of a parameterized encoder
$\phi_{\theta}:\mathcal{G}\rightarrow\mathcal{Z}$, where
$\mathcal{Z}\in\mathbb{R}^{n_{z}}$, transition layer
$\tau_{\theta}:\mathcal{Z}\times\mathcal{U}\rightarrow\mathcal{Z}$ and decoder
$\psi_{\theta}:\mathcal{Z}\rightarrow\mathcal{G}$. The composition of these
three gives the overall model
$\mathcal{M}_{\theta}=\\{\psi\circ\tau\circ\phi:\mathcal{G}\rightarrow\mathcal{G}\\}$.
The encoder $\phi$ consists of 4 convolution layers followed by 3 2-layer
ResNets and a final dense layer to map from the state space $\mathcal{G}$ to
the locally linear latent space $\mathcal{Z}$. The decoder $\psi$ conversely
maps back from latent space to state space using a dense layer followed by 3
2-layer transposed ResNets and 4 transposed convolution layers. The transition
layer $\tau$ in between aims to model the latent transition function
$f^{\text{lat}}$ of the dynamics given by
$\textbf{z}_{t+1}=f^{\text{lat}}(\textbf{z}_{t},\textbf{u}_{t})$. Given the
state and action sequences
$\textbf{z}_{1:T}=\\{\textbf{z}_{1},...,\textbf{z}_{T}\\}$ and
$\textbf{u}_{1:T}=\\{\textbf{u}_{1},...,\textbf{u}_{T}\\}$, optimal controls
which give rise to the trajectory $\textbf{z}_{1:T}$ for a given
$f^{\text{lat}}$ can be determined using traditional optimal control
algorithms. These algorithms approximate global non-linear dynamics with local
linear dynamics and it can be shown (section 2.2 of Watter, M. et al. (Watter
et al., 2015)) that for a reference trajectory $\bar{\textbf{z}}_{1:T}$ and
controls $\bar{\textbf{u}}_{1:T}$, the system is linearized as

$\textbf{z}_{t+1}=\textbf{A}(\bar{\textbf{z}}_{t})\textbf{z}_{t}+\textbf{B}(\bar{\textbf{z}}_{t})\textbf{u}_{t}+\textbf{o}(\bar{\textbf{z}}_{t}),$

where A($\bar{\textbf{z}}_{t}$) = $\frac{\delta
f^{\text{lat}}(\textbf{z}_{t},\textbf{u}_{t})}{\delta\bar{\textbf{z}}_{t}}$,
B($\bar{\textbf{z}}_{t}$) = $\frac{\delta
f^{\text{lat}}(\textbf{z}_{t},\textbf{u}_{t})}{\delta\bar{\textbf{u}}_{t}}$
are local Jacobians and o($\bar{\textbf{z}}_{t}$) is an offset. This equation
is the inspiration for the transition layer and the local Jacobians and
offsets are parameterized as trainable weights in the model for calculating
$\textbf{z}_{t+1}$.

The loss function of the model is a composition of traditional auto-encoder
loss functions and physics-informed loss functions as follows,

$\mathcal{L}=\mathcal{L}_{rec}+\mathcal{L}_{pred}+\theta_{trans}\times\mathcal{L}_{trans}+\theta_{flux}\times\mathcal{L}_{flux}$

$\mathcal{L}_{rec}=\|\textbf{x}_{t}-\hat{\textbf{x}}_{t}\|_{2},\quad\mathcal{L}_{pred}=\|\textbf{x}_{t}-\hat{\textbf{x}}_{t+1}\|_{2},\quad\mathcal{L}_{trans}=\|\textbf{z}_{t+1}-\hat{\textbf{z}}_{t+1}\|_{2}$

where $\hat{\textbf{x}}_{t}=\phi(\psi(\textbf{x}_{t}))$ is the reconstruction
of $\textbf{x}_{t}$,
$\hat{\textbf{x}}_{t+1}=\mathcal{M}(\textbf{x}_{t},\textbf{u}_{t})$ is the
next state prediction,
$\hat{\textbf{z}}_{t+1}=\phi(\tau(\textbf{x}_{t},\textbf{u}_{t}))$ is the next
state prediction in latent space and $\textbf{z}_{t+1}=\phi(\textbf{x}_{t+1})$
is the latent representation of true state $\textbf{x}_{t+1}$. The physics-
informed losses for the model are more specific to the reservoir simulation
task, where $\mathcal{L}_{flux}$ aims to minimize the difference in pressure
flux passing in and out of a given cell. The flux term is split up into two
components, one to calculate the flux loss for reconstruction and one for
prediction as follows,

$\mathcal{L}_{flux}=\|\textbf{k}\mathcal{F}^{rec}\|_{2}+\|\textbf{k}\mathcal{F}^{pred}\|_{2}$

$\mathcal{F}^{rec}=[k_{ro}(\textbf{S}^{w}_{t})\Delta\textbf{p}_{t}-k_{ro}(\hat{\textbf{S}}^{w}_{t})\Delta\hat{\textbf{p}}_{t}]+[k_{rw}(\textbf{S}^{w}_{t})\Delta\textbf{p}_{t}-k_{rw}(\hat{\textbf{S}}^{w}_{t})\Delta\hat{\textbf{p}}_{t}]$

$\mathcal{F}^{pred}=[k_{ro}(\textbf{S}^{w}_{t+1})\Delta\textbf{p}_{t+1}-k_{ro}(\hat{\textbf{S}}^{w}_{t+1})\Delta\hat{\textbf{p}}_{t+1}]+[k_{rw}(\textbf{S}^{w}_{t+1})\Delta\textbf{p}_{t+1}-k_{rw}(\hat{\textbf{S}}^{w}_{t+1})\Delta\hat{\textbf{p}}_{t+1}]$

where k is the permeability tensor, $k_{ro}$ and $k_{rw}$ are relative
permeability fields of the respective components given the saturation field
$\textbf{S}_{t}$ and $\Delta\textbf{p}$ represents pressure drop across
adjacent grid cells. Following conservation laws, we require the
$\mathcal{L}_{flux}$ to be minimal in order for the model to represent a
physical system.

###  3.2 E2CO Reduced Order Model

The E2CO model expands on this by modifying the transition layer to prediction
well outputs $\textbf{y}_{t+1}$. It does so by approximating global non-linear
dynamics with local linear dynamics similar to the latent representation
$\textbf{z}_{t}$. Here we parametrize two more local Jacobians
$\textbf{C}(\bar{\textbf{z}}_{t})$ and $\textbf{D}(\bar{\textbf{z}}_{t})$ and
calculate the well outputs as following

$\hat{\textbf{y}}_{t+1}=\textbf{C}(\bar{\textbf{z}}_{t})\hat{\textbf{z}}_{t+1}+\textbf{D}(\bar{\textbf{z}}_{t})\textbf{u}_{t}.$

We thus introduce an additional loss term to train for the well outputs
simultaneously given below.

$\mathcal{L}=\mathcal{L}_{rec}+\mathcal{L}_{pred}+\theta_{trans}\times\mathcal{L}_{trans}+\theta_{flux}\times\mathcal{L}_{flux}+\theta_{well}\times\mathcal{L}_{well}$

$\mathcal{L}_{well}=\|\textbf{y}_{t+1}-\hat{\textbf{y}}_{t+1}\|_{2},$

where $\hat{\textbf{y}}_{t+1}$ is the predicted well output for time $t+1$ and
$\textbf{y}_{t+1}$ is the actual well output. Notice here that we are adding
additional complexity to the model by trying to predict another quantity so we
expect to have a deterioration in accuracy when trying to predict
$\textbf{x}_{t}$. Hence, we use E2CO only when we explicitly require well
outputs and employ E2C otherwise for comparing results.

###  3.3 Localized Learning for E2C(O)

Figure 4: Saturation values predicted by LL-E2C for a single 2D test case.
Each time step is 30 days.

The E2C reduced order model, when scaled to larger grids, was found to grow
significantly in training time. For a $60\times 220\times 40$ grid, training
time exceeds 47.2 hours on an NVIDIA Tesla V100 GPU. To tackle this problem we
propose Localized Learning for Embed to Control (LL-E2C) (see figure 3), a
method where, instead of training on a full grid, we train on a random subset
of structured sub-grids (sectors).

The initial formulation is as follows. We chose at random input sectors
$\textbf{s}_{t}$ from
$\mathcal{S}=\\{\textbf{s}:\textbf{s}\in\mathbb{R}^{n_{s}}\supset\mathcal{G},\;n_{s}<n\\}$
and train the model to predict sector state $\textbf{s}_{t+1}$. At inference
time, we pass the elements of the set of all non-overlapping sectors which are
required to fully reconstruct the grid for state $\textbf{s}_{t}$,
$\,\mathcal{I}=\\{\textbf{s}^{i}_{t}:||^{n/n_{s}}_{i}\textbf{s}^{i}_{t}=\textbf{x}_{t},\;\cap^{n/n_{s}}_{i}\textbf{s}^{i}_{t}=\emptyset\\}$,
and stitch up the predicted outputs $\textbf{s}^{i}_{t+1}$ to produce the full
next state image $\textbf{x}_{t+1}$.

The key intuition behind sector training is that the underlying physics
governing the time evolution of the subsurface properties is constant. This
implies that whichever part of the grid we train on, the physics the neural
network learns to emulate will remain the same (while respecting boundary
conditions), hence sector-wise training loses no generality in solution.

In reality however, when sectors are chosen for training, we are artificially
imposing Dirichlet boundary conditions at the edges where they might otherwise
be Von Neumann. For example, when we train using a sector taken from the
middle of a grid, all the edges of the sector exhibit Von Neumann boundary
conditions but there is no way that the model would be able to capture this
without further information. Furthermore, when a sector is chosen from a
corner or side, the boundary conditions are a combination of both Dirichlet
and Von Neumann. To tackle this problem, we employ a mechanism in which we
train on sectors of size $n_{s}+n_{p}$ (where $n_{p}<n_{s}$), infer on sectors
of size $n_{s}+n_{p}$ but only use sub-sectors of size $n_{s}$ to stitch up to
the final $\textbf{x}_{t+1}$ state, as shown in figure 2. The location of the
sub-sector is chosen such that its edges line up with the edges of the full
image when the inference sector is around the periphery (to capture Dirichlet
boundary conditions) and its edges are within the edge of the inference sector
when it is taken from the middle of the image (to capture Von Neumann boundary
conditions by allowing there to be ‘flow’ out of the sub-sector). We hence
modify our formulation to take as inputs expanded sectors from
$\tilde{\mathcal{S}}=\\{\tilde{\textbf{s}}:\tilde{\textbf{s}}\in\mathbb{R}^{n_{s}+n_{p}}\\}$,
such that $\mathcal{S}\subset{\tilde{\mathcal{S}}}$ and infer using the set
$\,\tilde{\mathcal{I}}=\\{\tilde{\textbf{s}}^{i}_{t}:||^{n/n_{s}}_{i}\textbf{s}^{i}_{t}=\textbf{x}_{t},\;\cap^{n/n_{s}}_{i}\textbf{s}^{i}_{t}=\emptyset\\}$,
where the mapping between $\tilde{\textbf{s}}^{i}_{t}$ and
$\textbf{s}^{i}_{t}$ is given by the description above.

Another issue which arises from training on sectors is that, although the
underlying physics is the same in each sector, the relative locations of the
applied controls vary depending on where a sector is extracted from. To tackle
this, we add positional encoding by appending the position of the extracted
sector and the relative control locations (with the locations of controls
outside the sector zeroed out) to the control vector $\textbf{u}_{t}$. Further
to this, permeability information for each cell is passed in as separate
channels to provide flow information, aiding the greatly reduced information
available to the model.

For the case of predicting well outputs, we extend the above technique to E2CO
and propose Localized Learning for Embed to Control and Observe (LL-E2CO),
seen in figure 3. Here we make the same modification to the transition layer
as the original E2CO model but now, since we are training on sectors, we need
to take into account the fact that each sector only has information about well
controls within its boundaries and does not have access to information outside
its boundaries. As such, when passing controls $\textbf{u}_{t}$ into the
transition layer to calculate $\textbf{y}_{t+1}$, we zero out the control
values of all wells not pertaining to the sector in question. This also acts
as an additional proxy for positional encoding of the sector as the model
should, in theory, learn the position of the sector based on which well
controls are zeroed out.

##  4 Results

Table 1: Training time results in hours Method | Datasets
---|---
without Physics-informed Loss | with Physics-informed Loss
2D | 3D | 2D | 3D
E2C | 0.4 | 8.3 | 0.7 | 47.2
LL-E2C | 0.5 | 0.9 | 0.6 | 2.6
E2CO | - | 9.1 | - | 50
LL-E2CO | - | 0.9 | - | 2.8
Figure 5: The left plot shows the pressure values from the simulator for a
specific layer in the z direction, the predicted pressure values of the LL-E2C
model and the absolute error between the two. The right plot shows the same
for saturation values. Each time step is 60 days. Figure 6: Mean absolute
errors of pressure and saturation predictions for the LL-E2C model. Each dot
corresponds to an individual test case with the bars signifying 1$\sigma$
confidence intervals. These values correspond to the 3D case with each time
step of 60 days. Figure 7: Field production rates of oil and water for 3
different test cases for the LL-E2CO model. The values correspond to the 3D
case with each time step corresponding to 60 days.

Testing was conducted for the 2D and 3D cases detailed earlier and the
respective results are compared to ‘ground truth’ simulation data. In the 2D
$60\times 60$ case, empirically, we found that sector sizes of
$(20+4)\times(20+4)$ with an inference size of $20\times 20$ produced the best
results for the LL-E2C model. There was a slight improvement in training times
between the E2C and LL-E2C models but we expect to see much a larger
difference as we scale up to real world grid sizes. The saturation scalar
values for the 2D case are shown in figure 4. Looking at the errors we see
that they mainly peak around the wave fronts of the diffusion process. This
can be attributed to the fact that this is where we see the most non-linearity
in the system and hence the linearization process taking place in the
transition layer does not approximate as well.

For the 3D $60\times 220\times 40$ case, which is much closer to the real
world case, with sector sizes of $(20+4)\times(60+8)\times(20+4)$ and an
inference size of $20\times 60\times 20$, we present a significant speed up
time in training from 47.2 hours to 2.6 hours between the E2C and LL-E2C
models and 50 hours to 2.8 hours between the E2CO and LL-E2CO models
respectively as seen in table 1. In table 1, we also see the training times of
the models with and without the use of physics-informed losses. Noting that we
see a significant increase in accuracy with the inclusion of physics-informed
losses, we can conclude that the LL-E2C(O) models are much more powerful for
practical use as their training times rival even the E2C(O) models without
their physics-informed losses incorporated.

Figure 5 shows the pressure and saturation predictions for 20 time steps using
the LL-E2C model for a slice of the grid in the z direction. By analysing the
test errors, we notice that saturation predictions appear to be more accurate
than pressure predictions. This is likely due to the fact that pressure is a
more global state variable and hence Localized Learning naturally finds it
more difficult to model. A more thorough analysis of these errors were carried
out and is presented in figure 6. The model was trained and run for all the
different test cases and the mean absolute errors in pressure and saturation
values are plotted. We observe that the absolute errors in predictions are
well within the acceptable range for the purposes of reservoir simulation
surrogate modelling, with mean average absolute errors over all test cases
ranging between 50 psi to 125 psi for pressure and 0.002 to 0.007 fraction for
saturation. We also observe that, although mean average errors increase as we
predict further into the future, which is expected due to compounding errors,
the model still produces much better results than can be reasonably expected
in latter predictions. We attribute this to the fact that the system exhibits
maximal non-linearity early on in the time evolution and as the wave fronts of
pressure and saturation diffuse out, the transition layer becomes a better
approximation of the locally linear dynamics. This leads to a better
performance of the LL-E2C model.

When testing the LL-E2CO model, we are mainly concerned with well outputs. As
such, in this specific study, we look at the well production rates of oil and
water in a reservoir under different well controls. The same analysis can be
applied for the $CO_{2}$ storage task in CCS processes. The model predicts the
outputs of each individual well in the system for both the 2D and 3D cases and
in figure 7 we present the field production rates of the 3D case. Field rates
are the aggregation of all the well production rates and are a good indication
of the accuracy of the model. From the figure we see that the predicted
production rates of oil are very close to the actual values produced by the
simulator for the 3 cases. At all time steps, and even at later time steps
when prediction are expected to diverge more, the values fall within $5\%$
error. This is likely due to the same reasoning mentioned previously for
LL-E2C.

A major advantage of machine learning models over traditional numerical
solvers is their ability to parallelize inference by making batchwise
computations. We are able to pass multiple test cases simultaneously into our
model (which is what we would like to do when trying to find optimal control
solutions over a large search space) and extract individual predictions where
necessary. Thus, comparing inference times of the proposed LL-E2C(O) with the
computation time of HFS, we observe a speed up of the order $10^{4}$ which
brings to light the true strength of machine learning models in this task.

##  5 Conclusion

In this work, we motivate the application of machine learning to subsurface
fluid modelling and propose a novel method called Localized Learning to
predict future states of an advection-diffusion-reaction system, with only a
fraction of the available data. Our method reduces training times by over an
order of magnitude for typical 3D cases, while also providing comparable
accuracy results for all time steps. In the future, we hope to expand this
work to unstructured grids and meshes.

##  6 Acknowledgements

The authors would like to thank SLB management for permission to publish this
work. This work was done while Surya T. Sathujoda was a research intern at
SLB.

## References

  * TPW (2015) _Retraining Criteria for TPWL/POD Surrogate Based Waterflodding Optimization_ , volume Day 3 Wed, February 25, 2015 of _SPE Reservoir Simulation Conference_ , 02 2015.  doi: SPE-173252-MS.  D031S011R007.
  * Atadeger et al. (2022) Atadeger, A., Sheth, S., Vera, G., Banerjee, R., and Onur, M.  Deep learning-based proxy models to simulate subsurface flow of three-dimensional reservoir systems.  2022(1):1–32, 2022.  ISSN 2214-4609.  doi: https://doi.org/10.3997/2214-4609.202244049.
  * Bhattacharya et al. (2021) Bhattacharya, K., Hosseini, B., Kovachki, N. B., and Stuart, A. M.  Model reduction and neural networks for parametric pdes, 2021.
  * Cardoso & Durlofsky (2010) Cardoso, M. A. and Durlofsky, L. J.  Linearized reduced-order models for subsurface flow simulation.  _J. Comput. Phys._ , 229(3):681–700, 2 2010\.  ISSN 0021-9991.  doi: 10.1016/j.jcp.2009.10.004.
  * Coutinho et al. (2021) Coutinho, E. J. R., Dall’Aqua, M., and Gildin, E.  Physics-aware deep-learning-based proxy reservoir simulation model equipped with state and well output prediction.  _Frontiers in Applied Mathematics and Statistics_ , 7, 2021.  ISSN 2297-4687.  doi: 10.3389/fams.2021.651178.
  * He & Durlofsky (2013) He, J. and Durlofsky, L.  Reduced-order modeling for compositional simulation using trajectory piecewise linearization.  _SPE Journal_ , 19, 02 2013.  doi: 10.2118/163634-MS.
  * He & Durlofsky (2015) He, J. and Durlofsky, L. J.  Constraint reduction procedures for reduced-order subsurface flow models based on pod–tpwl.  _International Journal for Numerical Methods in Engineering_ , 103(1):1–30, 2015.  doi: https://doi.org/10.1002/nme.4874.
  * He et al. (2011) He, J., Sætrom, J., and Durlofsky, L. J.  Enhanced linearized reduced-order models for subsurface flow simulation.  _J. Comput. Phys._ , 230(23):8313–8341, 9 2011\.  ISSN 0021-9991.  doi: 10.1016/j.jcp.2011.06.007.
  * He et al. (2015) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition, 2015.
  * Jin & Durlofsky (2018) Jin, Z. L. and Durlofsky, L. J.  Reduced-order modeling of co2 storage operations.  _International Journal of Greenhouse Gas Control_ , 68:49–67, 2018.  ISSN 1750-5836.  doi: https://doi.org/10.1016/j.ijggc.2017.08.017.
  * Jin et al. (2020) Jin, Z. L., Liu, Y., and Durlofsky, L. J.  Deep-learning-based surrogate model for reservoir simulation with time-varying well controls.  _Journal of Petroleum Science and Engineering_ , 192:107273, 2020.  ISSN 0920-4105.  doi: https://doi.org/10.1016/j.petrol.2020.107273.
  * Kovachki et al. (2023) Kovachki, N., Li, Z., Liu, B., Azizzadenesheli, K., Bhattacharya, K., Stuart, A., and Anandkumar, A.  Neural operator: Learning maps between function spaces, 2023.
  * Li et al. (2020a) Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A.  Multipole graph neural operator for parametric partial differential equations, 2020a.
  * Li et al. (2020b) Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A.  Neural operator: Graph kernel network for partial differential equations, 2020b.
  * Li et al. (2021) Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A.  Fourier neural operator for parametric partial differential equations, 2021.
  * Li et al. (2022) Li, Z., Zheng, H., Kovachki, N., Jin, D., Chen, H., Liu, B., Azizzadenesheli, K., and Anandkumar, A.  Physics-informed neural operator for learning partial differential equations, 2022.
  * Lu et al. (2021) Lu, L., Jin, P., Pang, G., Zhang, Z., and Karniadakis, G. E.  Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators.  _Nature Machine Intelligence_ , 3(3):218–229, mar 2021.  doi: 10.1038/s42256-021-00302-5.
  * Ronneberger et al. (2015) Ronneberger, O., Fischer, P., and Brox, T.  U-net: Convolutional networks for biomedical image segmentation, 2015\.
  * Rousset et al. (2014) Rousset, M. A. H., Huang, C. K., Klie, H., and Durlofsky, L. J.  Reduced-order modeling for thermal recovery processes.  _Computational Geosciences_ , 18(3):401–415, 8 2014.  ISSN 1573-1499.  doi: 10.1007/s10596-013-9369-8.
  * Watter et al. (2015) Watter, M., Springenberg, J. T., Boedecker, J., and Riedmiller, M.  Embed to control: A locally linear latent dynamics model for control from raw images.  In _Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2_ , NIPS’15, pp. 2746–2754, Cambridge, MA, USA, 2015. MIT Press.
  * Zhu & Zabaras (2018) Zhu, Y. and Zabaras, N.  Bayesian deep convolutional encoder–decoder networks for surrogate modeling and uncertainty quantification.  _Journal of Computational Physics_ , 366:415–447, 2018\.  ISSN 0021-9991.  doi: https://doi.org/10.1016/j.jcp.2018.04.018.


function which is given by the following Taylor series for fixed value of
$|z|<1$,

$\displaystyle
F({\tfrac{1}{2}},{\tfrac{1}{2}};\lambda;z)=\sum_{k=0}^{\infty}{\Gamma(k+{1\over
2})^{2}\,\Gamma(\lambda)\over\Gamma({1\over
2})^{2}\,\Gamma(k+\lambda)\,k!}\,z^{k}$ (B.2)

The first few terms are given by,

$\displaystyle F({\tfrac{1}{2}},{\tfrac{1}{2}};\lambda;z)=1+{z\over
4\lambda}+{9\,z^{2}\over 32\,\lambda(\lambda+1)}+{75\,z^{3}\over
128\,\lambda(\lambda+1)(\lambda+2)}+{\cal O}(z^{4})$ (B.3)

The series is absolutely convergent for any compact subset of the open disc
$|z|<1$ uniformly in $\lambda$ greater than any fixed number strictly greater
than one; for our purposes it suffices to choose $\lambda\geq 1$. For fixed
$|z|<1$, the absolute value of each term in the series strictly decreasing as
$\lambda\to+\infty$ and therefore the limit of the series as
$\lambda\to+\infty$ is simply given by,

$\displaystyle\lim_{\lambda\to+\infty}F({\tfrac{1}{2}},{\tfrac{1}{2}};\lambda;z)=1\qquad\hbox{
for all }\quad|z|<1$ (B.4)

Next, we consider the case where $\lambda\to-\lambda$ in the hypergeometric
function, and use the general formula below to relate this case to the
previous one,666The procedure of relating the cases for positive and negative
$\lambda$ was followed in [48], but the coefficient of the first term is
incorrect there. To establish the correct relation, one easily verifies that
all three functions satisfy the hypergeometric differential equation
$z(1-z)f^{\prime\prime}+(-\lambda-(a+b+1)z)f^{\prime}-abf=0$. The coefficient
of the second term on the right side may be determined by setting $z=0$ and
using Gauss’s formula for the hypergeometric function at unit argument, while
the coefficient of the first term may be determined using the asymptotics of
the left side $z\to 1$, using the formulas in section 2.7.1 of [22].

$\displaystyle F(a,b;-\lambda;z)$ $\displaystyle=$
$\displaystyle{z^{\lambda+1}\Gamma(-\lambda)\Gamma(a+1+\lambda)\Gamma(b+1+\lambda)\over(1-z)^{\lambda+a+b}\Gamma(a)\Gamma(b)\Gamma(\lambda+2)}F(1-a,1-b;\lambda+2;z)$
(B.5)
$\displaystyle+{\Gamma(a+1+\lambda)\Gamma(b+1+\lambda)\over\Gamma(a+b+1+\lambda)\Gamma(\lambda+1)}F(a,b;a+b+1+\lambda;1-z)$

For the special case $a=b={\tfrac{1}{2}}$ of interest here, we have the
following simplification,

$\displaystyle F({\tfrac{1}{2}},{\tfrac{1}{2}};-\lambda;z)$ $\displaystyle=$
$\displaystyle-{z^{\lambda+1}\over(1-z)^{\lambda+1}}{\Gamma(-1-\lambda)\Gamma(\frac{3}{2}+\lambda)^{2}\over\Gamma({1\over
2})^{2}\Gamma(\lambda+1)}F({\tfrac{1}{2}},{\tfrac{1}{2}};\lambda+2;z)$ (B.6)
$\displaystyle+{\Gamma(\tfrac{3}{2}+\lambda)^{2}\over\Gamma(\lambda+2)\Gamma(\lambda+1)}F({\tfrac{1}{2}},{\tfrac{1}{2}};\lambda+2;1-z)$

For the first term to admit a finite limit as $\lambda\to+\infty$, we must
require $|z|<|1-z|$, in which case the contribution from the first term tends
to zero. That this sufficient condition is also necessary may be established
numerically by taking the limit $z\to{1\over 2}$ and verifying that the limit
to be established below does not hold. In the absence of the first term, the
limit of the second term is then given by,

$\displaystyle
F({\tfrac{1}{2}},{\tfrac{1}{2}};-\lambda;z)={\Gamma(\tfrac{3}{2}+\lambda)^{2}\over\Gamma(\lambda+2)\Gamma(\lambda+1)}\left(1+{1-z\over
4\lambda}+{\cal O}(\lambda^{-2})\right)$ (B.7)

and thus,

$\displaystyle\lim_{\lambda\to+\infty}F({\tfrac{1}{2}},{\tfrac{1}{2}};-\lambda;z)=1\qquad\hbox{
for all }\quad|z|<\hbox{min}(1,|1-z|)$ (B.8)

#### Convergence of the series for ${\cal Q}_{\nu}$

We shall now use the results of the preceding subsection to investigate the
convergence properties of the series given in (B.1) for ${\cal Q}_{\nu}$. The
large $\mu$ behavior of the prefactor of $\Gamma$-functions is as follows,

$\displaystyle{\Gamma(\mu+\frac{4\nu-5}{12})\Gamma(\mu+\frac{4\nu+1}{12})\over\Gamma(\mu+\frac{\nu+2}{3})\Gamma(\mu+\frac{\nu+3}{3})}={1\over\mu^{2}}\left(1+{\cal
O}(\lambda^{-1})\right)$ (B.9)

In view of the asymptotics of the hypergeometric function for $\mu\to\infty$
in the domain,

$\displaystyle|\tfrac{v}{2}|<{\tfrac{1}{2}}\hskip
72.26999pt|\tfrac{v}{2}|<|1-\tfrac{v}{2}|$ (B.10)

the large $\mu$ behavior of the summand in ${\cal Q}_{\nu}$ is as follows,

$\displaystyle{1\over\mu^{2}}\left({4u^{3}\over
27v^{2}}\right)^{\mu+\frac{\nu}{3}}$ (B.11)

and the series is convergent provided,

$\displaystyle\left|{4u^{3}\over 27}\right|<|v|^{2}=|1-u_{0}|^{2}<1$ (B.12)

since the condition $|\tfrac{v}{2}|<{\tfrac{1}{2}}$ implies the condition
$|\tfrac{v}{2}|<|1-\tfrac{v}{2}|$.

###  B.4 Elliptic expression for the $(\mathfrak{a}_{1},\mathfrak{a}_{2})$
Kähler potential

For completeness, we add here the exact expression for the intrinsic periods
in terms of the elliptic formulation developed in subsection 2.4.3. The SW
differential is given by,

$\displaystyle\lambda=-i\frac{dz}{2\sqrt{2}(2\omega)^{5}}\left(12\wp(z)^{3}-g_{2}\wp(z)\right),$
(B.13)

Using the homology basis of subsection 2.4.3, $\mathfrak{A}=[0,2\pi
i],\mathfrak{B}=[0,2\pi i\tau]$, the SW periods may be read off from Theorem
2.6 by setting $k=\ell=m=0$, and we have,

$\displaystyle a$ $\displaystyle=-i\,\frac{{\sf E}_{2}{\sf E}_{4}-{\sf
E}_{6}}{720\sqrt{2\pi}(2\omega)^{5}}$ $\displaystyle a_{D}-\tau a$
$\displaystyle=\frac{-{\sf E}_{4}}{60(2\pi)^{\frac{3}{2}}(2\omega)^{5}}$
(B.14)

The right formula confirms that $a_{D}=\rho a$ for the AD theory since ${\sf
E}_{4}(\rho)=0$. As one approaches the AD point, $u,v\to 0$, which forces
$\omega\to\infty$ since ${\sf E}_{6}(\rho)$ is non-vanishing. Thus, both
periods tend to zero at the AD point, as expected. The Kähler potential is
given by,

$\displaystyle K_{\text{AD}}=\frac{{\rm Im\,}(\tau)}{2\pi}|a|^{2}-{{\rm
Re\,}\big{(}\bar{\sf E}_{4}({\sf E}_{2}{\sf E}_{4}-{\sf E}_{6})\big{)}\over
675\,\pi^{3}|4\omega|^{10}}$ (B.15)

One verifies that upon setting $\tau=\rho$ and then letting
$\omega\rightarrow\infty$, the intrinsic Kähler potential $K_{\text{AD}}$
tends to zero as expected.

## References

  * [1] N. Seiberg and E. Witten, “Electric - magnetic duality, monopole condensation, and confinement in N=2 supersymmetric Yang-Mills theory,” Nucl. Phys. B 426, 19-52 (1994) [erratum: Nucl. Phys. B 430, 485-486 (1994)] arXiv:hep-th/9407087.
  * [2] N. Seiberg and E. Witten, “Monopoles, duality and chiral symmetry breaking in N=2 supersymmetric QCD,” Nucl. Phys. B 431, 484-550 (1994) arXiv:hep-th/9408099.
  * [3] A. Klemm, W. Lerche, S. Theisen, S. Yankielowicz, “Simple Singularities and N=2 Supersymmetric Yang-Mills Theory,” Nucl. Phys. B 431, 484-550 (1994) arXiv:hep-th/9408099.
  * [4] P. C. Argyres and A. E. Faraggi, “The vacuum structure and spectrum of N=2 supersymmetric SU(n) gauge theory,” Phys. Rev. Lett. 74, 3931-3934 (1995) arXiv:hep-th/9411057.
  * [5] A. Hanany and Y. Oz, “On the quantum moduli space of vacua of N=2 supersymmetric SU(N(c)) gauge theories,” Nucl. Phys. B 452, 283-312 (1995) arXiv:hep-th/9505075.
  * [6] P. C. Argyres, M. R. Plesser and A. D. Shapere, “The Coulomb phase of N=2 supersymmetric QCD,” Phys. Rev. Lett. 75, 1699-1702 (1995) arXiv:hep-th/9505100.
  * [7] A. Klemm, W. Lerche and S. Theisen, “Nonperturbative effective actions of N=2 supersymmetric gauge theories,” Int. J. Mod. Phys. A 11, 1929-1974 (1996) arXiv:hep-th/9505150.
  * [8] K. A. Intriligator and N. Seiberg, “Lectures on supersymmetric gauge theories and electric-magnetic duality,” Nucl. Phys. B Proc. Suppl. 45BC, 1-28 (1996) arXiv:hep-th/9509066.
  * [9] E. D’Hoker, I. M. Krichever and D. H. Phong, “The Effective prepotential of N=2 supersymmetric SU(N(c)) gauge theories,” Nucl. Phys. B 489, 179-210 (1997) arXiv:hep-th/9609041.
  * [10] Y. Tachikawa, “N=2 supersymmetric dynamics for pedestrians,” arXiv:1312.2684.
  * [11] M. Martone, “The constraining power of Coulomb Branch Geometry: lectures on Seiberg-Witten theory,” arXiv:2006.14038.
  * [12] M. R. Douglas and S. H. Shenker, “Dynamics of SU(N) supersymmetric gauge theory,” Nucl. Phys. B 447, 271-296 (1995) arXiv:hep-th/9503163.
  * [13] E. D’Hoker and D. H. Phong, “Strong coupling expansions of SU(N) Seiberg-Witten theory,” Phys. Lett. B 397, 94-103 (1997) arXiv:hep-th/9701055.
  * [14] E. D’Hoker, T. T. Dumitrescu, E. Gerchkovitz and E. Nardoni, “Revisiting the multi-monopole point of SU(N) $\mathcal{N}$ = 2 gauge theory in four dimensions,” JHEP 09, 003 (2021) arXiv:2012.11843.
  * [15] E. D’Hoker, T. T. Dumitrescu and E. Nardoni, “Exploring the strong-coupling region of SU(N) Seiberg-Witten theory,” JHEP 11, 102 (2022) arXiv:2208.11502.
  * [16] P. C. Argyres and M. R. Douglas, “New phenomena in SU(3) supersymmetric gauge theory,” Nucl. Phys. B 448, 93-126 (1995) arXiv:hep-th/9505062.
  * [17] F. Ferrari and A. Bilal, “The Strong coupling spectrum of the Seiberg-Witten theory,” Nucl. Phys. B 469, 387-402 (1996) arXiv:hep-th/9602082.
  * [18] A. Bilal and F. Ferrari, “Curves of marginal stability, and weak and strong coupling BPS spectra in N=2 supersymmetric QCD,” Nucl. Phys. B 480, 589-622 (1996) arXiv:hep-th/9605101.
  * [19] A. Bilal and F. Ferrari, “The BPS spectra and superconformal points in massive N=2 supersymmetric QCD,” Nucl. Phys. B 516, 175-228 (1998) arXiv:hep-th/9706145.
  * [20] C. Córdova and T. T. Dumitrescu, “Candidate Phases for SU(2) Adjoint QCD4 with Two Flavors from $\mathcal{N}=2$ Supersymmetric Yang-Mills Theory,” arXiv:1806.09592.
  * [21] E. D’Hoker, T. T. Dumitrescu, E. Gerchkovitz and E. Nardoni, to appear.
  * [22] A. Erdelyi editor, Bateman Manuscript project, Higher transcendental Functions, Vol I, Chapter V, Robert E. Krieger Publishing Company, 1981.
  * [23] D. Gaiotto, “N=2 dualities,” JHEP 08, 034 (2012) arXiv:0904.2715.
  * [24] P. Argyres, M. Lotito, Y. Lü and M. Martone, “Geometric constraints on the space of $\mathcal{N}$ = 2 SCFTs. Part I: physical constraints on relevant deformations,” JHEP 02, 001 (2018) arXiv:1505.04814.
  * [25] P. C. Argyres, M. Lotito, Y. Lü and M. Martone, “Geometric constraints on the space of $\mathcal{N}$ = 2 SCFTs. Part II: construction of special Kähler geometries and RG flows,” JHEP 02, 002 (2018) arXiv:1601.00011.
  * [26] P. C. Argyres, M. Lotito, Y. Lü and M. Martone, “Expanding the landscape of $\mathcal{N}$ = 2 rank 1 SCFTs,” JHEP 05, 088 (2016) arXiv:1602.02764.
  * [27] P. Argyres, M. Lotito, Y. Lü and M. Martone, “Geometric constraints on the space of $\mathcal{N}$ = 2 SCFTs. Part III: enhanced Coulomb branches and central charges,” JHEP 02, 003 (2018) arXiv:1609.04404.
  * [28] P. C. Argyres and M. Martone, “Scaling dimensions of Coulomb branch operators of 4d N=2 superconformal field theories,” arXiv:1801.06554.
  * [29] J. Kaidi, M. Martone, L. Rastelli and M. Weaver, “Needles in a haystack. An algorithmic approach to the classification of 4d $\mathcal{N}$ = 2 SCFTs,” JHEP 03, 210 (2022) arXiv:2202.06959.
  * [30] E. D’Hoker and J. Kaidi, “Lectures on modular forms and strings,” arXiv:2208.07242.
  * [31] M. Alim, S. Cecotti, C. Cordova, S. Espahbodi, A. Rastogi and C. Vafa, “$\mathcal{N}=2$ quantum field theories and their BPS quivers,” Adv. Theor. Math. Phys. 18, no.1, 27-127 (2014) arXiv:1112.3984.
  * [32] D. Gaiotto, G. W. Moore and A. Neitzke, “Wall-crossing, Hitchin systems, and the WKB approximation,” Adv. Math. 234, 239-403 (2013) arXiv:0907.3987.
  * [33] M. Alim, F. Beck, A. Biggs and D. Bryan, “Special geometry, quasi-modularity and attractor flow for BPS structures,” [arXiv:2308.16854 [hep-th]].
  * [34] S. Cecotti, A. Neitzke and C. Vafa, “R-Twisting and 4d/2d Correspondences,” arXiv:1006.3435.
  * [35] S. Cecotti and C. Vafa, “Classification of complete N=2 supersymmetric theories in 4 dimensions,” arXiv:1103.5832.
  * [36] T. Eguchi and K. Hori, “N=2 superconformal field theories in four-dimensions and A-D-E classification,” arXiv:hep-th/9607125.
  * [37] D. J. Binder, S. M. Chester, S. S. Pufu and Y. Wang, “$\mathcal{N}$ = 4 Super-Yang-Mills correlators at strong coupling from string theory and localization,” JHEP 12, 119 (2019) arXiv:1902.06263.
  * [38] S. M. Chester, M. B. Green, S. S. Pufu, Y. Wang and C. Wen, “Modular invariance in superstring theory from $\mathcal{N}$ = 4 super-Yang-Mills,” JHEP 11, 016 (2020) arXiv:1912.13365.
  * [39] S. M. Chester, M. B. Green, S. S. Pufu, Y. Wang and C. Wen, “New modular invariants in $\mathcal{N}$ = 4 Super-Yang-Mills theory,” JHEP 04, 212 (2021) arXiv:2008.02713.
  * [40] I. Bah, F. Bonetti, R. Minasian and E. Nardoni, “Holographic Duals of Argyres-Douglas Theories,” Phys. Rev. Lett. 127, no.21, 211601 (2021) arXiv:2105.11567.
  * [41] I. Bah, F. Bonetti, R. Minasian and E. Nardoni, “M5-brane sources, holography, and Argyres-Douglas theories,” JHEP 11, 140 (2021) arXiv:2106.01322.
  * [42] A. Bissi, F. Fucito, A. Manenti, J. F. Morales and R. Savelli, “OPE coefficients in Argyres-Douglas theories,” JHEP 06, 085 (2022) arXiv:2112.11899.
  * [43] D. Xie, “Soft supersymmetry breaking of 4d $\mathcal{N}=2$ SCFT,” arXiv:1905.00345.
  * [44] K. Maruyoshi and J. Song, “$\mathcal{N}=1$ deformations and RG flows of $\mathcal{N}=2$ SCFTs,” JHEP 02, 075 (2017) arXiv:1607.04281.
  * [45] K. Maruyoshi and J. Song, “Enhancement of Supersymmetry via Renormalization Group Flow and the Superconformal Index,” Phys. Rev. Lett. 118, no.15, 151602 (2017) arXiv:1606.05632.
  * [46] P. Appell and J. Kampé de Fériet, Fonctions Hyper-géometriques and Hyper-sphériques - Polynomes d’Hermite, Gauthier-Villars, Paris (1929);
  * [47] B. Ananthanarayan, S. Friot, S. Ghosh and A. Hurier, “New analytic continuations for the Appell $F_{4}$ series from quadratic transformations of the Gauss ${}_{2}F_{1}$ function,” arXiv:2005.07170.
  * [48] N.M. Temme, “Large parameter cases of the Gauss hypergeometric function”, Journal of Computational and Applied Mathematics 153 (2003) 441-462.