# Octree-based hierarchical sampling optimization for the volumetric super-
resolution of scientific data

Xinjie Wang  Maoquan Sun  Yundong Guo<EMAIL_ADDRESS>Chunxin Yuan  Xiang
Sun  Zhiqiang Wei  Xiaogang Jin  College of Computer Science and Technology,
Ocean University of China, Qingdao, 266100, China  College of Computer Science
and Engineering, Shandong University of Science and Technology, Qingdao,
266590, China  School of Mathematical Sciences, Ocean University of China,
Qingdao, 266100, China  State Key Lab of CAD&CG, Zhejiang University,
Hangzhou, 310058, China

###### Abstract

When introducing physics-constrained deep learning solutions to the volumetric
super-resolution of scientific data, the training is challenging to converge
and always time-consuming. We propose a new hierarchical sampling method based
on octree to solve these difficulties. In our approach, scientific data is
preprocessed before training, and a hierarchical octree-based data structure
is built to guide sampling on the latent context grid. Each leaf node in the
octree corresponds to an indivisible subblock of the volumetric data. The
dimensions of the subblocks are different, making the number of sample points
in each randomly cropped training data block to be adaptive. We reconstruct
the octree at intervals according to loss distribution to perform the multi-
stage training. With the Rayleigh-Bénard convection problem, we deploy our
method to state-of-the-art models. We constructed adequate experiments to
evaluate the training performance and model accuracy of our method.
Experiments indicate that our sampling optimization improves the convergence
performance of physics-constrained deep learning super-resolution solutions.
Furthermore, the sample points and training time are significantly reduced
with no drop in model accuracy. We also test our method in training tasks of
other deep neural networks, and the results show our sampling optimization has
extensive effectiveness and applicability. The code is publicly available at
https://github.com/xinjiewang/octree-based_sampling.

###### keywords:

scientific data, volumetric super-resolution, octree, importance sampling,
physics-constrained deep learning

††journal: Journal of Computational Physics

##  1 Introduction

To study complex physical phenomena (e.g., turbulence flows[1, 2], ocean
currents [3, 4], climate systems[5, 6, 7]), scientists have to run numerical
simulations on supercomputers without interruption to continuously generate
high-resolution scientific data. These produced volumetric data with spatio-
temporal dimensions are vital for scientists to discover and verify physical
laws. It is worth noting that as the resolution of numerical simulations
increases, the scale of revealed physical processes decreases. Nevertheless,
this entails a greater demand for computational resources, and the simulation
time becomes intolerable. It severely constrains the study of small-scale
physical phenomena such as oceanic submesoscale processes.

To efficiently acquire high-resolution scientific data within a limited time,
it is recommended to develop a physics-constrained deep neural network that
can conduct super-resolution reconstruction [8, 9]. Since it can
simultaneously learn statistical and physical correlations between pairs of
low-resolution and high-resolution solutions, this deep learning technique has
been successfully applied in the volumetric super-resolution of flow
simulations [10, 11]. However, during the training process, deep learning
models encounter difficulty in converging within a reasonable time due to the
complexity of partial differential equations (PDEs) involved. Therefore, how
to enhance the training performance of physics-constrained super-resolution
models still remains a challenge.

When training a super-resolution model incorporating physical constraints,
dense sampling on the latent context grid is a prerequisite. The spatio-
temporal coordinate of each sample point along with its implicit feature is
fed into the decoder to produce the corresponding value of physical variables
at the respective coordinate. As a result, the decoder performs much more
times than the encoder, and the computational burden of training is greatly
impacted by the number of sample points. In physics-informed neural networks
(PINNs), implementing importance sampling, rather than random sampling, proves
to be a viable approach [12]. To this end, in this paper, we focus on
optimizing the distribution of the sample points to improve the training
performance of physics-constrained super-resolution models.

Some previous works have demonstrated that importance sampling is an efficient
method for accelerating the training of deep neural networks [13, 14, 15, 16]
and PINNs [17, 18, 12]. However, these methods focus on evaluating the
importance weight of each training sample or sample point, ignoring the
dynamic overall importance distribution of the whole data. Besides, there is
currently no importance sampling procedure specifically designed and optimized
for the volumetric data.

Inspired by the high efficiency of octree [19, 20, 21], we propose an octree-
based hierarchical sampling optimization method that is elaborately designed
for the training of physics-constrained super-resolution models. By building
an octree model of the training data, we are able to guide the sampling
process based on its importance distribution. This approach leads to a notable
reduction in the number of sample points and accelerates the convergence of
physics-constrained deep learning models.

Specifically, our main technical contributions in this paper are as follows:

  * 1.

We propose a novel octree-based approach for modeling volumetric scientific
data by leveraging its importance distribution, wherein each leaf node
corresponds to an indivisible subblock in the volumetric domain.

  * 2.

We present a hierarchical sampling optimization method, which performs
importance sampling based on the octree model to improve the distribution of
sample points in each training data block.

  * 3.

We construct a multi-stage training strategy to avoid failure of the proposed
sampling optimization, which involves updating the octree model whenever the
training encounters a bottleneck.

We apply the proposed octree-based hierarchical sampling optimization in two
physics-constrained super-resolution solutions and perform extensive
experiments on Rayleigh-Bénard convection problem. The results show that our
approach can significantly improve the training performance of deep learning
super-resolution models with physical constraints. Specifically, the octree-
based sampling optimization method can help the model achieve the same
accuracy with fewer sample points and a shorter time.

The rest parts of this paper are organized as follows. section 2 briefly
reviews the related work, including volumetric super-resolution of scientific
data and importance sampling methods for deep learning. section 3 describes
the proposed method, covering five technical details comprehensively. section
4 demonstrates the detailed experimental results and section 5 gives some
limitations and discussions. Finally, conclusions are drawn, and the
associated future work is given in section 6.

##  2 Related Work

We focus on developing a sampling optimization method for the deep learning-
based volumetric super-resolution. Therefore, related work includes two
aspects: the volumetric super-resolution of scientific data and the importance
sampling for deep learning.

###  2.1 Volumetric Super-resolution of Scientific Data

The volumetric super-resolution of scientific data aims to reconstruct the
fine-scale subgrid solutions from the coarse-scale ones. It is usually
implemented based on interpolation algorithms such as trilinear interpolation.
In recent years, due to its impressive ability to tackle multi-dimensional and
nonlinear issues, the deep learning technique has been introduced to
volumetric super-resolution tasks and is widely used.

Xie et al. [22] proposed a conditional generative adversarial network (GAN)
for the volumetric data of fluid flows. It is able to generate consistent and
detailed super-resolution results by using a new temporal discriminator.
Werhahn et al. [23] obtained full coverage for the volumetric domain by two
separate GANs and can leverage spatio-temporal supervision with a set of
discriminators. in addition to GAN, Fukami et al. [24] employed other deep
learning techniques such as convolutional neural networks and multi-scale
models with skip-connections to build and super-resolve 3D flow fields.
Besides, in the field of 3D visualization, SSR-VFD [25] was presented to
produce coherent spatial super-resolution of 3D vector field data, and TSR-TVD
[26] was presented to generate temporal super-resolution of time-varying data
using the recurrent generative network. Weiss et al. [27] also proposed a
fully convolutional neural network to learn an upscaling model for rendering
accurate isosurfaces in a volumetric field.

These methods can offer statistical features, but they do not consider the
physical laws that scientific data must adhere to. The lack of
interpretability is a common problem of deep learning, which leads to
unreliable super-resolution results of scientific data. As a solution to this
problem, researchers introduced PINNs [28, 29, 30, 31, 32, 33, 34, 35]. PINNs
are neural networks that learn distribution patterns of training data to
approximate the physical laws. It has been applied in a range of fields, such
as materials [36, 37], mechanics[38, 39], fluids[40, 41, 42] and
bioengineering [43, 44]. PINNs incorporate the physical laws governing
scientific data in the deep learning framework, resulting in improved accuracy
and credibility.

As for the volumetric super-resolution of scientific data based on PINNs,
MeshfreeFlowNet [8] builds a deep continuous spatio-temporal framework and
takes physical constraints into account to achieve grid-free fluid super-
resolution. TransFlowNet [10] uses Transformer to remodel the encoder for more
effective deep features and gives better super-resolution results. These
approaches show that it is feasible and effective to consider physical
constraints in a deep learning-based volumetric super-resolution framework.
However, they all require high computational power, take a long training time,
and prove difficult to converge. Furthermore, the massive sample points
required in the training could generate memory-related problems. Other
measures must be taken, such as reducing the batch size or the number of
sample points, which may cause suboptimal training outcomes.

###  2.2 Importance Sampling for Deep Learning

Deep learning necessitates substantial data for training, leading to a
significant increase in computing expenses. Nevertheless, it turns out that
not all data or samples are equally important, and a considerable portion can
be disregarded during training without affecting the final model. In recent
years, researchers have begun to explore using importance sampling to
accelerate the training of deep neural networks.

Alain et al. [45] presented a distributing neural network training method that
uses a cluster of GPU workers to search for the most informative samples to
train on. Chen et al. [14] used importance sampling to reformulate the loss
and the gradient for the fast learning of graph convolutional networks.
Katharopoulos and Fleuret [46] provided a practical upper bound to the
gradient norm of any neural network and showed that not all samples are equal
in the duration of training. By approximating the ideal sampling distribution
using robust optimization, Johnson and Guestrin [15] proposed a practical
importance sampling scheme for speeding up the training of deep learning
models. Banerjee et al. [16] presented a selection algorithm to generate a
deterministic sequence of mini-batches instead of a random one to train the
deep neural networks.

These methods mainly focus on the training acceleration of deep neural
networks via selecting training samples. In contrast, PINNs pay more attention
to the distribution of sample points in the spatio-temporal dimension. There
has also been some impressive work on importance sampling during PINNs
training. Nabian et al. [17] sampled the collocation points based on a
distribution proportional to the loss function, leading to an improvement in
convergence performance when training PINNs. Lu et al. [18] proposed a
residual-based adaptive refinement approach to improve the training efficiency
of PINNs. This method repeatedly adds sample points in locations where the
PDEs residual is large until the mean residual is smaller than a threshold.
Yang et al. [12] proposed dynamic mesh-based importance sampling to achieve
stable convergence by estimating sample weights efficiently.

PINNs are designed for obtaining an approximate solution of the differential
equation by training a neural network so that the fitting error is as small as
possible. These importance sampling methods for PINNs are committed to
evaluating the weight of each sample point, ignoring the dynamic overall
importance distribution. In the physics-constrained volumetric super-
resolution, the training data are cubes randomly cropped from the original
data. The calculation of PDEs loss relies on the sampling within each cube.
The distribution of sample points guides the gradient descent. Therefore, it
is necessary to dynamically estimate the importance distribution of the
original data during training.

For the deep learning-based volumetric super-resolution of scientific data, we
propose an octree-based hierarchical sampling optimization to achieve
importance sampling. It significantly improves the training performance by
reducing the number of sample points.

##  3 Method

In this section, we propose a new hierarchical sampling method based on octree
to perform multi-stage training. It focuses on training optimization and is
mainly oriented to the volumetric super-resolution task based on physics-
informed deep learning. Specifically, we use an octree-based data structure to
achieve importance sampling on the latent context grid, which reduces the
computation spent on the physics-constrained network. As a result, our method
can greatly reduce the number of sample points while maintaining good
convergence performance. Compared with other sampling methods applied to
PINNs, our approach is elaborately designed for volumetric data and shows
better stability.

###  3.1 Overview

The overview of our octree-based hierarchical sampling optimization in the
training pipeline is shown in Figure 1. Before training, we take advantage of
the variance information in the original data to generate an initial octree
model. The octree subdivides the volumetric domain of data into several cuboid
subblocks with different sizes. Each leaf node corresponds to an indivisible
subblock. Then, the first stage of training begins. The volumetric super-
resolution framework randomly crops cubic data blocks from the original data
as ground truth and feeds them one by one into the encoder after downsampling.
Based on the octree model, we perform hierarchical importance sampling on the
latent context grid produced by the encoder. The number of sample points is
adaptive to the training data block position in the original data. We compute
the corresponding prediction at each sampling point through the decoder and
estimate the average regression loss and PDEs loss for this training data
block. After updating weights based on the total loss, we repeat the previous
training process several times. When the first stage of training reaches a
bottleneck, i.e., the loss is almost no longer decreasing, we reconstruct the
octree based on the current best model. The updated octree ensures that more
sample points are collected in areas with larger regression loss. We update
the octree multiple times and after each update, a stage of training follows.
Finally, the training ends when one of the termination conditions is reached.

We explain five technical details of the proposed method in the following
subsections, including Volumetric Super-resolution, Octree Initialization,
Octree Reconstruction, Hierarchical Sampling Optimization, and Multi-stage
Training Strategy.

Figure 1: The proposed octree-based hierarchical sampling optimization in the
training pipeline. The octree model is initialized or reconstructed before
each stage of training to achieve importance sampling by hierarchical sampling
optimization. We number and highlight the five technical details corresponding
to the subsections.

###  3.2 Volumetric Super-resolution

Volumetric super-resolution refers to the generation of high-resolution
spatio-temporal scientific data from low-resolution ones. Different from the
interpolation methods, deep learning solutions use an encoder to extract
implicit features of low-resolution data and a physics-constrained decoder to
generate high-resolution data. The deep learning models work well but are
difficult to train. Since the original data has a large dimensional size,
several data blocks are randomly cropped to construct the training dataset.

We define the original data as $\mathbf{D}_{gt}\in\mathbb{R}^{h\times w\times
t\times c}$, where $c$ is the number of physical variables. The cropped
training data block is denoted as $\mathbf{D}_{gt}^{b}$. During training,
$\mathbf{D}_{gt}^{b}$ is first downsampled to low-resolution
$\mathbf{D}_{lr}^{b}$ and fed into the encoder to produce the latent context
grid. Then, a large number of coordinates are randomly sampled in the latent
context grid. Along with their corresponding features, these sample points are
decoded to the estimated values of the original physical variables, which are
used to compute the total loss. In one forward propagation, the encoder
performs only once, while the decoder performs many times. Specifically, the
decoder performs as many times as the number of sample points. Therefore, the
number and distribution of sample points have a decisive impact on training
efficiency. To this end, we improve the training performance of volumetric
super-resolution solutions by octree-based hierarchical sampling optimization.

###  3.3 Octree Initialization

We use the original data to initialize the octree as preprocessing before
training. The initialized octree will guide the importance sampling process on
the latent context grid during the first stage of training. Specifically, we
generate the first octree based on the variance of the original data to make
the first stage of training pay more attention to areas with more significant
numerical fluctuations.

First, we compute the mean and variance of each physical variable by

$\displaystyle E_{i}=\frac{1}{n}\sum_{j=1}^{n}x_{ij}$ (1) $\displaystyle
V_{i}=\frac{1}{n}\sum_{j=1}^{n}\left(x_{ij}-E_{i}\right)^{2},$ (2)

where $E_{i}$ and $V_{i}$ denote the mean and variance of the $i$-th physical
variable, respectively. $n$ is the number of grid points in the volumetric
data and equals $h\times w\times t$. $x_{ij}$ is the normalized value of the
$i$-th physical variable at the $j$-th grid point. The volumetric super-
resolution task for scientific data may require different weights for
different physical variables. We define the total variance $V_{tot}$ of the
volumetric data as

$V_{tot}=\frac{1}{c}\sum_{i=1}^{c}\alpha_{i}V_{i},$ (3)

where $c$ is the number of physical variables, and $\alpha_{i}$ denotes the
custom weight of the $i$-th physical variable. The sum of $\alpha_{i}$ equals
$c$. Obviously, $V_{tot}$ is a weighted sum of all $V_{i}$.

Then, we start to generate the first octree. Obviously, The first node of the
octree is the root node that denotes the entire volumetric domain of data. As
each dimension size of the volumetric data may be different, the volumetric
domain is a cuboid. Next, we recursively produce leaf nodes from the root
node. We first divide the cuboid into 8 cuboid subblocks of the same size,
corresponding to producing 8 leaf nodes from the root node. After the first
subdivision, we compute the variance $V_{tot}^{sub}$ for each subblock, in
turn, using the same way as Equation 3. If a subblock meets $V_{tot}^{sub}\geq
V_{tot}$ and one of its dimensions is greater than the custom size threshold
$s$, we continue to subdivide this subblock. Specifically, we subdivide the
subblock in three directions if all three dimensions are greater than $s$, in
two directions if only two dimensions are greater, and so on. Correspondingly,
each leaf node may produce 8, 4, or 2 leaf nodes. We perform the subdivision
recursively until all the cuboid subblocks, i.e., the leaf nodes, do not meet
the subdivision conditions.

At last, we complete the octree initialization, and an octree model example is
shown in Figure 2. The root node of octree denotes the entire volumetric data.
Each leaf node corresponds to an indivisible subblock. The detailed octree
subdivision algorithm is illustrated in algorithm 1.

(a)

(b)

Figure 2: An octree model example. The root node of octree denotes the entire
volumetric data, and each leaf node corresponds to an indivisible subblock. We
mark leaf nodes in the same depth and their corresponding subblocks with
identical colors.

Input: $\mathbf{D}_{gt}$, root, $V_{tot}$

Function  _Subdivide(_$\mathbf{D}_{gt}$ , root, $V_{tot}$_)_:

Function  _Count(_root_)_:

$n\longleftarrow 0$

if  _root’s height $\geq s_{h}$_ then $n\longleftarrow n+1$;

if  _root’s width $\geq s_{w}$_ then $n\longleftarrow n+1$;

if  _root’s length $\geq s_{l}$_ then $n\longleftarrow n+1$;

return $2^{n}$

End Function

Create Count(_root_) nodes and add them to the root’s leaves

foreach  _leaf $\in$ root’s leaves_ do

Calculate $V_{tot}^{sub}$ for each leaf

if  _$V_{tot}^{sub}\geq V_{tot}$_ then

Subdivide(_$\mathbf{D}_{gt}$ , leaf, $V_{tot}$_)

End Function

Algorithm 1 Octree Subdivision Algorithm

###  3.4 Octree Reconstruction

In the first stage of training, as the weights in the deep learning network
are updated iteratively, the variance-based initial octree gradually fails to
represent the distribution characteristics of training data under the current
model. This situation leads to importance sampling gradually playing an
irrelevant role in training, and the descending gradient of the total loss
will converge to 0. Therefore, we propose a loss-based octree reconstruction
scheme. The reconstructed octree represents the regression loss distribution
on training data, and further plays the significant role of importance
sampling to ensure the convergence and effectiveness of subsequent stages of
training.

Since the inference is time-consuming using the original data
$\mathbf{D}_{gt}$, we downsample it using a scale factor $a$ to obtain the
Ground Truth data $\mathbf{D}_{gt}^{a}$ for octree reconstruction. This pre-
processing will reduce the inference time to $\frac{1}{a^{3}}$. It is a
compromise between the accuracy of the importance distribution and the speed
of octree reconstruction.

First, we perform inference using $\mathbf{D}_{gt}^{a}$ and the current best
model. The high-resolution result is

$\mathbf{D}_{hr}^{k}=\mathcal{N}_{best}^{k}\left(\mathbf{D}_{lr}^{a}\right),$
(4)

where $\mathbf{D}_{lr}^{a}$ is the low-resolution input after downsampling
$\mathbf{D}_{gt}^{a}$. $\mathcal{N}_{best}^{k}(\cdot)$ is the best model
obtained after the $k$-th stage of training and $\mathbf{D}_{hr}^{k}$ is the
high-resolution output after inference.

Then, we use $\mathbf{D}_{hr}^{k}$ and $\mathbf{D}_{gt}^{a}$ to compute the
regression loss distribution. The regression loss at the $j$-th grid point is
defined as

$L_{j}^{k}=\frac{1}{c}\sum_{i=1}^{c}\alpha_{i}\|x_{ij}-\hat{x}_{ij}^{k}\|_{1},$
(5)

where $\hat{x}_{ij}^{k}$ is the estimated value of the $i$-th physical
variable at the $j$-th grid point, which can be obtained by indexing in
$\mathbf{D}_{hr}^{k}$. The total regression loss is

$L_{tot}^{k}=\frac{1}{n}\sum_{j=1}^{n}L_{j}^{k}.$ (6)

Finally, similar to the initial octree, we recursively reconstruct the octree
from the root node based on the regression loss. We denote the regression loss
of a subblock by $L_{tot}^{sub,k}$, which can be obtained in the same way as
Equation 6. The subdivision conditions are $L_{tot}^{sub,k}\geq L_{tot}^{k}$
and there is a dimension greater than $s$ in the subblock.

###  3.5 Hierarchical Sampling Optimization

As described in subsection 3.2, after randomly cropping a cubic data block
from the original data as the ground truth for a single training cycle, the
encoder of the volumetric super-resolution framework will extract features
from the downsampled low-resolution data block and build a latent feature
grid. Then, most methods randomly or uniformly take sample points from the
latent context grid and feed them to the decoder to estimate the values of
physical variables at the corresponding positions. Since the training
performance is significantly affected by the sample points, we present an
octree-based hierarchical sampling method to optimize the distribution of
sample points in each training data block.

As shown in Figure 2, each leaf node in the octree corresponds to an
indivisible cuboid subblock in the original data. We define the number of
sample points in each subblock with the same value $m$. Each leaf node may
have a different depth, so each subblock may have a different size. This setup
leads to different sample point densities in different subblocks, which is the
key to achieving importance sampling.

In a randomly cropped training data block, the implicit feature of a sample
point can be indexed in the latent context grid by its position, so the
sampling process is to determine the position distribution of sample points.
The schematic diagram of octree-based hierarchical sampling optimization is
shown in Figure 3.

Figure 3: The schematic diagram showcases the octree-based hierarchical
sampling optimization. The red cube indicates the position of the training
data block in the volumetric domain (left), and the red leaf nodes indicate
the subblocks intersecting with the training data block (right).

First, based on the size and position of the training data block, we retrieve
the leaf nodes that intersect with it and get several intersection spaces.
These intersection spaces correspond to the red leaf nodes in the right part
of Figure 3.

Then, we determine the number of sample points in each intersection space by

$M=\lceil m\frac{S_{int}}{S_{leaf}}\rceil,$ (7)

where $S_{int}$ and $S_{leaf}$ are the volumes of the intersection space and
its corresponding leaf node, respectively. When generating the octree, we can
easily obtain $S_{leaf}$ based on the size of each indivisible cuboid
subblock. Together with its position and the size and position of the training
data block, we can also compute $S_{int}$.

Finally, we perform random sampling in each intersection space in turn, and
the number of sample points is computed by Equation 7. Since the leaf nodes
corresponding to the intersection spaces may have different depths, the
sampling process on the training data block is hierarchical. As a result, we
achieve the octree-based importance sampling because of the different sampling
densities within each intersection space.

###  3.6 Multi-stage Training Strategy

As described in subsection 3.4, the octree-based importance sampling will
gradually fail as the training proceeds and the weights update. Therefore, we
perform the training process with several stages, called the multi-stage
training strategy. Whenever reaching a bottleneck, we terminate the current
stage of training and reconstruct the octree to start the next stage of
training. At each stage of training, we perform hierarchical sampling
optimization using the updated octree.

The key to multi-stage training is when to terminate the current stage and
start the next one. In order to avoid frequently starting a new stage and
performing the time-consuming octree reconstruction, we define $q$ as the
minimum number of epochs for each stage of training. As we all know, if the
loss is no longer decreasing and has not converged to the expected value, we
call the training reaching a bottleneck. Therefore, starting from the $q$-th
epoch ($q\geq 10$), we use loss values of the last 10 epochs to do linear
regression. If the mean value is less than or equal to threshold $Q$, we
terminate the training immediately and save the current best model as the
training result. If the mean value is higher than threshold $Q$ and the slope
of the linear regression model is greater than or equal to 0, we start the
next stage of training.

##  4 Experiments

There are two recently published state-of-the-art solutions of spatio-temporal
super-resolution for scientific data, called MeshfreeFlowNet [8] and
TransflowNet [10], respectively. We apply the proposed octree-based
hierarchical sampling optimization to these two deep-learning methods and
verify its effectiveness with extensive experiments on the Rayleigh-Bénard
convection problem. We first evaluate the performance of our sampling
optimization method in terms of both convergence speed and model accuracy.
Then we perform an ablation study to elucidate the role of the loss-based
octree reconstruction scheme and discuss the optimal configuration of custom
parameters. At last, we experiment with our method in the application of
solving PDEs using PINNs and compare it with other importance sampling methods
like DeepXDE [18]. The results show that the octree-based hierarchical
sampling optimization is an effective technology for importance sampling of
volumetric data, and it can be generalized and applied in PINNs-based
solutions. In addition, the experiments are implemented on a high-performance
computer with one NVIDIA GeForce RTX 3090.

###  4.1 Baseline

MeshfreeFlowNet is a deep continuous spatio-temporal super-resolution
framework with PDEs loss, and TransFlowNet is a physics-constrained volumetric
super-resolution solution with Transformer blocks. They focus on solving the
high-resolution reconstruction problem of 3D flow simulations using deep
learning technology. In the training phase of both frameworks, multiple dense
sampling of the latent context grid is required. The distribution of sample
points significantly impacts the training performance and results. We use them
as baselines and apply our octree-based hierarchical sampling optimization in
their training phase.

The specific training settings of baselines are detailed as follows. We train
the original MeshfreeFlowNet and TransFlowNet using an Adam optimizer with a
learning rate 0.001 for 100 epochs. In each epoch, we take 3000 randomly
cropped data blocks as training samples. Limited by the memory size, the batch
size is set to 10. The weighting coefficient of PDEs loss is 0.02. Besides, we
set the number of sample points in the latent context grid the same as the
original papers to 512.

###  4.2 Rayleigh-Bénard Convection

Problem description. Rayleigh-Bénard convection is a model for turbulent
convection of a fluid confined between two thermally conducting plates. The
upper and lower plates are cold and hot respectively. The fluid particles tend
to produce vortices and turbulent regiments due to a temperature gradient. We
are concerned with four physical variables, which are pressure $p$,
temperature $T$, velocity component $u$ in the $x$ direction, and velocity
component $w$ in the $z$ direction. In this section, we use the same setup
with MeshfreeFlowNet to run the flow simulation.

Dataset generation. We use Dedalus framework [47] to numerically solve the
governing PDEs of Rayleigh-Bénard convection problem. The obtained Ground
Truth data has spatio-temporal dimensions of $128\times 512\times 200$
$(h\times w\times t)$. During training, we continuously crop some data blocks
of $128\times 128\times 16$ to constitute the training samples for each epoch.
The cropped data blocks are downsampled to $16\times 16\times 4$ as the low-
resolution input of deep learning models. We also generate another Ground
Truth data for the evaluation by using different initial conditions and
boundaries.

Training details. The training settings of baselines are explained in
subsection 4.1. We use the same setup for our octree-based sampling
optimization method. In addition, we list the values of all hyper-parameters
in Table 1.

Table 1: Hyper-parameters used for Rayleigh-Bénard convection problem.
$\alpha_{i}$ is the weight of the $i$-th physical variable, $s$ is the
dimension size threshold, $a$ is the scale factor for octree reconstruction,
$m$ is the number of sample points in each indivisible subblock, $q$ is the
number of epochs to evaluate the training stage, and $Q$ is the loss
threshold.

Parameter | $\alpha_{i}$ | $s_{h}$ | $s_{w}$ | $s_{l}$ | $a$ | $m$ | $q$ | $Q$
---|---|---|---|---|---|---|---|---
Value | 1 | 128 | 128 | 16 | 2 | 64 | 10 | 0.02

Evaluation metrics. To compare training performance and convergence behavior,
we take snapshots of the deep learning models as they reach a specific level
of convergence. We report the number of iteration epochs and the training
time, denoted as $S$ and $T$, respectively. The number of sample points is
defined as $N$. We also provide convergence curves of 100 epochs to show the
detailed convergence process intuitively. Besides, we use peak signal to noise
ratio (PSNR) and structural similarity index (SSIM) to quantitatively evaluate
the prediction accuracy. We compute the average of the weighted physical
variables as the final results. We also report the total training time and the
total number of sample points of 100 epochs, denoted as $T_{100}$ and
$N_{100}$, respectively. The total number of octree reconstructions during
training is defined as $N_{re}$. In each table and figure, w/o is the original
version of the baseline, and w is the optimized version using our octree-based
sampling method.

Results comparison. Table 2 shows the quantitative evaluation of training
performance on Rayleigh-Bénard convection problem. It summarizes the number of
iteration epochs $S$, the training time $T$, and the number of sample points
$N$, at the convergence level of 0.04. Benefiting from importance sampling,
our method achieves a significant reduction of sampling points with the same
convergence level ($57.14\%$ reduction in MeshfreeFlowNet and $42.50\%$
reduction in TransFlowNet). As a result, the training time is $1.67\times$
faster in MeshfreeFlowNet and $1.32\times$ faster in TransFlowNet. Our method
also requires fewer epochs to achieve a convergence level consistent with the
baselines. Besides, as shown in Figure 4, the convergence curves indicate that
our octree-based sampling optimization method can help the training loss
descend more efficiently. Although the convergence acceleration of our method
is not noticeable at first, the reconstructed octree improves the convergence
performance by leaps and bounds, especially in MeshfreeFlowNet. Our method
eventually provides a lower training loss in the same number of iterations.

Table 2: Quantitative evaluation of training performance on Rayleigh-Bénard
convection problem. The convergence level is 0.04.

Method | MeshfreeFlowNet[8] | TransFlowNet[10]
---|---|---
w/o | w(ours) | w/o | w(ours)
$S$ | 27 | 20 | 22 | 21
$T$/min | 45 | 27 | 84 | 49
$N$/$10^{8}$ | 0.28 | 0.12 | 0.34 | 0.19

(a)

(b)

Figure 4: The convergence curves of MeshfreeFlowNet and TransFlowNet,
respectively. The solid green circles mark the places where the octree
reconstruction is performed.

Table 3 presents the quantitative comparison of prediction accuracy. For each
method, we use the best model after training 100 epochs to perform the
evaluation. Compared to baselines, the results of our octree-based sampling
method show improvements in both PSNR and SSIM. Not only that, our method uses
fewer sample points and shorter training time to produce better models. In
MeshfreeFlowNet, our sampling optimization makes the number of sample points
reduce by $37.25\%$, and the training accelerated by $17.58\%$. In
TransFlowNet, they are $36.36\%$ and $20.00\%$, respectively. As the time-
consuming octree reconstruction performs twice, the improvement in training
time is not as significant as the number of samples. We also provide intuitive
comparisons by visualizing the mean absolute error (MAE) of velocity component
$u$ at each position in the $x$-$z$ space. The MAE at each position is an
average of the absolute error at all moments. As shown in Figure 5, our method
gives fewer wrong predictions and significantly improves the results,
especially in MeshfreeFlowNet.

Table 3: Quantitative comparison of prediction accuracy on Rayleigh-Bénard
convection problem. The number of training epochs is 100.

Method | MeshfreeFlowNet[8] | TransFlowNet[10]
---|---|---
w/o | w(ours) | w/o | w(ours)
PSNR | 52.03 | 53.21 | 52.71 | 53.74
SSIM | 0.9894 | 0.9954 | 0.9952 | 0.9953
$T_{100}$/min | 165 | 136 | 386 | 237
$N_{100}$/$10^{8}$ | 1.02 | 0.64 | 1.54 | 0.96
$N_{re}$ | - | 2 | - | 2

(a)

(b)

Figure 5: The MAE of MeshfreeFlowNet and TransFlowNet at each position in the
$x$-$z$ space, respectively. We use the velocity component $u$ in the $x$
direction as an example.

###  4.3 Ablation Study

To further demonstrate the effectiveness of our octree-based sampling
optimization approach, we investigate a variety of octree reconstruction
strategies. Specifically, We study the impact of hyper-parameter $q$ using the
Rayleigh-Bénard convection dataset and MeshfreeFlowNet.

Table 4: Ablation studies of the octree reconstruction using the Rayleigh-
Bénard convection dataset and MeshfreeFlowNet. We first compare the training
performance at the convergence level of 0.04. Then, we use the best model
after training 100 epochs to perform the prediction evaluation. w/o is the
original version of MeshfreeFlowNet, and the others are different designs of
the octree reconstruction strategy.

Design | w/o[8] | $q=0$ | $q=5$ | $q=10$ | $q=15$ | $q=30$ | $q=+\infty$
---|---|---|---|---|---|---|---
$S$ | 27 | 23 | 23 | 20 | 22 | 22 | 20
$T$/min | 45 | 78 | 32 | 27 | 29 | 30 | 27
$N$/$10^{8}$ | 0.28 | 0.16 | 0.14 | 0.12 | 0.13 | 0.13 | 0.12
PSNR | 52.03 | 52.10 | 52.05 | 53.21 | 52.91 | 52.71 | 51.72
SSIM | 0.9894 | 0.9891 | 0.9920 | 0.9954 | 0.9930 | 0.9952 | 0.9927
$T_{100}$/min | 165 | 339 | 149 | 136 | 131 | 120 | 118
$N_{100}$/$10^{8}$ | 1.02 | 0.65 | 0.65 | 0.64 | 0.61 | 0.63 | 0.60
$N_{re}$ | - | 99 | 9 | 2 | 2 | 1 | 0

In different designs of MeshfreeFlowNet, w/o is the original version without
sampling optimization, and different values of $q$ indicate different octree
reconstruction strategies. $q=0$ means that once the training of an epoch is
finished, we update the octree and start the next stage of training
immediately without bottleneck evaluation. $q=5$ means we start bottleneck
evaluation from the 5th epoch and correspondingly use loss values of the last
5 epochs to do linear regression. We use the multi-stage training strategy
described in subsection 3.6 for $q\geq 10$. Besides, $q=+\infty$ means that we
never reconstruct the octree and use the initialized octree forever.

We use the eight evaluation metrics in subsection 4.2 to compare different
octree reconstruction strategies on training performance and results. We first
collect training data at the convergence level of 0.04 to evaluate training
performance. Then, we compare the predictive performance with the best model
after training for 100 epochs. We also provide the total training time and the
total number of sample points of 100 epochs. We summarize all the results in
Table 4.

As shown in the training performance lines of Table 4, when the octree-based
sampling optimization method is introduced, the number of sample points $N$ is
significantly reduced. At the same time, the number of iteration epochs $S$ at
the convergence level of 0.04 is slightly improved. The training time $T$ of
$q=0$ is longer than w/o because octree reconstruction is performed after each
training epoch. It takes more time than sample point reduction saves. In
addition, since only the initial octree is used before reaching the
convergence of 0.04, the training times of $q\geq 5$ are very close. The
slight difference in training time between different designs results from
random sampling within each intersection space.

As shown in the prediction performance lines of Table 4, our octree-based
sampling optimization method greatly reduces the total number of sampling
points $N_{100}$ for training 100 epochs. The PSNR and SSIM of $q=0$ are
nearly the same with w/o because frequent octree reconstructions fail to
achieve importance sampling. It also increases the training time by $105.45\%$
due to the time-consuming octree reconstructions, although the number of
sample points is reduced by $36.27\%$. The prediction performance of
$q=+\infty$ has no promotion either. As described in subsection 3.4, the
reason is the variance-based initial octree gradually fails to represent the
importance distribution of training data. Since octree is never reconstructed
during training, it accelerates the training by $28.48\%$ with a reduction of
sample points by $41.18\%$. The number of octree reconstructions $N_{re}$ of
$q=5$ is 9, leading to a minor improvement of PSNR, SSIM, and the total
training time $T_{100}$. It indicates the effectiveness of a suitable octree
reconstruction strategy in rebuilding the importance distribution of training
data.

The number of reconstructions should be determined with caution. We find a
balance between the training time and the prediction accuracy with $N_{re}=2$
when training 100 epochs. The $q=10$ design gives the best results in PSNR and
SSIM.

###  4.4 Octree-based Sampling Optimization for PINNs

Although our method is designed for the volumetric super-resolution of
scientific data, it can also be used in training tasks of other deep neural
networks. We further explore the feasibility of the octree-based sampling
optimization approach for the training of PINNs.

We use PINNs for solving the inverse problem of PDEs as an example. The
inverse problem of PDEs aims to estimate unknown parameters in equations based
on limited training data. It combines regression loss and PDEs loss as the
total loss for training. It is worth noting that we only have training data in
a specific area and PDEs with unknown parameters before training PINNs. Unlike
the volumetric super-resolution of scientific data, the Ground Truth data in
the whole spatio-temporal grid is unknown. Therefore, the sampling process
only takes place in the specific area, as does the octree initialization and
octree reconstruction. It means there is no need to compute the intersection
spaces described in the subsection 3.5. Using the octree generated for that
specific area, we can perform sampling sequentially in each indivisible
subblock that corresponds to each leaf node.

Baseline. We take the original PINNs [28] and DeepXDE [18] as baselines. The
original PINNs perform random sampling throughout training. The DeepXDE uses a
residual-based adaptive refinement approach to improve the training
efficiency. It achieves importance sampling by sampling more points near the
ones that produced a bigger loss in the last forward propagation.

Navier-Stokes equations. Navier-Stokes equations are the PDEs that describe
the flow of incompressible fluids in fluid mechanics. The 2D Navier-Stokes
equations with unknown parameters are given explicitly by

$\displaystyle\frac{\partial u}{\partial t}+\lambda_{1}(u\frac{\partial
u}{\partial x}+v\frac{\partial u}{\partial y})=-\frac{\partial p}{\partial
x}+\lambda_{2}(\frac{\partial^{2}u}{\partial
x^{2}}+\frac{\partial^{2}u}{\partial y^{2}})$ (8) $\displaystyle\frac{\partial
v}{\partial t}+\lambda_{1}(u\frac{\partial v}{\partial x}+v\frac{\partial
v}{\partial y})=-\frac{\partial p}{\partial
y}+\lambda_{2}(\frac{\partial^{2}v}{\partial
x^{2}}+\frac{\partial^{2}v}{\partial y^{2}})$ (9) $\displaystyle\frac{\partial
u}{\partial x}+\frac{\partial v}{\partial y}=0,$ (10)

where $u$ denotes the velocity component in the $x$ direction, $v$ the
velocity component in the $y$ direction, and $p$ the pressure. $\lambda_{1}$
and $\lambda_{2}$ are the unknown parameters. PINNs are used to learn the
parameters $\lambda_{1}$ and $\lambda_{2}$ that best describe the observed
data in a specific area, as well as the pressure $p$.

We consider the problem that incompressible flow passes a circular cylinder,
which is detailed described in [28]. $\lambda_{1}$ and $\lambda_{2}$ are set
to 1 and 0.01, respectively. We use the same high-resolution data set solved
by NekTar[48].

Implementation details. We initialize the first octree based on the limited
training data in a specific area of the cylinder wake. With the help of the
first octree, we train PINNs by importance sampling of input coordinates from
the beginning. When total loss reaches the bottleneck, we compute the loss
value at each coordinate in the spatio-temporal grid according to the current
best model. Then, we rebuild the octree based on the loss distribution.
Similar to subsection 3.4, we reconstruct the octree whenever the total loss
reaches a bottleneck, avoiding failure of the importance sampling. All
sampling operations are always performed in the entire domain by sampling in
each indivisible subblock sequentially.

Results comparison. We use the L2 relative error between the predicted values
and the exact solutions to evaluate the accuracy of models. The results are
summarized in Table 5.

Table 5: The results of PINNs solutions for 2D Navier-Stokes equations. w/o is
the original version of PINNs. We compare the L2 relative errors of physical
variables and unknown parameters, respectively. We also report the total
number of sample points of each method.

Method | w/o[28] | DeepXDE[18] | w(ours)
---|---|---|---
$u$ | 5.4e-04 | 4.1e-04 | 3.6e-04
$v$ | 1.9e-03 | 1.5e-03 | 1.3e-03
$p$ | 6.6e-02 | 5.1e-02 | 1.6e-02
$\lambda_{1}$ | 5.8e-02 | 5.9e-02 | 4.3e-02
$\lambda_{2}$ | 5.0e-03 | 4.7e-03 | 3.9e-03
$N$ | 5000 | 5000 | 3920

As shown in Table 5, our octree-based sampling optimization achieves
improvements of the original PINNs by $33.33\%$ on $u$, $31.58\%$ on $v$,
$75.76\%$ on $p$, $25.86\%$ on $\lambda_{1}$, and $22.00\%$ on $\lambda_{2}$,
respectively. At the same time, only $78.40\%$ of sample points are required.
It indicates that our method produces higher solution accuracy with fewer
sample points. In addition, compared to DeepXDE, our octree-based sampling
optimization can provide better importance sampling on training PINNs.

##  5 Limitations and Discussions

As described in subsection 4.2 and subsection 4.3, our octree-based sampling
optimization is able to greatly reduce the number of sample points required to
achieve a specific level of convergence. However, since the indispensable
octree reconstruction relies on the time-consuming computation of the loss
distribution, the saving of training time is not as dramatic as sample points.
Training time is the most concern for processing massive scientific data. We
must explore how to accelerate the estimation of the loss distribution. As
mentioned in subsection 3.4, we currently downsample the training data to
reduce the time for rebuilding the loss distribution. But a larger
downsampling factor will affect the performance of the reconstructed octree.
Besides, the minimum number of epochs for each stage of training is a key
hyperparameter of our method. How to determine it reasonably according to the
specific application is a problem that needs to be further solved.

##  6 Conclusion

We propose a novel octree-based hierarchical sampling optimization for the
volumetric super-resolution of scientific data. It effectively implements
importance sampling by exploiting the importance distribution of training
data. We model the volumetric scientific data by leveraging its importance
distribution using octree, wherein each leaf node corresponds to an
indivisible subblock in the volumetric domain. In particular, we reconstruct
the octree whenever the loss reaches a bottleneck, avoiding failure of the
sampling optimization. We conduct adequate experiments to show that our method
greatly reduces the number of sample points required to achieve a specific
level of convergence. Besides, our method can also be used in training tasks
of other deep neural networks like PINNs. Future work will focus on the
acceleration of the loss distribution computation. We hope that the octree-
based sampling optimization will facilitate research on accelerating the
training of deep learning models.

## Acknowledgements

Xinjie Wang was supported by the National Key R&D Program of China (#
2022YFC2803805), the Fundamental Research Funds for the Central Universities
(# 202313035), the Shandong Provincial Natural Science Foundation of China (#
ZR2021QF124), and the China Postdoctoral Science Foundation (# 2021M703031).
Xiaogang Jin was supported by the National Natural Science Foundation of China
(# 62036010) and the Key R&D Program of Zhejiang Province (# 2022C03126).

## References

  * [1] T. J. Chung, Computational Fluid Dynamics, Cambridge University Press, 2002.  doi:10.1017/CBO9780511606205.
  * [2] P. K. Yeung, X. M. Zhai, K. R. Sreenivasan, Extreme events in computational turbulence, Proceedings of the National Academy of Sciences 112 (41) (2015) 12633–12638.  doi:10.1073/pnas.1517368112.
  * [3] J. W. Hurrell, M. M. Holland, P. R. Gent, S. Ghan, J. E. Kay, P. J. Kushner, J.-F. Lamarque, W. G. Large, D. Lawrence, K. Lindsay, W. H. Lipscomb, M. C. Long, N. Mahowald, D. R. Marsh, R. B. Neale, P. Rasch, S. Vavrus, M. Vertenstein, D. Bader, W. D. Collins, J. J. Hack, J. Kiehl, S. Marshall, The community earth system model: A framework for collaborative research, Bulletin of the American Meteorological Society 94 (9) (2013) 1339–1360.  doi:10.1175/BAMS-D-12-00121.1.
  * [4] G. K. Vallis, Atmospheric and Oceanic Fluid Dynamics: Fundamentals and Large-Scale Circulation, 2nd Edition, Cambridge University Press, 2017.  doi:10.1017/9781107588417.
  * [5] L. F. Richardson, P. Lynch, Weather Prediction by Numerical Process, 2nd Edition, Cambridge Mathematical Library, Cambridge University Press, 2007.  doi:10.1017/CBO9780511618291.
  * [6] A. S. Almgren, J. B. Bell, M. J. Lijewski, Z. Lukić, E. V. Andel, Nyx: A massively parallel amr code for computational cosmology, The Astrophysical Journal 765 (1) (2013) 39.  doi:10.1088/0004-637X/765/1/39.
  * [7] P. Bauer, A. Thorpe, G. Brunet, The quiet revolution of numerical weather prediction, Nature 525 (7567) (2015) 47–55.  doi:10.1038/nature14956.
  * [8] C. M. Jiang, S. Esmaeilzadeh, K. Azizzadenesheli, K. Kashinath, M. Mustafa, H. A. Tchelepi, P. Marcus, M. Prabhat, A. Anandkumar, Meshfreeflownet: A physics-constrained deep continuous space-time super-resolution framework, in: SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, 2020, pp. 1–15.  doi:10.1109/SC41405.2020.00013.
  * [9] R. Arora, A. Shrivastava, Spatio-temporal super-resolution of dynamical systems using physics-informed deep-learning, arXiv preprint arXiv:2212.04457 (2022).
  * [10] X. Wang, S. Zhu, Y. Guo, P. Han, Y. Wang, Z. Wei, X. Jin, Transflownet: A physics-constrained transformer framework for spatio-temporal super-resolution of flow simulations, Journal of Computational Science 65 (2022) 101906.  doi:https://doi.org/10.1016/j.jocs.2022.101906.
  * [11] P. Ren, C. Rao, Y. Liu, Z. Ma, Q. Wang, J.-X. Wang, H. Sun, Physics-informed deep super-resolution for spatiotemporal data, arXiv preprint arXiv:2208.01462 (2022).
  * [12] Z. Yang, Z. Qiu, D. Fu, Dmis: Dynamic mesh-based importance sampling for training physics-informed neural networks, arXiv preprint arXiv:2211.13944 (2022).
  * [13] A. Katharopoulos, F. Fleuret, Biased importance sampling for deep neural network training, arXiv preprint arXiv:1706.00043 (2017).
  * [14] J. Chen, T. Ma, C. Xiao, Fastgcn: Fast learning with graph convolu-tional networks via importance sampling, in: International Conference on Learning Representations, 2018.
  * [15] T. B. Johnson, C. Guestrin, Training deep models faster with robust, approximate importance sampling, in: Advances in Neural Information Processing Systems, Vol. 31, 2018.
  * [16] S. Banerjee, S. Chakraborty, Deterministic mini-batch sequencing for training deep neural networks, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, 2021, pp. 6723–6731.  doi:10.1609/aaai.v35i8.16831.
  * [17] M. A. Nabian, R. J. Gladstone, H. Meidani, Efficient training of physics-informed neural networks via importance sampling, Computer-Aided Civil and Infrastructure Engineering 36 (8) (2021) 962–977.
  * [18] L. Lu, X. Meng, Z. Mao, G. E. Karniadakis, Deepxde: A deep learning library for solving differential equations, SIAM review 63 (1) (2021) 208–228.  doi:10.1137/19M1274067.
  * [19] Z. Que, G. Lu, D. Xu, Voxelcontext-net: An octree based framework for point cloud compression, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 6042–6051.
  * [20] P.-S. Wang, Y. Liu, X. Tong, Dual octree graph networks for learning adaptive volumetric shape representations, ACM Transactions on Graphics 41 (4) (2022) 1–15.
  * [21] C. Fu, G. Li, R. Song, W. Gao, S. Liu, Octattention: Octree-based large-scale contexts model for point cloud compression, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36, 2022, pp. 625–633.
  * [22] Y. Xie, E. Franz, M. Chu, N. Thuerey, Tempogan: A temporally coherent, volumetric gan for super-resolution fluid flow, ACM Transactions on Graphics 37 (4) (2018).  doi:10.1145/3197517.3201304.
  * [23] M. Werhahn, Y. Xie, M. Chu, N. Thuerey, A multi-pass gan for fluid flow super-resolution, Proceedings of the ACM on Computer Graphics and Interactive Techniques 2 (2) (2019).  doi:10.1145/3340251.
  * [24] K. Fukami, K. Fukagata, K. Taira, Super-resolution reconstruction of turbulent flows with machine learning, Journal of Fluid Mechanics 870 (2019) 106–120.  doi:10.1017/jfm.2019.238.
  * [25] L. Guo, S. Ye, J. Han, H. Zheng, H. Gao, D. Z. Chen, J.-X. Wang, C. Wang, Ssr-vfd: Spatial super-resolution for vector field data analysis and visualization, in: 2020 IEEE Pacific Visualization Symposium, 2020, pp. 71–80.  doi:10.1109/PacificVis48177.2020.8737.
  * [26] J. Han, C. Wang, Tsr-tvd: Temporal super-resolution for time-varying data analysis and visualization, IEEE Transactions on Visualization and Computer Graphics 26 (1) (2020) 205–215.  doi:10.1109/TVCG.2019.2934255.
  * [27] S. Weiss, M. Chu, N. Thuerey, R. Westermann, Volumetric isosurface rendering with deep learning-based super-resolution, IEEE Transactions on Visualization and Computer Graphics 27 (6) (2021) 3064–3078.  doi:10.1109/TVCG.2019.2956697.
  * [28] M. Raissi, P. Perdikaris, G. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational Physics 378 (2019) 686–707.  doi:10.1016/j.jcp.2018.10.045.
  * [29] Y. Zhu, N. Zabaras, P.-S. Koutsourelakis, P. Perdikaris, Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data, Journal of Computational Physics 394 (2019) 56–81.  doi:10.1016/j.jcp.2019.05.024.
  * [30] G. Pang, L. Lu, G. E. Karniadakis, fpinns: Fractional physics-informed neural networks, SIAM Journal on Scientific Computing 41 (4) (2019) A2603–A2626.  doi:10.1137/18M1229845.
  * [31] A. D. Jagtap, G. Em Karniadakis, Extended physics-informed neural networks (xpinns): A generalized space-time domain decomposition based deep learning framework for nonlinear partial differential equations, Communications in Computational Physics 28 (5) (2020) 2002–2041.  doi:10.4208/cicp.OA-2020-0164.
  * [32] L. Yang, X. Meng, G. E. Karniadakis, B-pinns: Bayesian physics-informed neural networks for forward and inverse pde problems with noisy data, Journal of Computational Physics 425 (2021) 109913.  doi:10.1016/j.jcp.2020.109913.
  * [33] X. Jin, S. Cai, H. Li, G. E. Karniadakis, Nsfnets (navier-stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations, Journal of Computational Physics 426 (2021) 109951.  doi:10.1016/j.jcp.2020.109951.
  * [34] K. Shukla, A. D. Jagtap, G. E. Karniadakis, Parallel physics-informed neural networks via domain decomposition, Journal of Computational Physics 447 (2021) 110683.  doi:10.1016/j.jcp.2021.110683.
  * [35] K. Xu, E. Darve, Physics constrained learning for data-driven inverse modeling from sparse observations, Journal of Computational Physics 453 (2022) 110938.  doi:10.1016/j.jcp.2021.110938.
  * [36] K. Shukla, P. C. Di Leoni, J. Blackshire, D. Sparkman, G. E. Karniadakis, Physics-informed neural network for ultrasound nondestructive quantification of surface breaking cracks, Journal of Nondestructive Evaluation 39 (3) (2020).  doi:10.1007/s10921-020-00705-1.
  * [37] S. Goswami, M. Yin, Y. Yu, G. E. Karniadakis, A physics-informed variational deeponet for predicting crack path in quasi-brittle materials, Computer Methods in Applied Mechanics and Engineering 391 (2022) 114587.  doi:10.1016/j.cma.2022.114587.
  * [38] E. Haghighat, M. Raissi, A. Moure, H. Gomez, R. Juanes, A physics-informed deep learning framework for inversion and surrogate modeling in solid mechanics, Computer Methods in Applied Mechanics and Engineering 379 (2021) 113741.  doi:10.1016/j.cma.2021.113741.
  * [39] S. Sharma, R. Awasthi, Y. S. Sastry, P. R. Budarapu, Physics-informed neural networks for estimating stress transfer mechanics in single lap joints, Journal of Zhejiang University-SCIENCE A 22 (8) (2021) 621–631.  doi:10.1631/jzus.A2000403.
  * [40] R. Wang, K. Kashinath, M. Mustafa, A. Albert, R. Yu, Towards physics-informed deep learning for turbulent flow prediction, in: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, 2020, p. 1457–1466.  doi:10.1145/3394486.3403198.
  * [41] Z. Mao, A. D. Jagtap, G. E. Karniadakis, Physics-informed neural networks for high-speed flows, Computer Methods in Applied Mechanics and Engineering 360 (2020) 112789.  doi:10.1016/j.cma.2019.112789.
  * [42] D. Chen, X. Gao, C. Xu, S. Wang, S. Chen, J. Fang, Z. Wang, Flowdnn: A physics-informed deep neural network for fast and accurate flow prediction, Frontiers of Information Technology & Electronic Engineering 23 (2) (2022) 207–219.  doi:10.1631/FITEE.2000435.
  * [43] F. Sahli Costabal, Y. Yang, P. Perdikaris, D. E. Hurtado, E. Kuhl, Physics-informed neural networks for cardiac activation mapping, Frontiers in Physics 8 (2020).  doi:10.3389/fphy.2020.00042.
  * [44] G. Kissas, Y. Yang, E. Hwuang, W. R. Witschey, J. A. Detre, P. Perdikaris, Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4d flow mri data using physics-informed neural networks, Computer Methods in Applied Mechanics and Engineering 358 (2020) 112623\.  doi:10.1016/j.cma.2019.112623.
  * [45] G. Alain, A. Lamb, C. Sankar, A. Courville, Y. Bengio, Variance reduction in sgd by distributed importance sampling, arXiv preprint arXiv:1511.06481 (2015).
  * [46] A. Katharopoulos, F. Fleuret, Not all samples are created equal: Deep learning with importance sampling, in: International Conference on Machine Learning, 2018, pp. 2525–2534.  doi:10.1111/mice.12685.
  * [47] K. J. Burns, G. M. Vasil, J. S. Oishi, D. Lecoanet, B. P. Brown, Dedalus: A flexible framework for numerical simulations with spectral methods, Phys. Rev. Research 2 (2020) 023068.  doi:10.1103/PhysRevResearch.2.023068.
  * [48] G. E. Karniadakis, G. Karniadakis, S. Sherwin, Spectral/hp element methods for computational fluid dynamics, Oxford University Press on Demand, 2005.