# Convolutional neural networks for fluid flow analysis: toward effective
metamodeling and low-dimensionalization

Masaki Morimoto<EMAIL_ADDRESS>Department of Mechanical
Engineering, Keio University, Yokohama 223-8522, Japan  Kai Fukami
<EMAIL_ADDRESS>Department of Mechanical and Aerospace Engineering,
University of California, Los Angeles, CA 90095, USA  Department of Mechanical
Engineering, Keio University, Yokohama 223-8522, Japan  Kai Zhang
<EMAIL_ADDRESS>Department of Mechanical and Aerospace Engineering,
Rutgers University, Piscataway, NJ 08854, USA  Aditya G. Nair<EMAIL_ADDRESS>Department of Mechanical Engineering, University of Nevada, Reno, NV 89557,
USA  Koji Fukagata<EMAIL_ADDRESS>Department of Mechanical
Engineering, Keio University, Yokohama 223-8522, Japan

###### Abstract

We focus on a convolutional neural network (CNN), which has recently been
utilized for fluid flow analyses, from the perspective on the influence of
various operations inside the CNN by considering some canonical regression
problems with fluid flow data. We consider two types of the CNN-based fluid
flow analyses; 1. CNN metamodeling and 2. CNN autoencoder. For the first type
of CNN with the additional scalar inputs, which is one of the common forms of
CNN for fluid flow analysis, we investigate the influence of input placements
in the CNN training pipeline. As an example, the estimation of drag and lift
coefficients of laminar flows over a flat plate and two side-by-side cylinders
are considered. For the example of flat plate wake, we use the chord Reynolds
number $Re_{c}$ and the angle of attack $\alpha$ as the additional scalar
inputs to give a CNN the information of the complexity of wake and influence
on the flat plate. For the wake interaction problem comprising flows over two
side-by-side cylinders, the gap ratio and the diameter ratio are utilized as
the additional inputs. We find that care should be taken for the placement of
additional scalar inputs depending on the problem setting and the complexity
of flows users handle. We then move to the discussion for the influence of
various parameters and operations on CNN performance, with the utilization of
autoencoder (AE). A two-dimensional decaying homogeneous isotropic turbulence
is considered for the demonstration of AE. The results obtained through the AE
highly rely on the decaying nature. The investigation for the influence of
padding operation at a convolutional layer is also performed. The zero padding
shows reasonable ability compared to other methods which account for boundary
conditions of numerical data. Moreover, the effect of the dimensional
reduction/extension methods inside CNN is also examined. The CNN model is
robust to the dimension reduction operations, while it is sensitive to the
dimensional extension methods. The findings through the paper can help us
toward the practical uses of CNN-based fluid flow analyses.

##  I Introduction

Figure 1: Overview of the present study. We consider a general form of CNN-
based modeling with additional scalar input ${\bm{y}}={\cal
F}({\bm{x}},{\phi})$.

Convolutional neural networks (CNNs) have recently been recognized as a
powerful tool to analyze complex fluid flow phenomena [1]. Although they are
still in the phase of fundamental studies, the great potential of CNN-based
analyses can be found in turbulence modeling [2], data reconstruction [3], and
low-dimensionalization [4]. However, current uses of CNNs require not only the
tuning of bunch of parameters but also the incorporation of a priori knowledge
into its modeling. Due to these issues, the recent uses of CNNs for fluid
flows are based on trial-and-error iterations by users because of lack of
guidelines. In this paper, we address the aforementioned issues by focusing on
a general form of CNN-based regression tasks for fluid flow analysis
${\bm{y}}={\cal F}({\bm{x}},{\phi})$, where ${\bm{x}}$ and ${\bm{y}}$
respectively indicate input and output data of machine learning model ${\cal
F}$ with additional scalar inputs $\phi$.

One of hot uses of CNN for fluid flow analyses is turbulence modeling [2].
Lapeyre  _et al._ [5] utilized a U-net based CNN for estimating sub-grid scale
reaction rates in their large-eddy simulation (LES) framework. At that moment,
previous studies regarding NNs and LES sub-grid modeling had been conducted
using a multi-layer perceptron (MLP) [6]. Hence, the use of CNN for the LES
closure was one of the novelty of their study. The comparison between the MLP
and the CNN for LES closure modeling is well discussed by Pawar  _et al._ [7].
The CNN can also be applied to modeling for Reynolds-averaged Navier–Stokes
(RANS) simulations. Thuerey  _et al._ [8] considered the estimation for the
airfoil flow field obtained from RANS toward solving one of the issues that
many researchers have skeptical thinking for applications of NNs for RANS. In
addition to the aforementioned efforts on LES and RANS, Font  _et al._ [9] has
recently proposed the new concept for the decomposition of Navier–Stokes
equation called spanwise-averaged N–S equation. Analogous to LES and RANS
equations, it also has a closure term. They used a CNN based regressor to
close that term with a velocity field, while considering the example of wakes
behind a cylinder and ellipse.

CNN-based fluid flow data reconstruction can also be regarded as a promising
field. For instance, the CNN-based super-resolution analysis proposed by
Fukami et al. [10] is a proof of concept of fluid flow data recovery with
machine learning. The aim of super resolution is to reconstruct high-
resolution data from its low-resolution counterpart. Since this low-resolution
part can also be replaced for various forms which indicates low-resolution
fluid flow information, e.g., local sensor measurements and limited
availability of data, the super-resolution idea has been extended to not only
numerical [11, 12] but also experimental studies [13, 14, 15]. From the view
of estimation, the combination of CNN and MLP can widely be seen for their
purposes, since their output shape often has a form of scalar. For example,
Salehipour and Peltier [16] attempted to predict the small-scale structures in
the ocean turbulence called atoms using a CNN. Otherwise, Morimoto  _et al._
[17] visualized the internal procedures of CNN-MLP model with fluid flow
analyses toward practical applications from the perspective on
interpretability.

Furthermore, the CNN-based modeling is also capable of low-dimensionalization
of fluid flows, as a form of autoencoder (AE) [18]. The AE has the same output
data as the input while having the dimensional compression procedure inside
its structure. Because the AE is trained to output the same data as the input,
the latent variables, which can be extracted from the bottleneck layer inside
the AE, can be regarded as a low-dimensionalized representation of high-
dimensional data, if the reconstruction via the AE is well performed. To the
best of our knowledge, Milano and Koumoutsakos [19] first brought the idea of
AE into the fluid dynamics field. Although their model was based on the MLP,
the CNN has also recently been applied to build AE thanks to the CNN’s great
advantage against the MLP in terms of the number of weights inside the model
while being able to keep its accuracy [20, 21]. In addition, the extracted
low-dimensional representations via AE can also be utilized for the
construction of reduced-order modeling which has a similar form of proper
orthogonal decomposition-based Galerkin integration [22, 23, 24, 25, 26].

As discussed above, we are now able to appreciate the strong potential and
applicability of CNN for fluid flow analyses. However, the current success of
CNN and fluid flow analysis is based on trial-and-error iterations by fluid
mechanicians since we have no guideline for parameter decision inside the CNN.
For instance, the work on super-resolution analysis introduced above [10]
reported the importance of the utilization of multi-size filters inside CNNs
so as to account for a wide range of scales included in turbulence. The
similar idea can also be found in the construction of NN-based reduced order
modeling [27, 28] and surrogate models for high-fidelity simulations [29].
Otherwise, several reports utilize additional scalar inputs which highly
relates to fluid flow phenomena, e.g., angle of attack, Reynolds number, and
bluff body shapes, to improve the estimation or low-dimensionalization
abilities of CNNs [30, 31, 32, 33, 34, 35, 36, 37, 38]. Hence, we now arrive
at the question: how can we determine these strategies to achieve a better
performance of CNN and fluid flows? — In this paper, we tackle this vague
portion with regard to the use of CNN and fluid flow analyses.

As mentioned earlier, we consider CNN-based regression tasks ${\bm{y}}={\cal
F}({\bm{x}},{\phi})$ of fluid flow analyses. In this paper, we consider two
types of CNN-aided modeling, as illustrated in figure 1: 1. CNN-MLP based
metamodeling of unsteady laminar wakes and 2. CNN-AE for turbulence, so as to
investigate various considerable parameters inside CNN. The organization of
the present paper is as follows: We first introduce the basic principle of CNN
and the covered models based on CNN in section II.1. The information for fluid
flow data used in the present paper is offered in section II.2. Results and
discussions are provided in section III. We finally give concluding remarks in
section IV.

##  II Methods

###  II.1 Convolutional neural network (CNN) for fluid flow analyses

####  II.1.1 Basic principle of CNN

Figure 2:  Basic operations of CNN. $(a)$ A structure of typical CNN-AE with
convolutional layers, pooling layers and dimensional expanding layers. $(b)$
Convolutional operation at each convolutional layer to obtain $q^{s}$ from
$q^{s-1}$. $(c)$ Convolution, pooling and dimensional expansion. Gray zero
values indicate additionally embedded values for each operation.

Let us first introduce operations inside a convolutional neural network (CNN)
[39]. The CNN was originally developed in image recognition tasks. In
particular, various efforts on CNN-based image classification can now be
widely seen such as ResNet [40], GoogLeNet [41], and SENet [42]. Because the
CNN is good at handling high-dimensional data through the concept of weight
sharing, it is also becoming one of the promising tools to analyse fluid flows
[43, 44, 45].

A CNN is generally consisted with several types of layers, i.e., convolutional
layer, pooling layer, and up-sampling layer, as illustrated in figure 2$(a)$
with an example of CNN autoencoder. A main operation of CNN is performed at
the convolutional layer which extracts spatial features of input data through
filter operations, as presented in figure 2$(b)$. A fundamental operation of
convolutional layer shown in figure 2$(b)$ is taking a summation of an
Hadamard product of an arbitrary portion of input data and a filter $h$.
Output data of a convolutional layer $q^{(s)}$ can be expressed as,

$q^{(s)}_{ijn}=\varphi\left(\sum_{m=1}^{M}\sum_{p=0}^{H-1}\sum_{q=0}^{H-1}h^{(s)}_{pqmn}q^{(s-1)}_{i+p-G,j+q-G,m}+b_{n}^{(s)}\right),$
(1)

where $G=\lfloor H/2\rfloor$ (where $\lfloor\cdot\rfloor$ represents the
operation of rounding down the value to the nearest decimal), $H$ is width and
height of the filter, $M$ is the number of input channel, $n$ is the number of
output channel, $b$ is a bias, and $\varphi$ is an activation function,
respectively. Note that we showed the two-dimensional operation above since
two-dimensional flows are only handled through the present study, although its
extension to three-dimensional flows are rather straightforward [46, 27]
albeit computationally more expensive. The nonlinear activation function
$\varphi$ enables a machine learning model to account for nonlinearlities into
its estimation. There are various choices of nonlinear activation functions
[47]. We here utilize the ReLU function [48], which can avoid vanishing the
gradient of weights in deep CNNs. Weights ${\bm{w}}$ (values on filters) are
optimized through a backpropagation [49] to minimize the loss function between
estimated data and reference data ${\bm{q}}_{\rm Ref}$,

${\bm{w}}={\rm argmin}_{\bm{w}}||{\bm{q}}^{(s_{\rm max})}-{\bm{q}}_{\rm
Ref}||_{2},$ (2)

where ${\bm{q}}^{(s_{\rm max})}$ is an output of CNN at the last layer $s_{\rm
max}$.

We also utilize other layers to process data inside a CNN. One of them is a
pooling layer which downscales the data. This feature is useful because we
often want to reduce the dimension of data for the regression or
classification problems — for example, to estimate a scalar value from high-
dimensional data [1], and to extract key features using an autoencoder [50,
18, 21]. There are mainly two methods for the downsampling in the pooling
layer, i.e., max and average pooling, as illustrated in figure 2$(c)$. Both
methods are common in extracting a representative value from an arbitrary
area; the difference only lies at whether to extract, max or average values.
This difference can mathematically be written as,

$q_{ij}^{\rm LR}=\left(\frac{1}{\gamma^{2}}{\sum_{k,l\in P_{i,j}}(q_{kl}^{\rm
HR})^{P}}\right)^{1/P},$ (3)

where $P=1$ or $\infty$ provides average or max pooling, for the arbitrary
region ($\gamma\times\gamma$) of the input data $q^{\rm HR}$.

Contrary to the pooling operation, we can also consider to expand the
dimension of the data inside a CNN. For example, it is required in
constructing a CNN autoencoder (for decoder part), and in estimating the two-
dimensional sectional flow field from scalar values of sensor measurements
using CNN [1, 18, 19, 47, 51]. There are also mainly two techniques to expand
the dimension, i.e., up-sampling and transposed convolution, as shown in
figure 2$(c)$. The up-sampling is the simple operation which copies the value
onto an arbitrary region. On the other hand, the transposed convolution [52,
53] is fundamentally an inverse operation of regular convolution. As
illustrated in figure 2$(c)$, the input data is first expanded by embedding
zero value among grid points. The regular convolutional operation (equation 1)
is then applied to expand the dimension of data.

As introduced above, we have various candidates to construct a CNN depending
on users’ tasks. However, detailed investigation for the influence on various
parameters inside CNNs has not been performed yet to the best of our
knowledge, despite that these operations are deemed to keys for CNN-based
fluid flow analyses. In this study, we address this point by focusing on the
choice of downsampling operations and dimensional expanding techniques.

####  II.1.2 Covered CNN models

Figure 3:  CNN-MLP model covered in this study. Pink nodes indicate additional
scalar inputs of the flow; 1. $\\{g,r\\}$ for two side-by-side cylinders flow
and 2. $\\{Re_{c},\alpha\\}$ for flat plate wake. Figure 4:  CNN-AE utilized
with two-dimensional decaying turbulence. Pink nodes indicate additional
scalar inputs of the flow, $\\{Re_{0},Re_{\lambda}\\}$.

As mentioned above, we consider two types of convolutional neural network
(CNN)-based architectures: 1. combination of CNN and multi-layer perceptron
(CNN-MLP) and 2. CNN-based autoencoder (CNN-AE), which have widely been
utilized in both regression and classification tasks [54].

Most of image classification models are consisted with CNN-MLP model since it
generally outputs scalar values as a probability of each class from sectional
or volumetric data [55]. It is also utilized in fluid flow analyses when it
requires to output scalar variables, e.g., aerodynamics coefficients, from
two-dimensional data such as flow fields [33]. Following is the construction
of CNN-MLP model;

  1. 1.

Convolutional layers and pooling layers are respectively utilized to extract
key features and to downsample images, as shown in figure 3.

  2. 2.

An MLP is then adopted to output scalar values after reshaping (i.e.,
‘Flatten’ in figure 3) the data to one-column matrix.

In this study, we aim to estimate drag and lift coefficients of flows over two
side-by-side cylinders and a flat plate.

When utilizing CNNs for coefficient estimations of fluid flows, additional
scalar values which strongly relates to fluid flow phenomena, e.g., the
Reynolds number, the Mach number, and the angle of attack, are often inserted
to help its estimations [33, 34]. Although it is a popular technique, the
placement of these scalar inputs as additional information is carelessly
decided by users to date. To investigate this point, we consider additional
scalar inputs for each example,

  1. 1.

the chord Reynolds number $Re_{c}$ and the angle of attack $\alpha$ for a flat
plate wake, and

  2. 2.

the diameter ratio of two cylinders $r$ and distance between two cylinders $g$
for two parallel cylinders’ wake,

from four different cases of placements, as illustrated in figure 3. For cases
1 and 2, the dimension of scalar inputs are expanded using MLP and CNN to
concatenate with the output of the first and third convolutional layers. In
contrast, the scalar values are directly concatenated to the first and sixth
layers of MLP for cases 3 and 4.

Another well-known method of CNN-based fluid flow analyses is the utilization
as a CNN-AE. An AE [50] is used to map high-dimensional data into low-
dimensional feature space referred to as latent vector. In particular, CNN-AE,
an AE with convolutional layers, is known as a good candidate to extract key
features of two-dimensional images [18, 56, 31, 32, 27]. An encoder part — the
first half of CNN-AE — contains the pooling layers and the convolutional
layers to extract key features of input data while reducing the dimension of
data. A decoder part — the latter half of CNN-AE — then operates for expanding
the dimension of the data. If we can obtain the same output as the input data
through the bottleneck procedure, it implies that the information of input
data can successfully be low-dimensionalized into the latent space. In the
present study, the up-sampling layer and the transposed convolutional layer
are considered for the means of dimension expansion, as introduced in section
II.1.1.

For the investigation of CNN-AE, we utilize a decaying homogeneous isotropic
turbulence, as presented in figure 4. To concatenate the scalar values with
the convolutional layers, these scalars are expanded to two-dimensional
sectional data using the MLP and convolutional layers similar to the
investigation of CNN-MLP model. Here, we consider four different input
placements, i.e., the input layer (case 1), the third convolutional layer
(case 2), the latent space (case 3), and the 16th layer in the decoder part
(case 4).

###  II.2 Fluid flow data sets

####  II.2.1 Flat plate wake

Figure 5: Wakes of flow over a flat plate at (a) $Re_{c}=100$ and (b)
$Re_{c}=1000$. Shown are the instantaneous (first column) and time-averaged
(middle column) vorticity fields $\omega\in[-2,2]$, with blue and red
representing negative and positive values, respectively. The lift and drag
coefficients for each case are shown in the last column.

We consider a flat plate wake at the chord-based Reynolds number of
$Re_{c}=\\{100,1000\\}$ with the angle of attack
$\alpha=\\{35^{\circ},70^{\circ}\\}$. A two-dimensional direct numerical
simulation (DNS) with an immersed boundary projection method [57] is performed
to simulate the flows. The solver employs a discrete vorticity-stream function
formulation with an immersed boundary framework to generate the plate. The
leading edge of the flat plate is placed at $(x/c,y/c)=(0,0)$. A multi-domain
technique [58] with five grid levels is utilized with the finest inner-most
domain as $-0.2\leq x/c\leq 1.8,-1\leq y/c\leq 1$ and grid spacing of $\Delta
x/c=0.008$. The solver uses the Crank-Nicolson scheme for the viscous term and
an explicit second-order Adam-Bashforth method for the advective term. At the
far-field boundaries, uniform flow is prescribed. To possess sufficient
spatial and temporal resolution for estimation of lift and drag forces, we
collect $1200$ snapshots within the spatial domain $-2.5\leq x/c\leq
8.73,-4\leq y/c\leq 4$ at a sampling frequency of $fU_{\infty}/c=10$.

In all four cases analyzed, unsteady vortex shedding behavior is observed in
the wake. The complexity of the vortical structures increases with the angle
of attack and also with the Reynolds number, as shown in figure 5. As the
nonlinear interactions in the wake increases, the time-averaged flow fields
reveal the deviation of the vortical structures from the centerline. The time-
averaged lift $\bar{C}_{L}$ and drag forces $\bar{C}_{D}$ along with the
dominant shedding frequency $St$ for each case is also summarized in figure 5.
The drag forces increase dramatically with angle of attack while the shedding
frequency decreases.

Figure 6: $(a)$ Computational domain for flow over two side-by-side cylinders.
$(b)$ Wakes of flow over two parallel cylinders. Shown are the vorticity
fields $\omega\in[-1,1]$, with blue and red representing negative and positive
values, respectively.

####  II.2.2 Wake of two side-by-side cylinders

The wake interactions between two side-by-side circular cylinders with uneven
diameter is considered. A schematic view of the problem setup is shown in
figure 6$(a)$. The two circular cylinders with a size ratio of $r$ are
separated with a gap of $gD$, where $g$ is the gap ratio. The Reynolds number
is fixed at $Re_{D}=U_{\infty}D/\nu=100$. The two cylinders are placed $20D$
downstream of the inlet where a uniform flow with velocity $U_{\infty}$ is
prescribed, and $40D$ upstream of the outlet with zero pressure. The side
boundaries are specified as slip and are $40D$ apart. The flows over the two
cylinders are solved by the open-source CFD toolbox OpenFOAM [59], using
second-order discretization schemes in both time and space.

The flow physics is governed by two parameters; the size ratio $r$ and the gap
ratio $g$. The flow fields of $r=\\{1.00,1.15,1.30\\}$, and $g=\\{0.5-2.5\\}$
are shown in figure 6. For $g=\\{0.5,0.7,1.0\\}$, the wakes are generally
chaotic for all three size ratios. For higher gap ratios, the wakes restore
order, but are characterized by different features. In the case of two
identical cylinders ($r=1.00$), for $g=\\{1.5,2.0\\}$, the vortices shed from
the two cylinders are in phase with each other in the near wake, and merge
into a larger binary vortex street downstream. At $g=2.5$, the parallel vortex
streets are out-of-phase with each other, together the form a pair of
symmetric flow pattern with respect to $x$ axis. For cylinders with different
but close diameter ($r=1.15$), the two vortex streets with different natural
shedding frequencies can synchronize to form a single binary vortex street, as
is the case for $g=1.5$. With the increase in the gap ratio, the coupling
between the two vortex streets becomes weaker. As a result, the wakes of
$g=\\{2.0,2.5\\}$ are characterized by quasi-periodicity due to the
interactions between the two vortex streets with different shedding
frequencies. Such quasi-periodic flow also prevails for cases with
$(r,g)=(1.30,\\{1.5-2.5\\})$, for which synchronization does not occur due to
the large difference in the natural shedding frequencies.

####  II.2.3 Decaying homogeneous isotropic turbulence

Figure 7: Two-dimensional decaying isotropic homogeneous turbulence utilized
in this study. $(a)$ Vorticity field of flows at each initial Reynolds number
$Re_{0}$ and $(b)$ time evolution of their Taylor Reynolds numbers
$Re_{\lambda}$.

To examine the influence of CNN performance on various parameters inside the
CNN-AE, we also consider a two-dimensional decaying homogeneous isotropic
turbulence. The training data set is prepared by a DNS [60]. The governing
equation is the two-dimensional vorticity transport equation,

$\partial_{t}\omega+{\bm{u}}\cdot\nabla\omega=Re_{0}^{-1}\nabla^{2}\omega,$
(4)

where ${\bm{u}}=(u,v)$ and $\omega$ are the velocity and vorticity,
respectively. The size of the computational domain is $L_{x}=L_{y}=1$. In this
study, three initial Reynolds numbers $Re_{0}\equiv
u^{*}l_{0}^{*}/\nu=\\{80.4,177,442\\}$ are considered, where $u^{*}$ is the
characteristic velocity obtained by the square root of the spatially averaged
initial kinetic energy,
$l_{0}^{*}=[2{\overline{u^{2}}}(t_{0})/{\overline{\omega^{2}}}(t_{0})]^{1/2}$
is the initial integral length, and $\nu$ is the kinematic viscosity, as
presented in figure 7. The numbers of grid points are
$N_{x}=N_{y}=\\{128,256,512\\}$ for the covered initial Reynolds numbers
$Re_{0}=\\{80.4,177,442\\}$, respectively. For the input and output attributes
to the present AE, the vorticity field $\omega$ is utilized. Since the size of
input data must be consistent over the covered Reynolds numbers to feed into
the AE, the vorticity data generated at $Re_{0}=80.4$ and 442 are interpolated
to $256^{2}$ size when we handle with the AE. In addition, we also use the
instantaneous Taylor Reynolds number
$Re_{\lambda}(t)=u^{\\#}(t)\lambda(t)/\nu$, where $u^{\\#}(t)$ is the spatial
root-mean-square value for velocity of an instantaneous field and $\lambda(t)$
is the Taylor length scale of the instantaneous field, as the second scalar
input, as shown in the green portion of figure 1. For training the present AE,
we use 1000 snapshots for each initial Reynolds number.

##  III Results and Discussion

###  III.1 Example 1: scalar input-aided convolutional neural networks

Figure 8:  Dependence of $C_{D}$ and $C_{L}$ estimations on input scalar
placements for a flow over a flat plate. Comparison of the $L_{2}$ error norm
among flow types (right side) and input placements (left side) with
$\\{Re_{c},\alpha\\}$ inputs for $(a)$ $C_{D}$ and $(b)$ $C_{L}$. $(c)$ Time
traces of estimated coefficients. Figure 9: Dependence of the $L_{2}$ error
norm of force coefficient estimations for a flow over a flat plate with
$\\{Re_{c},\alpha\\}=\\{1000,70^{\circ}\\}$ on the number of training
snapshots. The standard deviation over the cross validation is also shown as
the gray line.

We first assess the influence on scalar input placements for the CNN-MLP
model. As stated in section II.1.2, our consideration is the estimation of
drag $C_{D}$ and lift $C_{L}$ coefficients of the flow over a flat plate and
two side-by-side cylinders. As already presented in figure 3, we investigate
four input placements for feeding input scalar values, referred to as cases 1,
2, 3 and 4. In the present formulation, the dimension of scalar inputs are
expanded to concatenate with convolutional layers with cases 1 and 2, while
they are directly connected to the MLP layers for cases 3 and 4. With both
flow examples, a single machine learning model handles all types of flows. For
instance, a single model is trained with four types of flows (combination
among $Re_{c}=\\{100,1000\\}$ and $\alpha=\\{35^{\circ},70^{\circ}\\}$) for
the flat plate wake, while other one is trained with eighteen types of flows
(combination among $r=\\{1.00,1.15,1.30\\}$ and
$g=\\{0.5,0.7,1.0,1.5,2.0,2.5\\}$) for the two side-by-side cylinders example.

The first example for the present analysis is the flat plate wake. The $L_{2}$
error norm $\epsilon=||{\bm{q}}_{\rm Pred}-{\bm{q}}_{\rm
True}||_{2}/||{\bm{q}}_{\rm True}||_{2}$ of the estimated coefficients are
summarized in figures 8$(a)$ and $(b)$. The left portion shows the
relationship among the covered flow types and the error, while the right
counterpart compares the cases of input placements. The basic trend here is
that it is tougher to estimate the coefficients at the higher $Re_{c}$ and
larger $\alpha$, although the highest error level can even achieve the $L_{2}$
error norm of 0.1. The $L_{2}$ error norms of $C_{D}$ and $C_{L}$ are almost
at same level in this case. It is striking that the cases with scalar inputs
at the upperstream layers tend to show the better estimation than those with
the scalar inputs at the downstream layers. This is likely because feeding the
scalar inputs from the upperstream layer promotes the model’s robustness for
test data by merging biases and nonlinear activation functions inside machine
learning model. This trend can also be observed from the time series of
estimated coefficients as shown in figure 8$(c)$. The result of case 4 (scalar
inputs at downstream layer) shows slight disagreement with the reference data
especially for the flow at $(Re_{c},\alpha)=(1000,70^{\circ})$.

We also investigate the influence on the amount of training snapshot for the
input placements of supplemental scalar values considering the flat plate
wake. We here consider six cases, $n_{\rm
snapshot}=\\{50,100,200,300,500,1000\\}$. The averaged $L_{2}$ error norms of
$C_{D}$ and $C_{L}$ values for a flat plat wake with
$\\{Re_{c},\alpha\\}=\\{1000,70^{\circ}\\}$ are shown in figure 9. We also
present the error variance over the cases 1 to 4 for each number of snapshots
as the gray line. As the number of training snapshot increases, the overall
error decreases as well as the variance among the input placement cases.
Moreover, what is notable here is that the dependence on the input placements
of scalar values can be suppressed by using a large number of training
snapshot. For the large number of training snapshots, we can also see the
better estimation by adding scalar inputs at the earlier layers.

Next, we consider a flow over two side-be-side cylinders to investigate the
influence on the scalar input placements. As mentioned above, we utilize the
diameter ratio $r$ between cylinders I and II and the distance between two
cylinders $g$ as the additional inputs such that three cases for
$r=\\{1.00,1.15,1.30\\}$, and six cases for $g=\\{0.5,0.7,1.0,1.5,2.0,2.5\\}$.
In this example, a single machine learning model trained with 18 cases
estimates drag and lift coefficients for both cylinders
${\bm{y}}=(C_{D,1},C_{D,2},C_{L,1},C_{D,2})\in{\mathbb{R}}^{4}$ from the
vorticity field $\omega$.

Figure 10:  Dependence of $C_{D}$ and $C_{L}$ estimations on input scalar
placements for a flow over two side-by-side cylinders. Comparison of the
$L_{2}$ error norm among flow types for each coefficient; $(a)$ $C_{D}$ of
cylinder I, $(b)$ $C_{D}$ of cylinder II, $(c)$ $C_{L}$ of cylinder I, and
$(d)$ $C_{L}$ of cylinder II. Case 5 denotes the model without the additional
scalar value input. Time traces of estimated coefficients with $g=1.0$ for
each coefficient are also shown. Figure 11: Robustness against noisy input for
a flow over two side-by-side cylinders. We assess with the ensemble $L_{2}$
error norm over $C_{D}$ and $C_{L}$ values on both cylinders.

The $L_{2}$ error norms of the estimated four coefficients, $C_{D}$ and
$C_{L}$ for both cylinders, are summarized in figure 10. The error rate
becomes higher for the flows at $g$ lower than $1.0$, corresponding to chaotic
flows, contrary to the flows at higher $g$, i.e., periodic and quasi-periodic
flows. In contrast to the example of flat plate wake, the $L_{2}$ error norms
of $C_{L}$ tend to be higher than that of $C_{D}$. This is likely because of
the difference in magnitude — the value of $C_{L}$ has the order of ${\cal
O}(10^{0})$ while that of $C_{D}$ is ${\cal O}(10^{-1})$. More concretely, a
tiny error, e.g., ${\cal O}(10^{-2})$, cannot be assessed fairly for $C_{D}$
and $C_{L}$ since its relative magnitude for them should be different with
each other. To avoid this issue, standardization or normalization of the data
can be a good candidate [61]. Noteworthy here is that the dependence of the
CNN performance on the input placements of scalar values are lower than that
with the flat plate examples. This suggests that users should care the input
placement depending on data that they handle.

The robustness against noisy input and its dependence on input placements are
also examined with the two side-by-side cylinders example, as shown in figure
11. A Gaussian noise is added to the input vorticity field. We consider three
magnitudes of the noise whose signal to noise ratio (SNR) is set to ${\rm
SNR}^{-1}=\\{0.012,0.11,0.31\\}$, where ${\rm SNR}=\sigma^{2}_{\rm
Data}/\sigma^{2}_{\rm Noise}$. In figure 11, the bars indicate the $L_{2}$
error norm averaged by the distance factor $g$ over four coefficients. As
mentioned above, there is no clear difference among cases without noise,
presented as the bright-green colorbars. On the other hand, the cases with the
strong magnitude of noise, i.e., blue colored bars, show notable differences
over the input placements. Case 1 (i.e., inputs the scalars at the 1st
convolutional layer) reports the least $L_{2}$ error norm, which is the same
trend as that in the flat plate wake example. This also supports our
observation above that the scalar inputs from the upperstream layer enables
the model to be robust by merging biases and nonlinear activation functions
inside models.

###  III.2 Example 2: investigation of CNN parameters for fluid flow analyses
with autoencoder

In this section, let us investigate the influence on CNN parameters for fluid
flow analyses using CNN-AE. Here are the parameters we focus in this study;

  1. 1.

Size of the filter for the convolutional operation (section III.2.1).

  2. 2.

Compression rate of CNN-AE (section III.2.2).

  3. 3.

Padding operation for the convolutional operation (section III.2.3).

  4. 4.

Input placements of supplemental scalar value (section III.2.4).

  5. 5.

Methods to reduce/expand the dimension of the data (section III.2.5).

    1. (a)

Up-sampling vs transposed convolution (dimension expansion)

    2. (b)

Max vs average pooling (dimension reduction)

Hereafter, we use a vorticity field of two-dimensional decaying homogeneous
isotropic turbulence as the input and output attribute of the AE. Since it
contains various scales of structures and also the data is generated with the
biperiodical boundary condition, it can be regarded as a suitable example for
us to establish a benchmark investigation of CNN parameters, such as filter
size, padding operation, and etc.

####  III.2.1 Size of the filter for the convolutional operation

One of the biggest feature of CNN is the filter operation which can extract
key spatial structures of the input data to establish a relationship between
input and output data. Because the size of filter directly relates to the
number of weights inside CNN, we can easily expect that an appropriate choice
for the filter size can lead to an improvement in estimation ability. It is no
exception for fluid mechanics applications — in fact, Fukami et al. [10]
utilized a customized CNN, which contains multiple sizes of filters, for fluid
flow super-resolution analysis and reported its great ability in handling
turbulence compared to a CNN with single type of filter size. Otherwise, Lee
and You [29] also capitalized on a CNN which includes multiple size of filters
for the construction of surrogate modeling for high-fidelity simulation.
Despite that a size of the filter plays a crucial role in the CNN-based
modeling, the dependence on the filter size for fluid flow data has not been
investigated clearly yet. Here, we consider five sizes of filter
$H=\\{3,5,7,9,11\\}$ and investigate its influence on the ability of CNN-AE.
For this investigation, the training data is generated at $Re_{0}=80.4$, and
the number of latent vector is set to be 1024.

Figure 12: $(a)$ Dependence of the $L_{2}$ error norm on the size of the
filter size of CNN-AE. Time series of $L_{2}$ error norm and integral length
scale are shown. $(b)$ Kinetic energy spectrum at $t=3.18$ and $t=4.79$ from
the reference DNS data.

The time series of $L_{2}$ error norm of the reconstructed field is shown in
figure 12$(a)$. The model with the larger size of the filters shows lower
error level over the time. This is simply because the number of weight
increases for the larger size of the filters and ease the problem setting for
the machine learning model. Along the time series of $L_{2}$ error norms, we
also plot the integral length scale $L=(\pi/2u^{\prime})\int E(k)/kdk$, where
$u^{\prime}$ is the root-mean-squared velocity, $E$ is the total kinetic
energy, and $k$ is the wavenumber, so as to check the relationship between the
filter size and the length scale of flows. As shown, the curve of $L_{2}$
error norm follows along the curve of integral length scale $L$ up to $t=4$.
After passing $t=4$, the $L_{2}$ error norm starts to decrease and reports
lowest error at approximately $t=4.3$ contrary to the rapid rise of $L$. The
reason for this behavior can be speculated from the perspective of energy
spectrum. The instantaneous energy spectrum at $t=3.18$ and $4.79$ from the
reference DNS are shown in figure 12$(b)$. According to the decaying nature of
the flow, the high frequency components ($6\times 10^{1}\leq k\leq 3\times
10^{2}$) decrease over time [60, 62]. Since a supervised model generally tends
to estimate lower frequency components prior to higher frequency counterparts
[46, 47, 63], it implies that the decrease of higher frequency component
simply leads to the better estimation for the machine-learned model.

####  III.2.2 Compression rate of CNN-AE

Figure 13: Dependence of the $L_{2}$ error norm on the number of latent modes
and the number of weights for CNN-AE with a two-dimensional decaying
turbulence.

We then investigate the dependence of the compression rate of the CNN-AE on
the reconstruction ability. Generally, the reconstruction performance of AE
becomes lower as the number of latent modes decreases [47]. Since this is the
well-known fact and intuitive for us, we also assess this point with regard to
the number of weights inside the CNN-AE, in additon to the number of latent
modes, as shown in figure 13. We here present the relationship between the
$L_{2}$ error norm of the reconstructed flow and the number of weights among
four cases of latent modes $n_{r}=\\{64,128,256,1024\\}$. Since we can
increase or decrease various parameters inside CNN including the size of the
filter and the number of the layers, we can have the variation for the number
of weights over the same numbers of latent modes. Here, the number of weights
are adjusted by changing only the size of the filters $H$. For the models with
the number of weights less than $10^{5}$, the error decreases as the number of
weights increases, as expected. However, the trend varies for the models with
number of weights larger than $2\times 10^{5}$. This is likely caused by the
fact that an optimization process for weights inside a neural network becomes
unstable when the number of weights is massive, which is well known as “curse
of dimensionality.”[64] This result encourages users to be careful on setting
the number of weights to avoid the unstable learning process, in addition to
the number of latent modes.

####  III.2.3 Padding operation for the convolutional operation

Figure 14: $(a)$ Padding operation covered in this study. $(b)$ Relationship
between the padding operations and the $L_{2}$ error norm for various filter
sizes.

As introduced above, we have recently been able to see various studies of CNN
and fluid flows. But in many case, there is a suspicious setting of CNN for
fluid flow analyses — a zero padding operation at convolutional layers.
Fundamentally, the convolutional operation usually reduces the data size
$L_{w}\times L_{h}$ to $(L_{w}-2\lfloor H/2\rfloor)\times(L_{h}-2\lfloor
H/2\rfloor)$. However, we often do not want the data size to be reduced via
convolutional operations because the dimension reduction using pooling layers
is known as the better way to acquire the robustness for spatial sensitivity
and noise [65]. To avoid the dimensional reduction at the convolutional
layers, it is very common to add zero values around the data (called zero
padding), as shown in figure 14$(a)$. By giving additional values around an
image, the dimension of the data can be retained through the convolutional
operation. Although it is an well-used technique, it is questionable from the
perspective of boundary conditions, especially with numerically generated
data. For instance, it can be easily suspected whether we can use the zero
padding for the present two-dimensional decaying turbulence having the spatial
biperiodic condition, or not. Here, we compare three types of padding
operations, 1. zero padding, 2. replication padding, and 3. periodic padding.
As illustrated in figure 14$(a)$, the replication padding is a mirroring
operation which embeds symmetrical value regarding the boundary of the data.
This operation is also non-physical like zero padding, although the boundary
becomes smooth. The other covered operation is the periodic padding, which
faithfully follows the boundary condition of the present turbulence data.

We investigate these three operations with various sizes of the filter $H$
from $3$ to $11$, as presented in figure 14$(b)$. The clear difference is not
observed up to $H\leq 9$, while the error variance becomes significantly
unstable using the replication padding with $H=11$. This negative influence of
replication padding with the larger size of filters can be regarded as natural
because more nonphysical values are given, which suggests that these values
would have the effect for the convolutional operation inside a CNN. In
contrast, what is striking here is that models with zero padding shows the
same error level as models with periodic padding. This is likely because the
influence of inserting zero values near the boundaries is much less than that
with nonphysical values by the replication padding in equation 1. The optimal
choice of padding operation may differ among flows with different boundary
conditions, although the result here encourages us to utilize the zero padding
which reported almost as the same error level as the model with the periodic
padding.

####  III.2.4 Input placements of supplemental scalar value

As introduced above, the supplemental scalar values are often utilized to
improve the estimation ability of CNN for fluid flow analyses. Similarly to
the investigation for the influence on input placements of scalar values with
the CNN-MLP model in section III.1, let us also examine this point with the
CNN-AE for the decaying isotropic homogeneous turbulence. As briefly explained
in section II.1.2, we consider four cases of input placements for additional
scalar values, i.e, the 1st convolutional layer (case 1), the 4th
convolutional layer (case 2), the latent space (case 3) and the 16th
convolutional layer (case 4). As illustrated in figure 4, the scalar values
are first expanded with MLP and then reshaped into two-dimensional sectional
data to concatenate with outputs provided at the hidden layer inside the CNN-
AE. As supplemental scalar values, we utilize two types of the Reynolds
number, the initial Reynolds number $Re_{0}$ and the instantaneous Taylor
Reynolds number $Re_{\lambda}$. While the initial Reynolds number $Re_{0}$
represents the initial condition and governs decaying nature of flows, the
instantaneous Taylor Reynolds number $Re_{\lambda}$ contains information of an
instantaneous flow snapshot. In addition, we also consider the case with no
additional scalar values as case 5.

Figure 15:  Dependence of supplemental scalar value
$\\{Re_{0},Re_{\lambda}\\}$ on input placements for the CNN-AE with two-
dimensional decaying turbulence. $(a)$ Relationship between the $L_{2}$ error
norm and initial Reynolds number depending on input placement. $(b)$
Relationship between the $L_{2}$ error norm and input placement depending on
initial Reynolds number.

The dependence of the $L_{2}$ error norm of reconstructed flows on the input
scalar placements is summarized in figure 15. As we can clearly see, case 5
(without the scalar inputs) reports higher error than other cases. This
implies that the use of supplemental scalars also works nicely for low-
dimensionalization. For the cases with scalar inputs, case 1 (input at the
first layer) shows the best estimation among the covered cases, which is
analogous to the investigation with the CNN-MLP model above. On the other
hand, case 3 (input from the latent space) reports the worst estimation. This
is likely due to the difference of the number of weights given at the MLP part
for merging the Reynolds number input and the main part of CNN-AE. Summarizing
above, the results indicate that scalar inputs at upstream layers helps the
estimation for CNN-AE, as well as the CNN-MLP model, and additionally a
sufficient number of weights are required before merging the input scalars
with CNN-AE.

####  III.2.5 Dimensional reducing/expanding methods

Figure 16:  Relationship between the $L_{2}$ error norm $\epsilon$ and filter
size $H$ for $(a)$ dimension reduction methods (max and average pooling), and
$(b)$ dimension expansion methods (up-sampling and transposed convolution).

Finally, we investigate the effect of the dimensional reduction and expansion
methods for CNN performance. The relationship between the $L_{2}$ error norm
and the dimension reduction ways are summarized in figure 16$(a)$. We compare
the max and average poolings, as introduced in section II.1.1. As an example,
the number of latent variables is set to be 128 with five cases of filter size
$H=\\{3,5,7,9,11\\}$. As shown, the error with both methods are approximately
at the same level for every cases. This result indicates that the present AE
is not very sensitive to the pooling operations.

We also check the influence of dimensional expansion methods, as presented in
figure 16$(b)$. As introduced in section II.1.2 and figure 2, the significant
difference between the transposed convolution and the up-sampling is whether
there is the trainable weights (filter) or not. Therefore, the number of
weights inside a model $n_{\rm weight}$ increases when up-sampling layers are
replaced with transposed convolutional layers, as shown by the pink curve in
figure 16$(b)$.

As summarized here, the $L_{2}$ error norm of the models with up-sampling
operation gradually decreases as the filter size increases. This exactly
corresponds to the increase of the number of weights. In contrast, the
variance of the model with the transposed convolutional operation is
significantly unstable with $H\geq 9$ where the number of the weights reaches
approximately $3.0\times 10^{5}$. This is analogous to the observation in
section III.2.2, known as “curse of dimensionality.” However, the model with
the transposed convolution generally shows better performance with less amount
of weights. Depending on the considered flows and their model configuration,
users can choose the dimension expansion and reduction methods inside CNNs.

##  IV Conclusions

Figure 17:  Graphical conclusion of the present study.

We investigated the applicability of convolutional neural networks (CNNs),
which has been utilized as a powerful tool in scientific machine learning, for
fluid flow analyses. Especially, we focused on the CNN application with
additional scalar input information such that ${\bm{y}}={\cal
F}({\bm{x}},{\phi})$. Capitalizing on the canonical fluid flow data sets with
the perspectives on metamodeling for aerodynamics characteristics and low-
dimensionalization, we attempted to clear vague portions of CNN and fluid flow
analyses as graphically summarized in figure 17.

We first considered the CNN-MLP model with additional scalar inputs, by
considering flows around a flat plate and two side-by-side cylinders. The
model attempted to estimate drag and lift coefficients from a vorticity field.
The supplemental scalar values were added to the model at four different
placements and we investigated the influence on the input placements for the
scalar values. For both types of the flows, we observed better estimations of
the models with scalar inputs at earlier layers, especially under relatively
tough problem settings, i.e., less amount of training data and noisy input.
This is likely because a neural network is able to be robust for the test data
by merging supplemental information with biases and nonlinear activation
functions inside neural network.

We also examined the influence on CNN parameters considering an autoencoder
(AE)-based low-dimensionalization. Through the investigations for the filter
size and the compression rate dependence, we found that the larger number of
weights contained in the model can lead to the better estimation. However,
care should be taken for cases which contains too many weights since it causes
unstableness of training procedure of CNN. The dependence on the padding
operation for convolution was also examined. Despite that the zero padding is
a non-physical operation from the perspective on boundary condition of the
flow, it showed approximately same error level as the model with periodic
padding, whose operation faithfully follows the boundary condition. This
result encourages users to consider the use of zero padding depending on flows
dealt with. Similar to the investigation on the CNN-MLP models, we also
checked the influence of input placements for supplemental scalar values on
estimation ability of CNN-AE. Here, we also observed that the scalar input at
earlier layers leads to improve the estimation more compared to the other
cases. Finally, the choice of dimensional reduction/expansion methods was also
considered. We found that the machine learning model is robust to the
dimensional reduction methods, while it is sensitive to the expansion methods,
i.e., up-sampling and transposed convolution. Since the model with the
transposed convolution contains large amount of weights than that with up-
sampling layers, the model may fail into the “curse of dimensionality.” Based
on above, users can attempt to choose an appropriate combination of weights
depending on their problem setting and computational environment.

The present investigations enable us to notice some remaining issues and
considerable extensions of CNN and fluid flow analyses. One of them is
applications of CNN to unstructured mesh fluid flow data. As expressed in
section II.1.1, the basic principle inside CNN is taking filters for fluid
flow image handled on structured/discretized grids arranged with a uniform
manner. However, as readers have already noticed, we often encounter
unstructured data for several fluid flow analyses, e.g., flows around an
airfoil at practical Reynolds numbers. Unfortunately, the conventional CNN
used through this study cannot be applied directly these data on unstructured
mesh. To overcome this issue, several ideas have recently been proposed, e.g.,
graph convolutional neural networks [66, 67], PointNet [68], PhyGeoNet [69],
and Voronoi-tessellation aided CNN model [70]. We can expect that the present
knowledge and the ways for analyzing CNN parameters with fluid flows can be
extended to the aforementioned models. Otherwise, readers’ interest for
practical manner must arrive at the perspective on interpretability of results
provided by CNNs. For real-world applications, we often desire grounds of
estimation by a model (not only machine learning). Actually, some studies have
tackled this point from various views, e.g., relationship between machine-
learned results and vortical motion [43], uncertainty quantification [71], and
visualization inside CNNs [17]. Although it is still on-going area of fluid
mechanics and CNN, these trends may be a good direction toward practical
applications.

## Acknowledgements

Masaki Morimoto, Kai Fukami, and Koji Fukagata were supported by JSPS KAKENHI
Grant Number 18H03758. Masaki Morimoto and Kai Fukami thank Mr. Taichi
Nakamura (Keio University) for insightful discussions. Aditya G. Nair
acknowledges the support by UNR VP Research Startup PG19679.

## Data Availability

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## References

## References

  * Fukami, Fukagata, and Taira [2020] K. Fukami, K. Fukagata, and K. Taira, “Assessment of supervised machine learning for fluid flows,” Theor. Comput. Fluid Dyn. 34, 497–519 (2020).
  * Duraisamy, Iaccarino, and Xiao [2019] K. Duraisamy, G. Iaccarino, and H. Xiao, “Turbulence modeling in the age of data,” Annu. Rev. Fluid. Mech. 51, 357–377 (2019).
  * Brenner, Eldredge, and Freund [2019] M. P. Brenner, J. D. Eldredge, and J. B. Freund, “Perspective on machine learning for advancing fluid mechanics,” Phys. Rev. Fluids 4, (100501) (2019).
  * Brunton, Hemati, and Taira [2020] S. L. Brunton, M. S. Hemati, and K. Taira, “Special issue on machine learning and data-driven methods in fluid dynamics,” Theor. Comput. Fluid Dyn. 34, 333–337 (2020).
  * Lapeyre  _et al._ [2019] C. J. Lapeyre, A. Misdariis, N. Cazard, D. Veynante, and T. Poinsot, “Training convolutional neural networks to estimate turbulent sub-grid scale reaction rates,” Combust. Flame 203, 255–264 (2019).
  * Rumelhart, Hinton, and Williams [1986] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagation errors,” Nature 322, 533—536 (1986).
  * Pawar  _et al._ [2020a] S. Pawar, O. San, A. Rasheed, and P. Vedula, “A priori analysis on deep learning of subgrid-scale parameterizations for kraichnan turbulence,” Theor. Comput. Fluid Dyn. , 429–455 (2020a).
  * Thuerey  _et al._ [2020] N. Thuerey, K. Weißenow, L. Prantl, and X. Hu, “Deep learning methods for Reynolds-averaged Navier–Stokes simulations of airfoil flows,” AIAA J. 58, 25–36 (2020).
  * Font  _et al._ [2020] B. Font, G. D. Weymouth, V.-T. Nguyen, and O. R. Tutty, “Deep learning the spanwise-averaged navier–stokes equations,” arXiv:2008.07528 (2020).
  * Fukami, Fukagata, and Taira [2019] K. Fukami, K. Fukagata, and K. Taira, “Super-resolution reconstruction of turbulent flows with machine learning,” J. Fluid Mech. 870, 106–120 (2019).
  * Liu  _et al._ [2020a] B. Liu, J. Tang, H. Huang, and X.-Y. Lu, “Deep learning methods for super-resolution reconstruction of turbulent flows,” Phys. Fluids 32, 025105 (2020a).
  * Kim  _et al._ [2020] H. Kim, J. Kim, S. Won, and C. Lee, “Unsupervised deep learning for super-resolution reconstruction of turbulence,” arXiv:2007.15324 (2020).
  * Deng  _et al._ [2019] Z. Deng, C. He, Y. Liu, and K. C. Kim, “Super-resolution reconstruction of turbulent velocity fields using a generative adversarial network-based artificial intelligence framework,” Phys. Fluids 31, 125111 (2019).
  * Morimoto, Fukami, and Fukagata [2020] M. Morimoto, K. Fukami, and K. Fukagata, “Experimental velocity data estimation for imperfect particle images using machine learning,” arXiv:2005.00756 (2020).
  * Cai  _et al._ [2019] S. Cai, S. Zhou, C. Xu, and Q. Gao, “Dense motion estimation of particle images via a convolutional neural network,” Exp. Fluids 60, 60–73 (2019).
  * Salehipour and Peltier [2019] H. Salehipour and W. R. Peltier, “Deep learning of mixing by two ‘atoms’ of stratified turbulence,” J. Fluid Mech. 861, R4 (2019).
  * Morimoto  _et al._ [2020] M. Morimoto, K. Fukami, K. Zhang, and K. Fukagata, “Generalization techniques of neural networks for fluid flow estimation,” arXiv:2011.11911 (2020).
  * Murata, Fukami, and Fukagata [2020] T. Murata, K. Fukami, and K. Fukagata, “Nonlinear mode decomposition with convolutional neural networks for fluid dynamics,” J. Fluid Mech. 882, A13 (2020).
  * Milano and Koumoutsakos [2002] M. Milano and P. Koumoutsakos, “Neural network modeling for near wall turbulent flow,” J. Comput. Phys. 182, 1–26 (2002).
  * Lusch, Kutz, and Brunton [2018] B. Lusch, J. N. Kutz, and S. L. Brunton, “Deep learning for universal linear embeddings of nonlinear dynamics,” Nature Commun. 9 (2018).
  * Fukami, Nakamura, and Fukagata [2020] K. Fukami, T. Nakamura, and K. Fukagata, “Convolutional neural network based hierarchical autoencoder for nonlinear mode decomposition of fluid field data,” Phys. Fluids 32, 095110 (2020).
  * Hasegawa  _et al._ [2019] K. Hasegawa, K. Fukami, T. Murata, and K. Fukagata, “Data-driven reduced order modeling of flows around two-dimensional bluff bodies of various shapes,” ASME-JSME-KSME Joint Fluids Engineering Conference, San Francisco, USA (2019).
  * Maulik, Lusch, and Balaprakash [2020] R. Maulik, B. Lusch, and P. Balaprakash, “Reduced-order modeling of advection-dominated systems with recurrent neural networks and convolutional autoencoders,” arXiv:2002.00470 (2020).
  * Fukami, Murata, and Fukagata [2020] K. Fukami, T. Murata, and K. Fukagata, “Sparse identification of nonlinear dynamics with low-dimensionalized flow representations,” arXiv:2010.12177 (2020).
  * Carlberg  _et al._ [2019] K. T. Carlberg, A. Jameson, M. J. Kochenderfer, J. Morton, L. Peng, and F. D. Witherden, “Recovering missing cfd data for high-order discretizations using deep neural networks and dynamics learning,” J. Comput. Phys. 395, 105 – 124 (2019).
  * Fukami  _et al._ [2019] K. Fukami, Y. Nabae, K. Kawai, and K. Fukagata, “Synthetic turbulent inflow generator using machine learning,” Phys. Rev. Fluids 4, 064603 (2019).
  * Nakamura  _et al._ [2020] T. Nakamura, K. Fukami, K. Hasegawa, Y. Nabae, and K. Fukagata, “Extension of CNN-LSTM based reduced order surrogate for minimal turbulent channel flow,” arXiv:2010.13351 (2020).
  * Liu  _et al._ [2020b] Y. Liu, C. Ponce, S. L. Brunton, and J. N. Kutz, “Multiresolution convolutional autoencoders,” arXiv:2004.04946 (2020b).
  * Lee and You [2019] S. Lee and D. You, “Data-driven prediction of unsteady flow fields over a circular cylinder using deep learning,” J. Fluid Mech. 879, 217–254 (2019).
  * Ling, Kurzawski, and Templeton [2016] J. Ling, A. Kurzawski, and J. Templeton, “Reynolds averaged turbulence modelling using deep neural networks with embedded invariance,” J. Fluid Mech. 807, 155–166 (2016).
  * Hasegawa  _et al._ [2020a] K. Hasegawa, K. Fukami, T. Murata, and K. Fukagata, “Machine-learning-based reduced-order modeling for unsteady flows around bluff bodies of various shapes,” Theor. Comput. Fluid Dyn. 34, 367–388 (2020a).
  * Hasegawa  _et al._ [2020b] K. Hasegawa, K. Fukami, T. Murata, and K. Fukagata, “CNN-LSTM based reduced order modeling of two-dimensional unsteady flows around a circular cylinder at different Reynolds numbers,” Fluid Dyn. Res. 52, 065501 (2020b).
  * Zhang, Sung, and Mavris [2018] Y. Zhang, W. J. Sung, and D. N. Mavris, “Application of convolutional neural network to predict airfoil lift coefficient,” in  _AIAA paper_ (2018) pp. 2018–1903.
  * Bhatnagar  _et al._ [2019] S. Bhatnagar, Y. Afshar, S. Pan, K. Duraisamy, and S. Kaushik, “Prediction of aerodynamic flow fields using convolutional neural networks,” Comput. Mech. 64, 525–545 (2019).
  * Kim and Lee [2020a] J. Kim and C. Lee, “Deep unsupervised learning of turbulence for inflow generation at various reynolds numbers,” J. Comput. Phys. 406, 109216 (2020a).
  * Xu and Duraisamy [2020] J. Xu and K. Duraisamy, “Multi-level convolutional autoencoder networks for parametric prediction of spatio-temporal dynamics,” Comput. Methods Appl. Mech. Eng. 372, 113379 (2020).
  * Zafar  _et al._ [2020] M. I. Zafar, H. Xiao, M. M. Choudhari, F. Li, C.-L. Chang, P. Paredes, and B. Venkatachari, “Convolutional neural network for transition modeling based on linear stability theory,” Phys. Rev. Fluids 5, 113903 (2020).
  * Pawar  _et al._ [2020b] S. Pawar, O. San, B. Aksoylu, A. Rasheed, and T. Kvamsdal, “Physics guided machine learning using simplified theories,” arxiv:2012.13343 (2020b).
  * LeCun  _et al._ [1998] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proc. IEEE 86, 2278–2324 (1998).
  * Simonyan and Zisserman [2014] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv:1409.1556 (2014).
  * Szegedy  _et al._ [2015] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in  _Proceedings of the IEEE conference on computer vision and pattern recognition_ (2015) pp. 1–9.
  * Hu, Shen, and Sun [2018] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in  _Proceedings of the IEEE conference on computer vision and pattern recognition_ (2018) pp. 7132–7141.
  * Kim and Lee [2020b] J. Kim and C. Lee, “Prediction of turbulent heat transfer using convolutional neural networks,” J. Fluid Mech. 882, A18 (2020b).
  * Huang, liu, and Cai [2019] J. Huang, H. liu, and W. Cai, “Online in situ prediction of 3-D flame evolution from its history 2-D projections via deep learning,” J. Fluid Mech. 875 (2019).
  * Gakhar, Koseff, and Ouellette [2020] S. Gakhar, J. R. Koseff, and N. T. Ouellette, “On the surface expression of bottom features in free-surface flow,” J. Fluid Mech. 900 (2020).
  * Fukami, Fukagata, and Taira [2021] K. Fukami, K. Fukagata, and K. Taira, “Machine learning based spatio-temporal super resolution reconstruction of turbulent flows,” J. Fluid Mech. 909 (2021).
  * Fukami  _et al._ [2020] K. Fukami, K. Hasegawa, T. Nakamura, M. Morimoto, and K. Fukagata, “Model order reduction with neural networks: Application to laminar and turbulent flows,” arXiv:2011.10277 (2020).
  * Nair and Hinton [2010] V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” In Proc. 27th International Conference on Machine Learning (2010).
  * Kingma and Ba [2014] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv:1412.6980 (2014).
  * Hinton and Salakhutdinov [2006] G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” Science 313, 504–507 (2006).
  * Peng  _et al._ [2020] J. Peng, X. Liu, N. Aubry, Z. Chen, and W. T. Wu, “Data-driven modeling of geometry-adaptive steady heat transfer based on convolutional neural networks: Heat conduction,” arxiv:2010.03854 (2020).
  * Dosovitskiy  _et al._ [2019] A. Dosovitskiy, P. Fischer, J. T. Springenberg, M. Riedmiller, and T. Brox, “Discriminative unsupervised feature learning with exemplar convolutional neural net- works,” IEEE Transactions on Pattern Analysis and Machine Intelligence 38 (2019).
  * Dumoulin and Visin [2016] V. Dumoulin and F. Visin, “A guide to convolution arithmetic for deep learning,” arXiv:1603.07285 (2016).
  * LeCun, Bengio, and Hinton [2015] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature 521, 436–444 (2015).
  * Simonyan and Zisserman [2015] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in  _International Conference on Learning Representations_ (2015).
  * Omata and Shirayama [2019] N. Omata and S. Shirayama, “A novel method of low-dimensional representation for temporal behavior of flow fields using deep autoencoder,” AIP Adv. 9, 015006 (2019).
  * Taira and Colonius [2007] K. Taira and T. Colonius, “The immersed boundary method: A projection approach,” J. Comput. Phys. 225, 2118–2137 (2007).
  * Colonius and Taira [2008] T. Colonius and K. Taira, “A fast immersed boundary method using a nullspace approach and multi-domain far-field boundary conditions,” Comput. Methods Appl. Mech. Eng. 197, 2131–2146 (2008).
  * Weller  _et al._ [1998] H. G. Weller, G. Tabor, H. Jasak, and C. Fureby, “A tensorial approach to computational continuum mechanics using object-oriented techniques,” Comput. Phys. 12, 620–631 (1998).
  * Taira, Nair, and Brunton [2016] K. Taira, A. G. Nair, and S. L. Brunton, “Network structure of two-dimensional decaying isotropic turbulence,” J. Fluid Mech. 795, R2 (2016).
  * Shanker, Hu, and Hung [1996] M. Shanker, M. Y. Hu, and M. S. Hung, “Effect of data standardization on neural network training,” Omega 24, 385–397 (1996).
  * Davidson [2015] P. Davidson,  _Turbulence: An Introduction for Scientists and Engineers_ (Oxford University Press, 2015).
  * Scherl  _et al._ [2020] I. Scherl, B. Strom, J. K. Shang, O. Williams, B. L. Polagye, and S. L. Brunton, “Robust principal component analysis for modal decomposition of corrupt fluid flows,” Phys. Rev. Fluids 5, 054401 (2020).
  * Domingos [2012] P. Domingos, “A few useful things to know about machine learning,” Communications of the ACM 55, 78–87 (2012).
  * He  _et al._ [2016] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” In Proceedings of Computer Vision and Pattern Recognition , 770–778 (2016).
  * Ogoke  _et al._ [2020] F. Ogoke, K. Meidani, A. Hashemi, and A. B. Farimani, “Graph convolutional neural networks for body force prediction,” arXiv:2012.02232 (2020).
  * Tencer and Potter [2020] J. Tencer and K. Potter, “Enabling nonlinear manifold projection reduced-order models by extending convolutional neural networks to unstructured data,” arXiv:2006.06154 (2020).
  * Kashefi, Rempe, and Guibas [2020] A. Kashefi, D. Rempe, and L. J. Guibas, “A point-cloud deep learning framework for prediction of fluid flow fields on irregular geometries,” arXiv:2010.09469 (2020).
  * H.Gao, Sun, and Wang [2020] H.Gao, L. Sun, and J.-X. Wang, “PhyGeoNet: Physics-informed geometry-adaptive convolutional neural networks for solving parameterized steady-state pdes on irregular domain,” J. Comput. Phys. , 110079 (2020).
  * Fukami  _et al._ [2021] K. Fukami, R. Maulik, N. Ramachandra, K. Fukagata, and K. Taira, “Global field reconstruction from sparse sensors with voronoi tessellation-assisted deep learning,” arXiv:2101.00554 (2021).
  * Maulik  _et al._ [2020] R. Maulik, K. Fukami, N. Ramachandra, K. Fukagata, and K. Taira, “Probabilistic neural networks for fluid flow surrogate modeling and data recovery,” Phys. Rev. Fluids 5, 104401 (2020).