# Delving into the Scale Variance Problem in Object Detection

Junliang Chen, Xiaodong Zhao and Linlin Shen *: Corresponding author Computer
Vision Institute, School of Computer Science and Software Engineering,
Shenzhen University, China, and Shenzhen Institute of Artificial Intelligence
of Robotics of Society, Shenzhen, China, and Guangdong Key Laboratory of
Intelligent Information Processing, Shenzhen University, Shenzhen 518060,
China
{chenjunliang2016<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Object detection has made substantial progress in the last decade, due to the
capability of convolution in extracting local context of objects. However, the
scales of objects are diverse and current convolution can only process single-
scale input. The capability of traditional convolution with a fixed receptive
field in dealing with such a scale variance problem, is thus limited. Multi-
scale feature representation has been proven to be an effective way to
mitigate the scale variance problem. Recent researches mainly adopt partial
connection with certain scales, or aggregate features from all scales and
focus on the global information across the scales. However, the information
across spatial and depth dimensions is ignored. Inspired by this, we propose
the multi-scale convolution (MSConv) to handle this problem. Taking into
consideration scale, spatial and depth information at the same time, MSConv is
able to process multi-scale input more comprehensively. MSConv is effective
and computationally efficient, with only a small increase of computational
cost. For most of the single-stage object detectors, replacing the traditional
convolutions with MSConvs in the detection head can bring more than 2.5%
improvement in AP (on COCO 2017 dataset), with only 3% increase of FLOPs.
MSConv is also flexible and effective for two-stage object detectors. When
extended to the mainstream two-stage object detectors, MSConv can bring up to
3.0% improvement in AP. Our best model under single-scale testing achieves
48.9% AP on COCO 2017 test-dev split, which surpasses many state-of-the-art
methods.

###### Index Terms:

object detection, scale variance, multi-scale convolution

Figure 1: Performance on COCO val-2017 split of multi-scale convolution in
various single-stage detectors including anchor-based FreeAnchor [1] and
anchor-free RepPoints [2]. Two-stage detectors like Faster R-CNN (Faster) [3],
Mask R-CNN (Mask) [4], and Cascade R-CNN (Cascade) [5] are provided for
reference. Our MSConv can significantly improve the APs of different
detectors. All the models are trained on ResNet-50 [6] backbone with a
resolution of $640\times 640$.

##  I Introduction

Object detection is a fundamental challenge in computer vision, which contains
two sub tasks: location and classification. For any input image, the object
detector is supposed to find out the position and category of all the objects
available in the image. Unlike object recognition which only requires
classification information, in object detection, we need to obtain the
features containing accurate scale information of objects to locate them. The
scale of different objects may vary in a wide range, making it difficult to
represent large and small objects using the same scale. To alleviate the scale
variance problem, researchers have made a lot of attempt.

In the earlier researches, image pyramid is an effective idea to solve the
scale variances. The input image is resized to different resolutions,
resulting in an image pyramid. Hand-engineered features are then extracted on
the image pyramid. In image recognition, with the development of convolutional
neuron networks (CNN), hand-engineered features are gradually replaced by the
features computed by CNN. CNN is more robust to the scale variance and
translation, thus improves the recognition performance based on a single
image. Many recent top methods in the ImageNet [7] and COCO [8] detection
challenges have utilized multi-scale testing to extract features from an image
pyramid using CNN. Each level of the image pyramid is passed through the same
CNN, to generate features at different scales. The idea to use CNN to extract
features from an image pyramid gives a multi-scale feature representation of
the original image. As all the levels of the image pyramid are passed through
the whole network, the relative features are semantically strong, including
the finest features with the highest resolution.

Image pyramid is a simple and good solution to represent the image at
different scales. However, it is time-consuming, due to repeatedly network
forward on different level of the image pyramid. To alleviate this problem,
researchers have tried to make full use of the inherent characteristic of deep
CNNs. Modern deep CNNs usually consist of many layers, including down-sample
layers which generate features with decreased resolutions, such as poolings
and strided convolutions. Given an input image, the deep CNN generates
features with different scales, i.e.,a feature hierarchy. As the depth goes
deeper, the corresponding layer becomes semantically stronger. Therefore, the
last layer of the network is representative and widely used for object
detection (e.g.,YOLOv1 [9]).

However, the semantically strongest features from the CNN have the lowest
resolution, which reduces the spatial representation capability. Besides, the
intrinsic feature hierarchy brings large semantic gap between the highest and
lowest level. Thus, to some extent, it is limited to only use the features
with strongest semantic for object classification and location. To solve this
issue, the Single Shot MultiBox Detector (SSD) [10] first attempts to utilize
the existing feature hierarchy generated by the CNN. SSD takes the pyramidal
features as inputs, and conducts prediction independently on each level of the
pyramid. In consideration of realtime processing, SSD builds up detection
heads from the high-level layers (e.g.,conv4_3 layer of VGG-16 network [11])
with low resolutions, which can not represent small objects well.

To make better use of the diverse semantics of features from different scales.
Feature Pyramid Networks (FPN) explores the connection patterns between the
multiple layers. FPN proposes the lateral connection of two adjacent layers in
a top-down manner, and takes advantage of the representation capability of the
high-resolution layers (e.g.,the last layer of the conv2_x block of ResNets
[6]). FPN gives an effective solution to explore the characteristic and
advantage of the feature pyramid. Nevertheless, FPN holds an information flow
from top to down, so the features at high levels are short of semantic
available at low-level ones. The perspective of FPN inspires the follow-up
researches on the exploration of building up better architectures to deal with
multi-scale features. [12, 13] introduce extra but limited information from
other scales beyond the top-down path of FPN. To a certain degree, the above
methods mitigate the scale variance problem existing in FPN, but ignore the
channel and spatial semantic differences between the multi-scale features.

Inspired by the above researches, we propose the multi-scale convolution
(MSConv) to effectively solve the mentioned problem. MSConv is an extension of
traditional convolution to process multi-scale input. MSConv is
computationally efficient and effective to improve the detection performance
of the object detectors.

In this paper, we mainly make the following contributions.

  * 1.

We propose the multi-scale convolution (MSConv), to extend the traditional
convolution to accept multi-scale input. MSConv can deal with the multi-scale
features across scale, channel and spatial dimension at the same time.

  * 2.

By replacing the convolutions with MSConvs in the detection head, mainstream
single-stage object detectors can get more than 2.5% improvement in AP. Our
best model based on FreeAnchor [1] detector achieves 48.9% AP on COCO test-dev
under single-scale testing, surpassing many state-of-the-art methods.

  * 3.

The proposed MSConv is computationally efficient, i.e.,only a small increase
of computation cost is required.

##  II Related Works

###  II-A Object Detectors

State-of-the-art object detection methods can usually be divided into two
categories: single-stage detectors and two-stage ones.

Two-stage Detectors. To locate the objects with different scales, R-CNN
systems [14, 15, 3, 5, 16] first generate region proposals at different
scales, then extract the features within the region proposals for further
classification or regression. Though the region proposals are at different
scales, the extracted features are resized to the same spatial size
(e.g.,$7\times 7$) using ROI pooling [15] or ROIAlign [4] resizing operation.
However, the scale variance problem still exists. The features extracted by
the region proposals are from objects with different scales, containing the
rich information on position and category of the objects. The larger objects
with larger area have more spatial information than smaller ones. The resizing
operation may bring inequality information loss for objects at different
scales. As the features of any object are resized to the same spatial size,
the information loss of larger objects is greater than that of the small ones.
Therefore, there still exists scale variance problem in two-stage detectors.

Single-stage Detectors. Given multi-scale input features, single-stage
detectors (e.g.,[10]) usually directly generate predictions. Anchor-based
RetinaNet [17] has made great progress in solving scale variance problem.
While dense anchors with different scales and aspect ratios are used by
RetinaNet to cover different objects, FPN [18] is used to enhance the features
with stronger semantic from higher levels. RetinaNet has better performance
than Faster R-CNN [3] and comparable performance with most two-stage
detectors. Anchor-free ones [19, 20] adopt per-pixel prediction of location
and classification. To avoid object ambiguity problem, each pixel is only
related to single object. Objects will be assigned to the corresponding levels
according to their scales. The larger objects are assigned to higher levels
for their larger receptive fields, the smaller objects are assigned to lower
levels for their finer spatial features. Therefore, each level is in charge of
prediction of objects at similar scales. However, this fixed assignment
strategy only associates any object with a certain scale, which is limited for
object represent. It may be better to predict the object using features from
more scales (e.g.,features that are semantically stronger or spatially finer),
instead of the fixed ones.

###  II-B Methods Dealing with Scale Variance Problem

Recently, many researchers are exploring methods to overcome the scale
variance problem. We can simply divide them into two categories: methods
fusing features from partial scales (partial connections for short) and
methods fusing features from all scales (full connections for short).

Partial Connections. FPN [18] first proposes lateral connection to merge
features from adjacent scales in a top-down path. PANet [12] brings an
additional bottom-up path on the basis of FPN to supplement the missing finer
spatial information from smaller scales. NAS-FPN [13] introduces neural
architecture search (NAS) to discover a better scheme that merges cross-scale
features from any scales instead of only adjacent ones. These methods enhance
the original features with semantics from other scales, but can only obtain
information from limited ones.

Full Connections. Beyond obtaining information from certain scales, many
researches are exploring methods aggregating features from all scales. Kong et
al.,[21] gather features from all levels to a medium level followed by
concatenation along channel dimension. For each level, they use global
attention and local configuration to enhance and refine the combined features.
The refined features are finally resized to the corresponding level and then
element-wisely summed up with the original input of this level. Libra R-CNN
[22] first gathers features from all levels to a medium level and does
element-wise summation. After that, a non-local [23] module is applied to
enhance the merged features. The enhanced features are then scattered to each
level and element-wisely summed up with the original input. The above methods
obtain features from all scales by gathering them to a medium level and then
merge the features. However, a feature representation at medium scale is
improper to describe objects at other scales. Therefore, the generated
features can fit well the scale of this level, but may not fit well for other
scales.

Besides, most of the above methods adopt a simple way to merge the features
(often element-wisely summation), which lacks nonlinearity and gives the
features from different scales the same weight. Nevertheless, we should let
the network learn proper weights to combine the features from different
scales.

##  III Multi-Scale Convolution

###  III-A Overview

In this section, we give the overview of multi-scale convolution. Multi-scale
convolution consists of two steps: feature gathering and feature blending. Let
the input features from $L$ different levels be $X=\\{X^{1},\dots,X^{L}\\}$.

In the feature gathering step, the multi-scale features will be gathered to
each level. The output of feature gathering step $Q=\\{Q^{1},\dots,Q^{L}\\}$
is obtained in a gather-scatter manner:

$\Phi=Gather(X,l_{gl}),\ l_{gl}\in\\{1,\dots,L\\}$ (1)

$Q=Scatter(\Phi,\\{l\\}_{l=1}^{L})$ (2)

where $\Phi$ denotes gathering result of all levels, $l_{gl}$ denotes the
gathering level (set to 1 if not specified). Gather and Scatter denote the
gathering and scattering process, respectively.

In the feature blending step, at each level, the gathered features are passed
to two modules: scale alignment (SA) and context attention (CA), to further
blend the multi-scale and original features. The output
$O=\\{O^{1},\dots,O^{L}\\}$ after CA can be computed as:

$O=\\{CA(W_{m}\ast SA(Q^{l},X^{l}))\\}_{l=1}^{L}$ (3)

where $W_{m}$ denotes the weight of the $1\times 1$ convolution to make the
channel number of the output of SA and $X^{l}$ equal. $\ast$ denotes the
convolution operation.

The final output $Y=\\{Y^{1},\dots,Y^{L}\\}$ of MSConv can be computed as:

$Y=\\{W_{Y}\ast(O^{l}\oplus X^{l})\\}_{l=1}^{L}$ (4)

where $\oplus$ denotes element-wise summation, $W_{Y}$ denotes the weight of
the $3\times 3$ convolution to generate the final output.

Figure 2: Architecture of the multi-scale convolution.

###  III-B Detailed Architecture

Figure 2 shows the architecture of our multi-scale convolution. In the feature
gathering step, we first reduce the channels of the input features and resize
them to the lowest level. Then we concatenate the gathered multi-scale
features and scatter them to each level. In the feature blending step, the
features at each level are then passed to a shared block for further
processing. In the shared block, the multi-scale features with the original
input features are passed to a scale alignment module to align the spatial
scale of the multi-scale features. The scale-aligned features are then passed
to a $1\times 1$ convolution to merge the channels. The merged features are
then rescaled by the context attention module with attention across scale,
channel and spatial dimension. After that, a $3\times 3$ convolution is
applied in the element-wise summation of the merged features and the original
input to generate the final output for each level.

####  III-B1 Feature Gathering

(a) Full connection. (b) Gather-scatter connection.

Figure 3: Different connections of multi-scale features.

Before merging the multi-scale features from each scale, we should find a
proper way of multi-scale feature representation for each level. The best
solution is to gather features from all scales to each level, by full
connection, shown in Figure 3 (a). In this way, the multi-scale features turn
into the same spatial representation for each level. However, the full
connection manner introduces too many additional operations including
upsampling and downsampling, with a complexity of $\mathcal{O}(CL^{2})$. As an
alternative, we adopt a gather-scatter connection to approximate full
connection.

To reduce the computation cost, we first separately use a $1\times 1$
convolution to reduce the number of channels of each input to $C_{r}$
($C_{r}\leq C$). The output of the $l$-th level is denoted as
$D^{l}\in\mathbb{R}^{C_{r}\times H^{l}\times W^{l}}$, where $H^{l}$ and
$W^{l}$ denotes the resolution of the input at the $l$-th level. $C_{r}$ is
set to 64 in our experiment if not specified.

After that, to simultaneously process the multi-scale features, we gather all
the features to the same level $l_{gl}$, and concatenate them along the
channel dimension. $l_{gl}$ is set to 1 so that the gathered features can keep
the largest resolution to avoid information loss. The gathering process to
generate output $\Phi\in\mathbb{R}^{LC_{r}\times H^{l_{gl}}\times W^{l_{gl}}}$
is:

$E=\\{Resize(D^{l},(H^{l_{gl}},W^{l_{gl}}))\\}_{l=1}^{L}$ (5) $\Phi=Concat(E)$
(6)

where $Resize$ denotes the resizing function, $Concat$ denotes concatenation
along channels.

Then we generate the features for each level for further processing by scatter
$\Phi$ to each level through resizing. The scatter process to generate $Q$ is:

$Q=\\{Resize(\Phi,(H^{l},W^{l}))\\}_{l=1}^{L}$ (7)

In this way, for each level, the features from all scales obtain the same
spatial scale of the current level, so that the detector can get a proper
multi-scale feature representation, which can be processed across scale,
channel and spatial dimension. The complexity of our connection manner is
$\mathcal{O}(C_{r}L)<\mathcal{O}(CL^{2})$, which is far less than that of the
full connection manner.

####  III-B2 Feature Blending

After feature gathering, we blend the features for each level. We first apply
the scale alignment module to neutralize the spatial offset generated by the
pooling operation during feature preparation. Then we merge the multi-scale
features through a $1\times 1$ convolution and use the context attention
module to dynamically rescale the weight of each channel of the merged
features. The network can thus dynamically select the useful features and
suppress the useless features. At each level, the rescaled features are then
element-wisely summed up with the original input and passed to a $3\times 3$
convolution to generate the final output.

Figure 4: The scale alignment module. The rectangle represents a feature, in
which the number of its channels is shown.“k” denotes the kernel size.

Scale Alignment. The pooling operation we use during feature preparation has
translation invariance, and is not sensitive to position variation. Therefore,
each pixel of the multi-scale features after feature gathering for each level
has a spatial offset from the concatenated features $\Phi$. To deal with this
problem, we propose the scale alignment (SA) module to neutralize the spatial
offset. Figure 4 shows the architecture of SA module. Firstly, we use a
$k\times k$ convolution on the original single-scale input to generate the
deformable offset and mask on the basis of the current scale. The multi-scale
features with the offset and mask together are then passed to a $k\times k$
modulated deformable convolution [24] with groups=$L$, to generate the scale-
aligned features $M^{l}$ at level $l$ ($k$ is set to 1 if not specified). Then
a $1\times 1$ convolution is applied in the multi-scale features to merge the
channels. The merges features $M^{l}$ has the same channel number as $X^{l}$.

Figure 5: The context attention module.

Context Attention. The merged features $M^{l}$ contains different context both
in spatial and depth dimension. As the unnecessary context may bring influence
to the final feature representation, we should keep the useful features and
suppress the useless ones. In order to achieve this goal, we propose the
context attention (CA) module to rescale the features by an attention across
depth and spatial dimension. The architecture of CA module is shown in Figure
5. We first independently use a $3\times 3$ local average pooling (LAP) and a
global average pooling (GAP) to extract local features $L^{l}$ and global
features $G^{l}$:

$L^{l}=LAP(M^{l}),\ G^{l}=GAP(M^{l})$ (8)

Then a dependent $1\times 1$ convolution is separately applied to $L^{l}$ and
$G^{l}$ to extract features across channels:

$\mathcal{L}^{l}=W_{\mathcal{L}}\ast L^{l},\
\mathcal{G}^{l}=W_{\mathcal{G}}\ast G^{l}$ (9)

where $\mathcal{L}^{l}$ and $\mathcal{G}^{l}$ are the features generated from
$L^{l}$ and $G^{l}$, respectively. $W_{\mathcal{L}}$ and $W_{\mathcal{G}}$ are
the weights of the convolution of $L^{l}$ and $G^{l}$ shared across all
levels, respectively.

To generate the attention $S^{l}$ across depth (exactly across scale and
channel) and spatial dimension, we apply a $1\times 1$ convolution followed by
a sigmoid function on the element-wise summation of $\mathcal{L}^{l}$ and
$\mathcal{G}^{l}$:

$S^{l}=\sigma(W_{\sigma}\ast(\mathcal{L}^{l}\oplus\mathcal{G}^{l}))$ (10)

where $W_{\sigma}$ denotes the weight of the convolution before sigmoid
function, and $\sigma$ denotes the sigmoid function.

The output of CA module $O^{l}$ is an element-wise product of the merged
features $M^{l}$ and the attention $S^{l}$:

$O^{l}=M^{l}\otimes S^{l}$ (11)

(a) RetinaNet head with traditional convolutions.

(b) RetinaNet head with multi-scale convolutions.

Figure 6: RetinaNet head with different convolutions. TABLE I: Ablation studies of component effectiveness on COCO val-2017.“SA” and “CA” denote scale alignment and context attention module, respectively. The number in [] is the relative improvement over RetinaNet baseline. |  | $AP$ | $AP_{50}$ | $AP_{75}$ | $AP_{S}$ | $AP_{M}$ | $AP_{L}$ | Params (M) | FLOPs (G)
---|---|---|---|---|---|---|---|---|---
RetinaNet baseline | 33.2 | 52.5 | 35.4 | 15.8 | 37.1 | 46.5 | 37.74 | 95.56
Ours |  |  |  |  |  |  |  |
SA | CA | $AP$ | $AP_{50}$ | $AP_{75}$ | $AP_{S}$ | $AP_{M}$ | $AP_{L}$ | Params (M) | FLOPs (G)
|  | 34.8[+1.6] | 55.1 | 37.2 | 17.0 | 38.8 | 48.8 | 37.60 | 93.20
✓ |  | 35.6[+2.4] | 56.1 | 38.2 | 18.7 | 39.9 | 50.3 | 37.70 | 94.13
| ✓ | 35.6[+2.4] | 56.2 | 38.1 | 18.3 | 39.9 | 49.2 | 38.39 | 97.69
✓ | ✓ |  35.9[+2.7] | 56.4 | 38.9 | 18.4 | 40.0 | 50.6 | 38.49 | 98.61

(a) Original image.

(b) Score map w/o SA.

(c) Score map w/ SA.

Figure 7: Visualization of the confidence score maps without or with scale alignment module. TABLE II: The effect of different gathering levels on COCO val-2017. Gathering Level | $AP$ | $AP_{50}$ | $AP_{75}$ | $AP_{S}$ | $AP_{M}$ | $AP_{L}$
---|---|---|---|---|---|---
P7 | 34.6 | 55.2 | 37.0 | 17.6 | 38.6 | 48.1
P5 | 35.2 | 55.7 | 37.7 | 17.4 | 39.1 | 49.2
P3 | 35.9 | 56.4 | 38.9 | 18.4 | 40.0 | 50.6
TABLE III: Comparisons with other pyramid architectures based on RetinaNet detector on COCO val-2017. The number in [] is the relative improvement over FPN [18]. Method | $AP$ | $AP_{50}$ | $AP_{75}$ | $AP_{S}$ | $AP_{M}$ | $AP_{L}$ | Params (M) | FLOPs (G)
---|---|---|---|---|---|---|---|---
FPN [18] | 33.2 | 52.5 | 35.4 | 15.8 | 37.1 | 46.5 |  37.74(1.00x) | 95.56(1.00x)
PANet [12] | 33.4[+0.2] | 52.5 | 35.4 | 16.0 | 37.7 | 46.9 | 40.10(1.06x) | 97.92(1.02x)
PConv [25] | 33.8[+0.6] | 53.8 | 36.1 | 16.7 | 38.1 | 46.9 | 41.28(1.09x) | 96.78(1.01x)
Libra [22] | 33.9[+0.7] | 53.8 | 36.1 | 16.5 | 38.0 | 47.6 | 38.01(1.01x) | 95.75(1.00x)
NAS-FPN [13] | 35.1[+1.9] | 53.9 | 37.3 | 17.1 | 39.7 | 49.8 | 59.72(1.58x) | 138.60(1.45x)
BiFPN [26] | 35.2[+2.0] | 54.4 | 37.7 | 17.5 | 39.5 | 49.3 | 55.60(1.47x) | 122.34(1.28x)
SEPC-Lite [25] | 35.3[+2.1] | 55.3 | 37.6 | 17.6 | 39.4 | 49.9 | 41.37(1.10x) | 96.99(1.01x)
MSConv |  35.9[+2.7] | 56.4 | 38.9 | 18.4 | 40.0 | 50.6 | 38.49(1.02x) | 98.61(1.03x)

####  III-B3 Head Design

In this section, we introduce how to integrate our MSConv into single-stage
detectors. We take RetinaNet as an example to elaborate how to replace the
traditional convolution used in the detection head of single-stage detectors.
Figure 6 shows the difference between head design of traditional convolutions
and that of our MSConvs.

In the original RetinaNet, the multi-scale inputs are separately processed by
a shared head with two branches: classification and regression. The two
branches have independent weights but share the same input. At each branch,
the input features are passed through a fully convolution network (FCN)
consisting of several (4 by default) stacked convolutions to extract features
specially for classification or regression. Finally, a $3\times 3$ convolution
is applied to the extracted features to generate the final prediction.

It is easily to replace the traditional convolutions with MSConvs at each
branch. However, each MSConv still brings additional computation. To make a
compromise, the two branches share the same MSConv. To keep the difference
between classification and regression, we introduce an extra $3\times 3$
convolution for each branch after the shared MSConvs. The final prediction at
each branch is generated by a $3\times 3$ convolution.

##  IV Experiments

###  IV-A Dataset and Evaluation Metrics

We carry out our experiments on COCO [8] dataset. We use the data in
train-2017 split containing around 115k images to train our models, and
evaluate the performance for ablation studies on val-2017 split with about 5k
images. Main results are reported on the test-dev split (20k images without
available public annotations). We report all the results in the standard COCO-
style average precision (AP).

###  IV-B Experimental Settings

For fair comparisons, we conduct our experiments on MMDetecion [27] platform
in PyTorch [28] framework. If not specified, all the settings are the same
with described in MMDetection [27]. Modulated deformable convolution [24] is
applied.

Training Settings. We adopt ResNet-50 [6] as our default backbone network, and
RetianNet [17] as our default object detector. The backbone networks are
initialized with the weight of the models pretrained on ImageNet [7]. Our
models are trained using stochastic gradient descent (SGD) optimizer for 12
epochs with an initial learning rate of 0.01, which is divided by 10 after 8
and 11 epochs. Due to memory limitation, the batchsize (16 by default) will be
adjusted with a linearly scaled learning rate. Momentum and weight decay are
set to 0.9 and $1e^{-4}$, respectively. The resolution of the input images is
set to $640\times 640$.

Inference Settings. For each input image, we execute the following steps to
get the predictions. We collect the predictions with top 1000 confidences from
each prediction layer and use a confidence threshold of 0.05 for each class to
filter out the predictions with low confidences. For each class, we apply non
maximum suppression (NMS) with threshold of 0.5 to filter the low-quality
predictions. After NMS, we select the predictions with top 100 confidences as
the final results.

###  IV-C Ablation Studies

####  IV-C1 The effectiveness of each component

We analyze if each component of our model is effective for improvement of
detection. The experimental results are listed in Table I. The performance of
RetinaNet is shown in the first group, while the performance of our methods
are shown in the second group.

The third line of the second group in the table shows that our plain model
without any extra module can simply achieve a better performance than
RetinaNet baseline (+1.6% AP), even with fewer parameters and less
computational cost.

If we only introduce the scale alignment module, the detector can get 0.8% AP
improvement over our plain model, which is 2.4% higher than the original
RetinaNet. As shown in Figure 7, combined with the SA module, the foreground
regions of the objects are more accurate. Introducing the context attention
module, the detector gets an improvement of 2.4% AP over RetinaNet baseline,
with a little increase of computation.

The last line of the table reveals that applying both SA and CA module can
boost the performance of the detector to 35.9% AP, which is 2.7% higher than
that of the baseline.

####  IV-C2 The effectiveness of different gathering levels

In this section, we analyze the effect of different gathering level of MSConv.
As shown in Table II, as the gathering level goes down (from P7 to P3), the
performances under all metrics goes higher. The results show that the level
with the largest scale (P3) can keep the most information with the least loss,
which justifies the effectiveness to gather multi-scale features to the level
with the largest resolution.

TABLE IV: Applied in single-stage and two-stage detectors on COCO val-2017. The number in [] is the relative improvement over the original detector. Detector | MSConv | $AP$ | $AP_{50}$ | $AP_{75}$
---|---|---|---|---
single-stages |  |  |  |
RetinaNet |  | 33.2 | 52.5 | 35.4
RetinaNet | ✓ |  35.9[+2.7] | 56.4 | 38.9
FreeAnchor |  | 36.4 | 54.6 | 39.1
FreeAnchor | ✓ |  39.0[+2.6] | 58.1 | 42.1
RepPoints |  | 35.8 | 56.2 | 37.9
RepPoints | ✓ |  37.5[+1.7] | 57.6 | 40.4
two-stages |  |  |  |
Faster R-CNN |  | 34.6 | 55.7 | 36.9
Faster R-CNN | ✓ |  37.2[+2.6] | 57.8 | 39.9
Mask R-CNN |  | 35.2 | 56.4 | 37.9
Mask R-CNN | ✓ |  38.2[+3.0] | 57.6 | 43.0
Cascade R-CNN |  | 38.1 | 55.9 | 41.1
Cascade R-CNN | ✓ |  39.7[+1.6] | 57.6 | 43.0
TABLE V: Comparisons with state-of-the-art methods on COCO test-dev under single model and single-scale testing settings. Method | Backbone | $AP$ | $AP_{50}$ | $AP_{75}$ | $AP_{S}$ | $AP_{M}$ | $AP_{L}$
---|---|---|---|---|---|---|---
Two-stage methods |  |  |  |  |  |  |
Faster R-CNN w/ FPN [18] | ResNet-101 | 36.2 | 59.1 | 39.0 | 18.2 | 39.0 | 48.2
Mask R-CNN [4] | ResNet-101 | 38.2 | 60.3 | 41.7 | 20.1 | 41.1 | 50.2
Mask R-CNN [4] | ResNeXt-101 | 39.8 | 62.3 | 43.4 | 22.1 | 43.2 | 51.2
LH R-CNN [29] | ResNet-101 | 41.5 | - | - | 25.2 | 45.3 | 53.1
Cascade R-CNN [5] | ResNet-101 | 42.8 | 62.1 | 46.3 | 23.7 | 45.5 | 55.2
TridentNet [30] | ResNet-101 | 42.7 | 63.6 | 46.5 | 23.9 | 46.6 | 56.6
TridentNet [30] | ResNet-101-DCN | 46.8 | 67.6 | 51.5 | 28.0 | 51.2 | 60.5
TSD [31] | ResNet-101 | 43.2 | 64.0 | 46.9 | 24.0 | 46.3 | 55.8
One-stage methods |  |  |  |  |  |  |
RetinaNet [17] | ResNet-101 | 39.1 | 59.1 | 42.3 | 21.8 | 42.7 | 50.2
RetinaNet [17] | ResNeXt-101 | 40.8 | 61.1 | 44.1 | 24.1 | 44.2 | 51.2
FreeAnchor [1] | ResNet-101 | 43.1 | 62.2 | 46.4 | 24.5 | 46.1 | 54.8
FreeAnchor [1] | ResNeXt-101 | 44.9 | 64.3 | 48.5 | 26.8 | 48.3 | 55.9
FCOS [19] | ResNet-101 | 41.5 | 60.7 | 45.0 | 24.4 | 44.8 | 51.6
FCOS [19] | ResNeXt-101 | 44.7 | 64.1 | 48.4 | 27.6 | 47.5 | 55.6
ATSS [32] | ResNet-101 | 43.6 | 62.1 | 47.4 | 26.1 | 47.0 | 53.6
ATSS [32] | ResNet-101-DCN | 46.3 | 64.7 | 50.4 | 27.7 | 49.8 | 58.4
ATSS [32] | ResNeXt-101-DCN | 47.7 | 66.6 | 52.1 | 29.3 | 50.8 | 59.7
SAPD [33] | ResNet-101 | 43.5 | 63.6 | 46.5 | 24.9 | 46.8 | 54.6
SAPD [33] | ResNet-101-DCN | 46.0 | 65.9 | 49.6 | 26.3 | 49.2 | 59.6
SAPD [33] | ResNeXt-101-DCN | 46.6 | 66.6 | 50.0 | 27.3 | 49.7 | 60.7
RepPoints v2 [34] | ResNeXt-101 | 47.8 | 67.3 | 51.7 | 29.3 | 50.7 | 59.5
RepPoints v2 [34] | ResNet-101-DCN | 48.1 | 67.5 | 51.8 | 28.7 | 50.9 | 60.8
GFL [35] | ResNet-101 | 45.0 | 63.7 | 48.9 | 27.2 | 48.8 | 54.5
GFL [35] | ResNet-101-DCN | 47.3 | 66.3 | 51.4 | 28.0 | 51.1 | 59.2
GFL [35] | ResNeXt-101-DCN | 48.2 | 67.4 | 52.6 | 29.2 | 51.7 | 60.2
FreeAnchor w/ MSConv | ResNet-101-DCN | 47.7 | 67.5 | 52.2 | 29.6 | 51.2 | 58.5
FreeAnchor w/ MSConv | ResNeXt-101-DCN | 48.9 | 68.8 | 53.4 | 31.8 | 51.8 | 60.2

###  IV-D Comparisons with other Pyramid Architectures

In order to justify that our model is more effective and efficient , we
compare the performance of our method with state-of-the-art pyramid
architectures in Table III based on RetinaNet detector. Though NAS-FPN [13]
and BiFPN [26] achieve huge improvement over FPN [18], they require much more
computational cost. The last line of the table shows that our model achieves
the best performance among the state-of-the-art methods, with only a small
increase of parameters and FLOPs.

###  IV-E Application in Single-stage and Two-stage Detectors

In this section, we conduct experiments to evaluate the effectiveness of our
model on single-stage detectors, including anchor-based ones or anchor-free
ones, such as FreeAnchor [1] and RepPoints [2], and two-stage detectors.

As shown in the first group of Table IV, when combined with our method, the
single-stage detectors can get a significant improvement in AP. MSConv can
provide an increase of larger than 2.5% AP for the single-stage detectors. The
comparison of MSConv in various single-stage detectors is also shown in Figure
1.

In additional to single-stage detectors, MSConv is also effective for two-
stage detectors. The second group in Table IV lists the experimental results
of MSConv applied to two-stage detectors. When combined with MSConvs, two-
stage detectors get more than 2.5% improvement in AP. MSConv provides the most
increase in AP of 3.0% for Mask R-CNN.

###  IV-F Comparison with State-of-the-art Methods

In this section, we compare the performance of our method on COCO test-dev
split with state-of-the-art methods under single model single-scale testing
settings, which are shown in Table V. We use FreeAnchor as our detector. For
training, we adopt 2$\times$ learning schedule with scale-jitter. With
ResNet-101-DCN backbone, the AP (47.7%) of our method surpasses most of the
state-of-the-art methods using the same backbone. The backbone of
ResNeXt-101-DCN further improves our AP to 48.9%, which surpasses the AP of
all other competitors.

##  V Conclusions

In this paper, we propose the multi-scale convolution (MSConv), an extension
of the traditional convolution, to accept and deal with multi-scale input. The
proposed MSConv can meanwhile process the multi-scale input across channel,
spatial and scale dimension. MSConv can dramatically boost the detection
performance of single-stage object detectors with only a small increase of
computation. The results also suggest that MSConv is flexible and able to
bring considerable improvement as well to two-stage object detectors.

## References

  * [1] X. Zhang, F. Wan, C. Liu, R. Ji, and Q. Ye, “FreeAnchor: Learning to match anchors for visual object detection,” in _NeurIPS_ , 2019.
  * [2] Z. Yang, S. Liu, H. Hu, L. Wang, and S. Lin, “Reppoints: Point set representation for object detection,” in _ICCV_ , 2019.
  * [3] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” in _NeurIPS_ , 2015.
  * [4] K. He, G. Gkioxari, P. Dollar, and R. Girshick, “Mask R-CNN,” in _ICCV_ , 2017.
  * [5] Z. Cai and N. Vasconcelos, “Cascade r-cnn: Delving into high quality object detection,” in _CVPR_ , 2018.
  * [6] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016.
  * [7] J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2009.
  * [8] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _ECCV_ , 2014.
  * [9] J. Redmon, S. K. Divvala, R. B. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _CVPR_ , 2016.
  * [10] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “SSD: Single shot multibox detector,” in _ECCV_ , 2016.
  * [11] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _ICLR_ , 2015.
  * [12] S. Liu, L. Qi, H. Qin, J. Shi, and J. Jia, “Path aggregation network for instance segmentation,” in _CVPR_ , 2018.
  * [13] G. Ghiasi, T.-Y. Lin, and Q. V. Le, “NAS-FPN: Learning scalable feature pyramid architecture for object detection,” in _CVPR_ , 2019.
  * [14] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic 2014.
  * [15] R. Girshick, “Fast r-cnn,” in _ICCV_ , 2015.
  * [16] K. Chen, J. Pang, J. Wang, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Shi, W. Ouyang, C. C. Loy, and D. Lin, “Hybrid task cascade for instance segmentation,” in _CVPR_ , 2019.
  * [17] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar, “Focal loss for dense object detection,” in _ICCV_ , 2017.
  * [18] T.-Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _CVPR_ , 2017.
  * [19] Z. Tian, C. Shen, H. Chen, and T. He, “FCOS: Fully convolutional one-stage object detection,” in _ICCV_ , 2019.
  * [20] T. Kong, F. Sun, H. Liu, Y. Jiang, L. Li, and J. Shi, “Foveabox: Beyond anchor-based object detector,” _TIP_ , 2020.
  * [21] T. Kong, F. Sun, C. Tan, H. Liu, and W. Huang, “Deep feature pyramid reconfiguration for object detection,” in _ECCV_ , 2018.
  * [22] J. Pang, K. Chen, J. Shi, H. Feng, W. Ouyang, and D. Lin, “Libra R-CNN: Towards balanced learning for object detection,” in _CVPR_ , 2019.
  * [23] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in _CVPR_ , 2018.
  * [24] X. Zhu, H. Hu, S. Lin, and J. Dai, “Deformable convnets v2: More deformable, better results,” in _CVPR_ , 2019.
  * [25] X. Wang, S. Zhang, Z. Yu, L. Feng, and W. Zhang, “Scale-equalizing pyramid convolution for object detection,” in _CVPR_ , 2020.
  * [26] M. Tan, R. Pang, and Q. V. Le, “Efficientdet: Scalable and efficient object detection,” in _CVPR_ , 2020.
  * [27] K. Chen, J. Wang, J. Pang, Y. Cao, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Xu _et al._ , “MMDetection: Open mmlab detection toolbox and benchmark,” _arXiv:1906.07155_ , 2019.
  * [28] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” in _NeurIPS_ , 2019.
  * [29] Z. Li, C. Peng, G. Yu, X. Zhang, Y. Deng, and J. Sun, “Light-head R-CNN: In defense of two-stage object detector,” _arXiv:1711.07264_ , 2017.
  * [30] Y. Li, Y. Chen, N. Wang, and Z. Zhang, “Scale-aware trident networks for object detection,” in _ICCV_ , 2019.
  * [31] G. Song, Y. Liu, and X. Wang, “Revisiting the sibling head in object detector,” in _CVPR_ , 2020.
  * [32] S. Zhang, C. Chi, Y. Yao, Z. Lei, and S. Z. Li, “Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection,” in _CVPR_ , 2020.
  * [33] C. Zhu, F. Chen, Z. Shen, and M. Savvides, “Soft anchor-point object detection,” in _ECCV_ , 2020.
  * [34] Y. Chen, Z. Zhang, Y. Cao, L. Wang, S. Lin, and H. Hu, “Reppoints v2: Verification meets regression for object detection,” in _NeurIPS_ , 2020\.
  * [35] X. Li, W. Wang, L. Wu, S. Chen, X. Hu, J. Li, J. Tang, and J. Yang, “Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection,” in _NeurIPS_ , 2020.


# Applying Machine Learning to Study Fluid Mechanics

Steven L. Brunton1∗
1 Department of Mechanical Engineering, University of Washington, Seattle, WA
98195, United States

###### Abstract

This paper provides a short overview of how to use machine learning to build
data-driven models in fluid mechanics. The process of machine learning is
broken down into five stages: (1) formulating a problem to model, (2)
collecting and curating training data to inform the model, (3) choosing an
architecture with which to represent the model, (4) designing a loss function
to assess the performance of the model, and (5) selecting and implementing an
optimization algorithm to train the model. At each stage, we discuss how prior
physical knowledge may be embedding into the process, with specific examples
from the field of fluid mechanics.

_Keywords–_ Machine learning, fluid mechanics, physics-informed machine
learning, neural networks, deep learning

††∗ Corresponding author (sbrunton@uw.edu).

##  1 Introduction

The field of fluid mechanics is rich with data and rife with problems, which
is to say that it is a perfect playground for machine learning. Machine
learning is the art of building models from data using optimization and
regression algorithms. Many of the challenges in fluid mechanics may be posed
as optimization problems, such designing a wing to maximize lift while
minimizing drag at cruise velocities, estimating a flow field from limited
measurements, controlling turbulence for mixing enhancement in a chemical
plant or drag reduction behind a vehicle, among myriad others. These
optimization tasks fit well with machine learning algorithms, which are
designed to handle nonlinear and high-dimensional problems. In fact, machine
learning and fluid mechanics both tend to rely on the same assumption that
there are patterns that can be exploited, even in high-dimensional systems
[1]. Often, the machine learning algorithm will model some aspect of the
fluid, such as the lift profile given a particular airfoil geometry, providing
a _surrogate_ that may be optimized over. Machine learning may also be used to
directly solve the fluid optimization task, such as designing a machine
learning model to manipulate the behavior of the fluid for some engineering
objective with active control [2, 3, 4].

In either case, it is important to realize that machine learning is _not_ an
automatic or turn-key procedure for extracting models from data. Instead, it
requires expert human guidance at every stage of the process, from deciding on
the problem, to collecting and curating data that might inform the model, to
selecting the machine learning architecture best capable of representing or
modeling the data, to designing custom loss functions to quantify performance
and guide the optimization, to implementing specific optimization algorithms
to train the machine learning model to minimize the loss function over the
data. A better name for machine learning might be “expert humans teaching
machines how to learn a model to fit some data,” although this is not as
catchy. Particularly skilled (or lucky!) experts may design a learner or a
learning framework that is capable of learning a variety of tasks,
generalizing beyond the training data, and mimicking other aspects of
intelligence. However, such artificial intelligence is rare, even more so than
human intelligence. The majority of machine learning models are just that,
models, which should fit directly into the decades old practice of model-based
design, optimization, and control [5].

With its unprecedented success on many challenging problems in computer vision
and natural language processing, machine learning is rapidly entering the
physical sciences, and fluid mechanics is no exception. The simultaneous
promise, and over-promise, of machine learning is causing many researchers to
have a healthy mixture of optimism and skepticism. In both cases, there is a
strong desire to understand the uses and limitations of machine learning, as
well as best practices for how to incorporate it into existing research and
development workflows. It is also important to realize that while it is now
relatively simple to train a machine learning model for a well-defined task,
it is still quite difficult to create a new model that outperforms traditional
numerical algorithms and physics-based models. Incorporating partially known
physics into the machine learning pipeline well tend to improve model
generalization and improve interpretability and explainability, which are key
elements of modern machine learning [6, 7].

Figure 1: Schematic of the five stages of machine learning on an example of
reduced-order modeling. In this case, the goal is to learn a low dimensional
coordinate system $\mathbf{z}=\bm{f}_{1}(\mathbf{x},\bm{\theta}_{1})$ from
data in a high-dimensional representation $\mathbf{x}$, along with a dynamical
system model $\dot{\mathbf{z}}=\bm{f}_{2}(\mathbf{z},\bm{\theta}_{2})$ for how
the state $\mathbf{z}$ evolves in time. Finally, this latent state derivative
$\dot{\mathbf{z}}$ must be able to approximate the high dimensional derivative
$\dot{\mathbf{x}}$ through the decoder
$\dot{\mathbf{x}}\approx\bm{f}_{3}(\dot{\mathbf{z}},\bm{\theta}_{3})$. The
loss function $\mathcal{L}(\bm{\theta},\mathbf{X})$ defines how well the model
performs, averaged over the data $\mathbf{X}$. Finally, the parameters
$\bm{\theta}=\\{\bm{\theta}_{1},\bm{\theta}_{2},\bm{\theta}_{3}\\}$ are found
through optimization.

##  2 Physics Informed Machine Learning for Fluid Mechanics

Applied machine learning may be separated into a few canonical steps, each of
which provides an opportunity to embed prior physical knowledge: (1) choosing
the problem to model or the question to answer; (2) choosing and curating the
data used to train the model; (3) deciding on a machine learning architecture
to best represent or model this data; (4) designing loss functions to quantify
performance and to guide the learning process; and (5) implementing an
optimization algorithm to train the model to minimize the loss function over
the training data. See Fig. 1 for a schematic of this process on the example
of reduced-order modeling. This organization of steps is only approximate, and
there are considerable overlaps and tight interconnections between each stage.
For example, choosing the problem to model and choosing the data to inform
this model are two closely related decisions. Similarly, designing a custom
loss function and implementing an optimization algorithm to minimize this loss
function are tightly coupled. In most modern machine learning workflows, it is
common to iteratively revisit earlier stages based on the outcome at later
stages, so that the machine learning researcher is constantly asking new
questions and revising the data, the architecture, the loss functions, and the
optimization algorithm to improve performance. Here, we discuss these
canonical stages of machine learning, investigate how to incorporate physics,
and review examples in the field of fluid mechanics. This discussion is
largely meant to be a high-level overview, and many more details can be found
in recent reviews [8, 9, 5, 10].

###  2.1 The problem

Data science is the art of asking and answering questions with data. The sub-
field of machine learning is concerned with leveraging historical data to
build models that may be deployed to automatically answer these questions,
ideally in real-time, given new data. It is critical to select the right
system to model, motivated by a problem that is both important and tractable.
Choosing a problem involves deciding on input data that will be readily
available in the future, and output data that will represent the desired
output, or prediction, of the model. The output data should be determinable
from the inputs, and the functional relationship between these is precisely
what the machine learning model will be trained to capture.

The nature of the problem, specifically what outputs will be modeled given
what inputs, determines the large classes of machine learning algorithms:
_supervised_ , _unsupervised_ , and _reinforcement learning_. In supervised
learning, the training data will have expert labels that should be predicted
or modeled with the machine learning algorithm. These output labels may be
discrete, such as a categorical label of a ‘dog’ or a ‘cat’ given an input
image, in which case the task is one of _classification_. If the labels are
continuous, such as the average value of lift or drag given a specified
airfoil geometry, then the task is one of _regression_. In unsupervised
learning, there are no expert labels, and structure must be extracted from the
input data alone; thus, this is often referred to as _data mining_ , and
constitutes a particularly challenging field of machine learning. Again, if
the structure in the data is assumed to be discrete, then the task is
_clustering_. After the clusters are identified and characterized, these
groupings may be used as proxy labels to then classify new data. If the
structure in the data is assumed to be continuously varying, then the task is
typically thought of as an _embedding_ or _dimensionality reduction_ task.
Principal component analysis (PCA) or proper orthogonal decomposition (POD)
may be thought of as unsupervised learning tasks that seek a continuous
embedding of reduced dimension [11]. Reinforcement learning is a third, large
branch of machine learning research, in which an _agent_ learns to make
control decisions to interact with an environment for some high level
objection [12]. Examples include learning how to play games [13, 14], such as
chess and go.

#### Embedding physics:

Deciding on what phenomena to model with machine learning is often inherently
related to the underlying physics. Although classical machine learning has
been largely applied to “static” tasks, such as image classification and the
placement of advertisements, increasingly it is possible to apply these
techniques to model physical systems that evolve in time according to some
_rules_ or _physics_. For example, we may formulate a learning problem to find
and represent a conserved quantity, such as a Hamiltonian, purely from data
[15]. Alternatively, the machine learning task may be to model time-series
data as a differential equation, with the learning algorithm representing the
dynamical system [16, 17, 18, 19, 20]. Similarly, the task may involve
learning a coordinate transformation where these dynamics become simplified in
some _physical_ way; i.e., coordinate transformations to linearize or
diagonalize/decouple dynamics [21, 22, 23, 24, 25, 26, 27, 28].

#### Examples in fluid mechanics:

There are many _physical_ modeling tasks in fluid mechanics that are
benefiting from machine learning [9, 5]. A large field of study focuses on
formulating turbulence closure modeling as a machine learning problem [8, 29],
such as learning models for the Reynolds stresses [30, 31] or sub-gridscale
turbulence [32, 33]. Designing useful input features is also an important way
that prior physical knowledge is incorporated into turbulence closure modeling
[34, 35, 36]. Similarly, machine learning has recently been focused on the
problem of improving computational fluid dynamics (CFD) solvers [37, 38, 39,
40]. Other important problems in fluid mechanics that benefit from machine
learning include super-resolution [41, 42], robust modal decompositions [1,
43, 44], network and cluster modeling [45, 46, 47], control [48, 4] and
reinforcement learning [49, 50], and design of experiments in cyberphysical
systems [51]. Aerodynamics is a large related field with significant data-
driven advances [52]. The very nature of these problems embeds the learning
process into a larger physics-based framework, so that the models are more
physically relevant by construction.

###  2.2 The data

Data is the lifeblood of machine learning, and our ability to build effective
models relies on what data is available or may be collected. As discussed
earlier, choosing data to inform a model is closely related to choosing what
to model in the first place, and therefore this stage cannot be strictly
separated from the choice of a problem above. The quality and quantity of data
directly affects the resulting machine learning model. Many machine learning
architectures, such as deep neural networks, are essentially sophisticated
interpolation engines, and so having a diversity of training data is essential
to these models being useful on unseen data. For example, modern deep
convolutional neural networks rose to prominence with their unprecedented
classification accuracy [53] on the ImageNet data base [54], which contains
over $14$ million labeled images with over $20,000$ categories, providing a
sufficiently large and rich set of examples for training. This pairing of a
vast labeled data set with a novel deep learning architecture is widely
regarded as the beginning of the modern era of deep learning [55].

#### Embedding physics:

The training data provides several opportunities to embed prior physical
knowledge. If a system is known to exhibit a symmetry, such translational or
rotational invariance, then it is possible to augment and enrich the training
data with shifted or rotated examples. More generally, it is often assumed
that with an abundance of training data, these physical invariances will
automatically be learned by a sufficiently expressive architecture. However,
this approach tends to require considerable resources, both to collect and
curate the data, as well as to train increasingly large models, making it more
appropriate for industrial scale, rather than academic scale, research. In
contrast, it is also possible to use physical intuition to craft new features
from the training data, for example by applying a coordinate transformation
that may simplify the representation or training. Physical data often comes
from multiple sources with different fidelity, such as from numerical
simulations, laboratory experiments, and in-flight tests. This is an important
area of research for flight testing and unsteady aerodynamics [52], and
recently physics informed neural networks have been used with multifidelity
data to approximate PDEs [56].

#### Examples in fluid mechanics:

Fluids data is notoriously vast and high-dimensional, with individual flow
fields often requiring millions (or more!) degrees of freedom to characterize.
Moreover, these flow fields typically evolve in time, resulting in a time
series of multiple snapshots. Although vast in the spatial and/or temporal
dimensions, data is often rather sparse in parameter space, as it is expensive
to numerically or experimentally investigate multiple geometries, Reynolds
numbers, etc. Thus there are many algorithms designed for both rich and sparse
data. Other considerations involve exciting transients and observing how the
system evolves when it is away from its natural state. In many other cases,
fluids data might be quite limited, for example given by time-series data from
a few pressure measurements on the surface of an airfoil, or from force
recordings on an experimental turbine.

###  2.3 The architecture

Once a problem has been identified, and data is collected and curated, it is
necessary to choose an architecture with which to represent the machine
learning model. Typically, a machine learning model is a function that maps
inputs to outputs

$\displaystyle\mathbf{y}=\mathbf{f}(\mathbf{x};\bm{\theta})$ (1)

and this function is generally represented within a specified family of
functions parameterized by values in $\bm{\theta}$. For example, a linear
regression model would model outputs as a linear function of the inputs, with
$\bm{\theta}$ parameterizing this linear map, or matrix. Neural networks have
emerged as a particularly powerful and flexible class of models to represent
functional relationships between data, and they have been shown to be able to
approximate arbitrarily complex functions with sufficient data and depth [57,
58]. There is a tremendous variety of potential neural network architectures
[11], limited only by the imagination of the human designer. The most common
architecture is a simple feedforward network, in which data enters through an
input layer and maps sequentially through a number of computational layers
until an output layer. Each layer consists of nodes, where data from nodes in
the previous layer are combined in a weighted sum and processed through an
activation function, which is typically nonlinear. In this way, neural
networks are fundamentally compositional in nature. The parameters
$\bm{\theta}$ determine the network weights for how data is passed from one
layer to the next, i.e. the weighted connectivity matrices for how nodes are
connected in adjacent layers. The overarching network topology (i.e., how many
layers, how large, what type of activation functions, etc.) is specified by
the architect or determined in a meta-optimization, thus determining the
family of functions that may be approximated by that class of network. Then,
the network weights for the specific architecture are optimized over the data
to minimize a given loss function; these stages are described next.

It is important to note that not all machine learning architectures are neural
networks, although they are one of the most powerful and expressive modern
architectures, powered by increasingly big data and high performance
computing. Before the success of deep convolutional networks on the ImageNet
dataset, neural networks were not even mentioned in the list of top ten
machine learning algorithms [59]. Random forests [60] and support vector
machines [61] are two other leading architectures for supervised learning.
Bayesian methods are also widely used, especially for dynamical systems [62].
Genetic programming has also been widely used to learn human-interpretable,
yet flexible representations of data for modeling [63, 16, 64, 65] and control
[4]. In addition, standard linear regression and generalized linear regression
are still widely used for modeling time-series data, especially in fluids. The
dynamic mode decomposition (DMD) [17, 66, 1] employs linear regression with a
low-rank constraint in the optimization to find dominant spatiotemporal
coherent structures that evolve linearly in time. The sparse identification of
nonlinear dynamics (SINDy) [18] algorithm employs generalized linear
regression, with either a sparsity promoting loss function [67] or a sparse
optimization algorithm [18, 68], to identify a differential equation model
with as few model terms as are necessary to fit the data.

#### Embedding physics:

Choosing a machine learning architecture with which to model the training data
is one of the most intriguing opportunities to embed physical knowledge into
the learning process. Among the simplest choices are convolutional networks
for translationally invariant systems, and recurrent networks, such as long-
short-time memory (LSTM) networks [20] or reservoir computing [19, 69], for
systems that evolve in time. LSTMs have recently been used to predict
aeroelastic responses across a range of Mach numbers [70]. More generally,
equivariant networks seek to encode various symmetries by construction, which
should improve accuracy and reduce data requirements for physical systems [71,
72, 73, 74]. Autoencoder networks enforce the physical notion that there
should be low-dimensional structure, even for high-dimensional data, by
imposing an information bottleneck, given by a constriction of the number of
nodes in one or more layers of the network. Such networks uncover nonlinear
manifolds where the data is compactly represented, generalizing the linear
dimensionality reduction obtained by PCA and POD. It is also possible to embed
physics more directly into the architecture, for example by incorporating
Hamiltonian [75, 76] or Lagrangian [77, 78] structure. There are numerous
successful examples of physics-informed neural networks (PINNs) [79, 80, 81,
82, 83], which solve supervised learning problems while being constrained to
satisfy a governing physical law. Graph neural networks have also shown the
ability to learn generalizable physics in a range of challenging domains [84,
64, 85]. Deep operator networks [86] are able to learn continuous operators,
such as governing partial differential equations, from relatively limited
training data.

#### Examples in fluid mechanics:

There are numerous examples of custom neural network architectures being used
to enforce physical solutions for applications in fluid mechanics. The work of
Ling et al. [30] designed a custom neural network layer that enforced Galilean
invariance in the Reynolds stress tensors that they were modeling. Related
Reynolds stress models have been developed using the SINDy sparse modeling
approach [87, 88, 89]. Hybrid models that combine linear system identification
and nonlinear neural networks have been used to model complex aeroelastic
systems [90]. The hidden fluid mechanics (HFM) approach is a physics-informed
neural network strategy that encodes the Navier-Stokes equations while being
flexible to the boundary conditions and geometry of the problem, enabling
impressive physically quantifiable flow field estimations from limited data
[91]. Sparse sensing has also been used to recover pressure distributions
around airfoils [92]. The Fourier neural operator is a novel operator network
that performs super-resolution upscaling and simulation modeling tasks [93].
Equivariant convolutional networks have been designed and applied to enforce
symmetries in high-dimensional complex systems from fluid dynamics [73].
Physical invariances have also been incorporated into neural networks for
subgrid-scale scalar flux modeling [94]. Lee and Carlberg [95] recently showed
how to incorporate deep convolutional autoencoder networks into the broader
reduced-order modeling framework [96, 97, 98], taking advantage of the
superior dimensionality reduction capabilities of deep autoencoders.

###  2.4 The loss function

The loss function is how we quantify how well the model is performing, often
on a variety of tasks. For example, the $L_{2}$ error between the model output
and the true output, averaged over the input data, is a common term in the
loss function. In addition, other terms may be added to regularize the
optimization (e.g., the $L_{1}$ or $L_{2}$ norm of the parameters
$\bm{\theta}$ to promote parsimony and prevent overfitting). Thus, the loss
function typically balances multiple competing objectives, such as model
performance and model complexity. The loss function may also incorporate terms
used to promote a specific behavior across different sub-networks in a neural
network architecture. Importantly, the loss function will provide valuable
information used to approximate gradients required to optimize the parameters.

#### Embedding physics:

Most of the physics-informed architectures described above involve custom loss
functions to promote the efficient training of accurate models. It is also
possible to incorporate physical priors, such as sparsity, by adding $L_{1}$
or $L_{0}$ regularizing loss terms on the parameters in $\bm{\theta}$. In
fact, parsimony has been a central theme in physical modeling for century,
where it is believed that balancing model complexity with descriptive
capability is essential in developing models that generalize. The sparse
identification of nonlinear dynamics algorithm [18] learns dynamical systems
models with as few terms from a library of candidate terms as are needed to
describe the training data. There are several formulations involving different
loss terms and optimization algorithms that promote additional physical
notions, such as stability [99] and energy conservation [100]. Stability
promoting loss functions based on notions of Lyapunov stability have also been
incorporated into autoencoders, with impressive results on fluid systems
[101].

#### Examples in fluid mechanics:

Sparse nonlinear modeling has been used extensively in fluid mechanics, adding
sparsity-promoting loss terms to learn parsimonious models that prevent
overfitting and generalize to new scenarios. SINDy has been used to generate
reduced-order models for how dominant coherent structures evolve in a flow for
a range of configurations [100, 102, 103, 104, 105]. These models have also
been extended to develop compact closure models [87, 88, 89]. Recently, the
physical notion of _boundedness_ of solutions, which is a fundamental concept
in reduced-order models of fluids [106], has been incorporated into the SINDy
modeling framework as a novel loss function. Other physical loss functions may
be added, such as adding the divergence of a flow field as a loss term to
promote solutions that are incompressible [107].

###  2.5 The optimization algorithm

Ultimately, machine learning models are trained using optimization algorithms
to find the parameters $\bm{\theta}$ that best fit the training data.
Typically, these optimization problems are both high-dimensional and non-
convex, leading to extremely challenging optimization landscapes with many
local minima. While there are powerful and generic techniques for convex
optimization problems [108, 109], there are few generic guarantees for
convergence or global optimality in non-convex optimization. Modern deep
neural networks have particularly high-dimensional parameters $\bm{\theta}$
and require large training data sets, which necessitate stochastic gradient
descent algorithms. In a sense, the optimization algorithm is the engine
powering machine learning, and as such, it is often abstracted from the
decision process. However, developing advanced optimization algorithms is the
focus of intense research efforts. It is also often necessary to explicitly
consider the optimization algorithm when designing a new architecture or
incorporating a novel loss term.

#### Embedding physics:

There are several ways that the optimization algorithm may be customized or
modified to incorporate prior physical knowledge. One approach is to
explicitly add constraints to the optimization, for example that certain
coefficients must be non-negative, or that other coefficients must satisfy a
specified algebraic relationship with each other. Depending on the given
machine learning architecture, it may be possible to enforce energy
conservation [100] or stability constraints [99] in this way. Another approach
involves employing custom optimization algorithms required to minimize the
physically motivated loss functions above, which are often non-convex. In this
way, the line between loss function and optimization algorithm are often
blurred, as they are typically tightly coupled. For example, promoting
sparsity with the $L_{0}$ norm is non-convex, and several relaxed optimization
formulations have been developed to approximately solve this problem. The
sparse relaxed regularized regression (SR3) optimization framework [68] has
been developed specifically to handle challenging non-convex loss terms that
arise in physically motivated problems.

#### Examples in fluid mechanics:

Loiseau [100] showed that it is possible to enforce energy conservation for
incompressible fluid flows directly by imposing skew-symmetry constraints on
the quadratic terms of a sparse generalized linear regression (i.e. SINDy)
model. These constraints manifest as equality constraints on the sparse
coefficients $\bm{\theta}$ of the SINDy model. Because the standard SINDy
optimization procedure is based on a sequentially thresholded least-squares
procedure, it is possible to enforce these equality constraints at every stage
of the regression, using the Karush–Kuhn–Tucker (KKT) conditions. The SR3
optimization package [68] was developed to generalize and extend these
constrained optimization problems to more challenging constraints, and to more
generic optimization problems. This is only one of many examples of custom
optimization algorithms being developed to train machine learning models with
novel loss functions or architectures.

##  3 Parting Thoughts

This brief paper has attempted to provide a high level overview of the various
stages of machine learning, how physics can be incorporated at each stage, and
how these techniques are being applied today in fluid mechanics. Machine
learning for physical systems requires careful consideration in each of these
steps, as every stage provides an opportunity to incorporate prior knowledge
about the physics. A working definition of physics is the part of a model that
generalizes, and this is one of the central goals of machine learning models
for physical systems. It is also important to note that machine learning is
fundamentally a collaborative effort, as it is nearly impossible to master
every stage of this process.

The nature of this topic is mercurial, as new innovations are being introduced
every day that improve our capabilities and challenge our previous
assumptions. Much of this work has deliberately oversimplified the process of
machine learning and the field of fluid mechanics. Machine learning is largely
concerned with fitting functions from data, and so it is important to pick the
right functions to fit. The inputs to the function are the variables and
parameters that we have access to or control over, and the outputs are
quantities of interest that we would like to accurately and efficiently
approximate in the future. It is a fruitful exercise to revisit classically
important problems where progress was limited by our ability to represent
complex functions. For example, Ling et al. [30] had great success revisiting
the classical Reynolds stress models of Pope [110] with powerful modern
techniques. More fundamentally, machine learning is about asking and answering
questions with data. We can’t forget why we are asking these questions in the
first place: because we are curious, and there is value in knowing the answer.

## Disclaimer

Any omission or oversight was the result of either ignorance, forgetfulness,
hastiness, or lack of imagination on my part. These notes are not meant to be
exhaustive, but rather to provide a few concrete examples from the literature
to guide researchers getting started in this field. This field is growing at
an incredible rate, and these examples provide a tiny glimpse into a much
larger effort. I have tried to sample from what I consider some of the most
relevant and accessible literature. However, a disproportionate number of
references are to work by my close collaborators, as this is the work I am
most familiar with. If I have missed any important references or connections,
or mis-characterized any works cited here, please let me know and I’ll try to
incorporate corrections in future versions of these notes.

## Acknowledgments

SLB acknowledges many valuable discussions and perspectives gained from
collaborators and coauthors Petros Koumoutsakos, J. Nathan Kutz, Jean-
Christophe Loiseau, and Bernd Noack.

## References

  * [1] Kunihiko Taira, Steven L Brunton, Scott Dawson, Clarence W Rowley, Tim Colonius, Beverley J McKeon, Oliver T Schmidt, Stanislav Gordeyev, Vassilios Theofilis, and Lawrence S Ukeiley.  Modal analysis of fluid flows: An overview.  AIAA Journal, 55(12):4013–4041, 2017.
  * [2] Jean Rabault, Miroslav Kuchta, Atle Jensen, Ulysse Réglade, and Nicolas Cerardi.  Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control.  Journal of fluid mechanics, 865:281–302, 2019.
  * [3] Feng Ren, Hai-bao Hu, and Hui Tang.  Active flow control using machine learning: A brief review.  Journal of Hydrodynamics, 32(2):247–253, 2020.
  * [4] Yu Zhou, Dewei Fan, Bingfu Zhang, Ruiying Li, and Bernd R Noack.  Artificial intelligence control of a turbulent jet.  Journal of Fluid Mechanics, 897, 2020.
  * [5] Steven L. Brunton, Bernd R. Noack, and Petros Koumoutsakos.  Machine learning for fluid mechanics.  Annual Review of Fluid Mechanics, 52:477–508, 2020.
  * [6] Mengnan Du, Ninghao Liu, and Xia Hu.  Techniques for interpretable machine learning.  Communications of the ACM, 63(1):68–77, 2019.
  * [7] Christoph Molnar.  Interpretable machine learning.  Lulu. com, 2020.
  * [8] Karthik Duraisamy, Gianluca Iaccarino, and Heng Xiao.  Turbulence modeling in the age of data.  Annual Reviews of Fluid Mechanics, 51:357–377, 2019.
  * [9] MP Brenner, JD Eldredge, and JB Freund.  Perspective on machine learning for advancing fluid mechanics.  Physical Review Fluids, 4(10):100501, 2019.
  * [10] Michael P Brenner and Petros Koumoutsakos.  Machine learning and physical review fluids: An editorial perspective.  Physical Review Fluids, 6(7):070001, 2021.
  * [11] S. L. Brunton and J. N. Kutz.  Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control.  Cambridge University Press, 2019.
  * [12] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction, volume 1.  MIT press Cambridge, 1998.
  * [13] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  Nature, 518(7540):529, 2015.
  * [14] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al.  Mastering the game of go without human knowledge.  nature, 550(7676):354–359, 2017.
  * [15] Eurika Kaiser, J Nathan Kutz, and Steven L Brunton.  Discovering conservation laws from data for control.  In 2018 IEEE Conference on Decision and Control (CDC), pages 6415–6421. IEEE, 2018.
  * [16] Michael Schmidt and Hod Lipson.  Distilling free-form natural laws from experimental data.  Science, 324(5923):81–85, 2009.
  * [17] P. J. Schmid.  Dynamic mode decomposition of numerical and experimental data.  Journal of Fluid Mechanics, 656:5–28, August 2010.
  * [18] S. L. Brunton, J. L. Proctor, and J. N. Kutz.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems.  Proceedings of the National Academy of Sciences, 113(15):3932–3937, 2016.
  * [19] Jaideep Pathak, Zhixin Lu, Brian R Hunt, Michelle Girvan, and Edward Ott.  Using machine learning to replicate chaotic attractors and calculate lyapunov exponents from data.  Chaos: An Interdisciplinary Journal of Nonlinear Science, 27(12):121102, 2017.
  * [20] Pantelis R Vlachas, Wonmin Byeon, Zhong Y Wan, Themistoklis P Sapsis, and Petros Koumoutsakos.  Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networks.  Proc. R. Soc. A, 474(2213):20170844, 2018.
  * [21] Bethany Lusch, J Nathan Kutz, and Steven L Brunton.  Deep learning for universal linear embeddings of nonlinear dynamics.  Nature Communications, 9(1):4950, 2018.
  * [22] Christoph Wehmeyer and Frank Noé.  Time-lagged autoencoders: Deep learning of slow collective variables for molecular kinetics.  The Journal of Chemical Physics, 148(241703):1–9, 2018.
  * [23] Andreas Mardt, Luca Pasquali, Hao Wu, and Frank Noé.  VAMPnets: Deep learning of molecular kinetics.  Nature Communications, 9(5), 2018.
  * [24] Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi.  Learning koopman invariant subspaces for dynamic mode decomposition.  In Advances in Neural Information Processing Systems, pages 1130–1140, 2017.
  * [25] Qianxiao Li, Felix Dietrich, Erik M Bollt, and Ioannis G Kevrekidis.  Extended dynamic mode decomposition with dictionary learning: A data-driven adaptive spectral decomposition of the koopman operator.  Chaos: An Interdisciplinary Journal of Nonlinear Science, 27(10):103111, 2017.
  * [26] Enoch Yeung, Soumya Kundu, and Nathan Hodas.  Learning deep neural network representations for koopman operators of nonlinear dynamical systems.  arXiv preprint arXiv:1708.06850, 2017.
  * [27] Samuel E Otto and Clarence W Rowley.  Linearly-recurrent autoencoder networks for learning dynamics.  SIAM Journal on Applied Dynamical Systems, 18(1):558–593, 2019\.
  * [28] K. Champion, B. Lusch, J. Nathan Kutz, and Steven L. Brunton.  Data-driven discovery of coordinates and governing equations.  Proceedings of the National Academy of Sciences, 116(45):22445–22451, 2019.
  * [29] Shady E Ahmed, Suraj Pawar, Omer San, Adil Rasheed, Traian Iliescu, and Bernd R Noack.  On closures for reduced order models $-$ a spectrum of first-principle to machine-learned avenues.  arXiv preprint arXiv:2106.14954, 2021.
  * [30] Julia Ling, Andrew Kurzawski, and Jeremy Templeton.  Reynolds averaged turbulence modelling using deep neural networks with embedded invariance.  Journal of Fluid Mechanics, 807:155–166, 2016.
  * [31] J Nathan Kutz.  Deep learning in fluid dynamics.  Journal of Fluid Mechanics, 814:1–4, 2017.
  * [32] Romit Maulik, Omer San, Adil Rasheed, and Prakash Vedula.  Subgrid modelling for two-dimensional turbulence using neural networks.  Journal of Fluid Mechanics, 858:122–144, 2019.
  * [33] Guido Novati, Hugues Lascombes de Laroussilhe, and Petros Koumoutsakos.  Automating turbulence modelling by multi-agent reinforcement learning.  Nature Machine Intelligence, 3(1):87–96, 2021.
  * [34] Jian-Xun Wang, Jin-Long Wu, and Heng Xiao.  Physics-informed machine learning approach for reconstructing reynolds stress modeling discrepancies based on dns data.  Physical Review Fluids, 2(3):034603, 2017.
  * [35] Linyang Zhu, Weiwei Zhang, Jiaqing Kou, and Yilang Liu.  Machine learning methods for turbulence modeling in subsonic flows around airfoils.  Physics of Fluids, 31(1):015105, 2019.
  * [36] Linyang Zhu, Weiwei Zhang, Xuxiang Sun, Yilang Liu, and Xianxu Yuan.  Turbulence closure for high reynolds number airfoil flows by deep neural networks.  Aerospace Science and Technology, 110:106452, 2021.
  * [37] Yohai Bar-Sinai, Stephan Hoyer, Jason Hickey, and Michael P Brenner.  Learning data-driven discretizations for partial differential equations.  Proceedings of the National Academy of Sciences, 116(31):15344–15349, 2019.
  * [38] Stephan Thaler, Ludger Paehler, and Nikolaus A Adams.  Sparse identification of truncation errors.  Journal of Computational Physics, 397:108851, 2019.
  * [39] Ben Stevens and Tim Colonius.  Enhancement of shock-capturing methods via machine learning.  Theoretical and Computational Fluid Dynamics, 34:483–496, 2020\.
  * [40] Dmitrii Kochkov, Jamie A Smith, Ayya Alieva, Qing Wang, Michael P Brenner, and Stephan Hoyer.  Machine learning accelerated computational fluid dynamics.  arXiv preprint arXiv:2102.01010, 2021.
  * [41] N Benjamin Erichson, Lionel Mathelin, Zhewei Yao, Steven L Brunton, Michael W Mahoney, and J Nathan Kutz.  Shallow neural networks for fluid flow reconstruction with limited sensors.  Proceedings of the Royal Society A, 476(2238):20200097, 2020.
  * [42] Kai Fukami, Koji Fukagata, and Kunihiko Taira.  Super-resolution reconstruction of turbulent flows with machine learning.  Journal of Fluid Mechanics, 870:106–120, 2019.
  * [43] Kunihiko Taira, Maziar S Hemati, Steven L Brunton, Yiyang Sun, Karthik Duraisamy, Shervin Bagheri, Scott Dawson, and Chi-An Yeh.  Modal analysis of fluid flows: Applications and outlook.  AIAA Journal, 58(3):998–1022, 2020.
  * [44] Isabel Scherl, Benjamin Strom, Jessica K Shang, Owen Williams, Brian L Polagye, and Steven L Brunton.  Robust principal component analysis for particle image velocimetry.  Physical Review Fluids, 5(054401), 2020.
  * [45] Aditya G Nair and Kunihiko Taira.  Network-theoretic approach to sparsified discrete vortex dynamics.  Journal of Fluid Mechanics, 768:549–571, 2015.
  * [46] E. Kaiser, B. R. Noack, L. Cordier, A. Spohn, M. Segond, M. Abel, G. Daviller, J. Osth, S. Krajnovic, and R. K. Niven.  Cluster-based reduced-order modelling of a mixing layer.  J. Fluid Mech., 754:365–414, 2014.
  * [47] Daniel Fernex, Bernd R Noack, and Richard Semaan.  Cluster-based network modeling—from snapshots to complex dynamical systems.  Science Advances, 7(25):eabf5006, 2021.
  * [48] Guy Y Cornejo Maceda, Yiqing Li, François Lusseyran, Marek Morzyński, and Bernd R Noack.  Stabilization of the fluidic pinball with gradient-enriched machine learning control.  Journal of Fluid Mechanics, 917, 2021.
  * [49] Dixia Fan, Liu Yang, Zhicheng Wang, Michael S Triantafyllou, and George Em Karniadakis.  Reinforcement learning for bluff body active flow control in experiments and simulations.  Proceedings of the National Academy of Sciences, 117(42):26091–26098, 2020.
  * [50] Siddhartha Verma, Guido Novati, and Petros Koumoutsakos.  Efficient collective swimming by harnessing vortices through deep reinforcement learning.  Proceedings of the National Academy of Sciences, 115(23):5849–5854, 2018.
  * [51] Dixia Fan, Gurvan Jodin, TR Consi, L Bonfiglio, Y Ma, LR Keyes, George E Karniadakis, and Michael S Triantafyllou.  A robotic intelligent towing tank for learning complex fluid-structure dynamics.  Science Robotics, 4(36), 2019.
  * [52] Jiaqing Kou and Weiwei Zhang.  Data-driven modeling for unsteady aerodynamics and aeroelasticity.  Progress in Aerospace Sciences, 125:100725, 2021.
  * [53] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in neural information processing systems, pages 1097–1105, 2012.
  * [54] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [55] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep Learning.  MIT Press, 2016.
  * [56] Xuhui Meng and George Em Karniadakis.  A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse pde problems.  Journal of Computational Physics, 401:109020, 2020.
  * [57] Kurt Hornik, Maxwell Stinchcombe, and Halbert White.  Multilayer feedforward networks are universal approximators.  Neural networks, 2(5):359–366, 1989.
  * [58] Kurt Hornik.  Approximation capabilities of multilayer feedforward networks.  Neural networks, 4(2):251–257, 1991.
  * [59] Xindong Wu, Vipin Kumar, J Ross Quinlan, Joydeep Ghosh, Qiang Yang, Hiroshi Motoda, Geoffrey J McLachlan, Angus Ng, Bing Liu, S Yu Philip, et al.  Top 10 algorithms in data mining.  Knowledge and Information Systems, 14(1):1–37, 2008.
  * [60] Leo Breiman.  Random forests.  Machine learning, 45(1):5–32, 2001.
  * [61] Bernhard Schölkopf and Alexander J Smola.  Learning with kernels: support vector machines, regularization, optimization, and beyond.  MIT press, 2002.
  * [62] Antoine Blanchard and Themistoklis Sapsis.  Bayesian optimization with output-weighted optimal sampling.  Journal of Computational Physics, 425:109901, 2021.
  * [63] Josh Bongard and Hod Lipson.  Automated reverse engineering of nonlinear dynamical systems.  Proceedings of the National Academy of Sciences, 104(24):9943–9948, 2007.
  * [64] Miles D Cranmer, Rui Xu, Peter Battaglia, and Shirley Ho.  Learning symbolic physics with graph networks.  arXiv preprint arXiv:1909.05862, 2019.
  * [65] Miles Cranmer, Alvaro Sanchez-Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho.  Discovering symbolic models from deep learning with inductive biases.  arXiv preprint arXiv:2006.11287, 2020.
  * [66] J. N. Kutz, S. L. Brunton, B. W. Brunton, and J. L. Proctor.  Dynamic Mode Decomposition: Data-Driven Modeling of Complex Systems.  SIAM, 2016.
  * [67] Robert Tibshirani.  Regression shrinkage and selection via the lasso.  Journal of the Royal Statistical Society. Series B (Methodological), pages 267–288, 1996.
  * [68] Peng Zheng, Travis Askham, Steven L Brunton, J Nathan Kutz, and Aleksandr Y Aravkin.  Sparse relaxed regularized regression: SR3.  IEEE Access, 7(1):1404–1423, 2019.
  * [69] Jaideep Pathak, Brian Hunt, Michelle Girvan, Zhixin Lu, and Edward Ott.  Model-free prediction of large spatiotemporally chaotic systems from data: a reservoir computing approach.  Physical review letters, 120(2):024102, 2018.
  * [70] Kai Li, Jiaqing Kou, and Weiwei Zhang.  Deep neural network for unsteady aerodynamic and aeroelastic modeling across multiple mach numbers.  Nonlinear Dynamics, 96(3):2157–2177, 2019.
  * [71] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley.  Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds.  arXiv preprint arXiv:1802.08219, 2018.
  * [72] Benjamin Kurt Miller, Mario Geiger, Tess E Smidt, and Frank Noé.  Relevance of rotationally equivariant convolutions for predicting molecular properties.  arXiv preprint arXiv:2008.08461, 2020.
  * [73] Rui Wang, Robin Walters, and Rose Yu.  Incorporating symmetry into deep dynamics models for improved generalization.  arXiv preprint arXiv:2002.03061, 2020.
  * [74] Simon Batzner, Tess E Smidt, Lixin Sun, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, and Boris Kozinsky.  Se (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials.  arXiv preprint arXiv:2101.03164, 2021.
  * [75] Samuel Greydanus, Misko Dzamba, and Jason Yosinski.  Hamiltonian neural networks.  Advances in Neural Information Processing Systems, 32:15379–15389, 2019.
  * [76] Marc Finzi, Ke Alexander Wang, and Andrew Gordon Wilson.  Simplifying hamiltonian and lagrangian neural networks via explicit constraints.  Advances in Neural Information Processing Systems, 33, 2020.
  * [77] Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho.  Lagrangian neural networks.  arXiv preprint arXiv:2003.04630, 2020.
  * [78] Yaofeng Desmond Zhong and Naomi Leonard.  Unsupervised learning of lagrangian dynamics from images for prediction and control.  Advances in Neural Information Processing Systems, 33, 2020.
  * [79] M Raissi, P Perdikaris, and GE Karniadakis.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  Journal of Computational Physics, 378:686–707, 2019.
  * [80] Guofei Pang, Lu Lu, and George Em Karniadakis.  fpinns: Fractional physics-informed neural networks.  SIAM Journal on Scientific Computing, 41(4):A2603–A2626, 2019.
  * [81] Liu Yang, Dongkun Zhang, and George Em Karniadakis.  Physics-informed generative adversarial networks for stochastic differential equations.  SIAM Journal on Scientific Computing, 42(1):A292–A317, 2020.
  * [82] Zhiping Mao, Ameya D Jagtap, and George Em Karniadakis.  Physics-informed neural networks for high-speed flows.  Computer Methods in Applied Mechanics and Engineering, 360:112789, 2020.
  * [83] George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang.  Physics-informed machine learning.  Nature Reviews Physics, 3(6):422–440, 2021.
  * [84] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al.  Relational inductive biases, deep learning, and graph networks.  arXiv preprint arXiv:1806.01261, 2018.
  * [85] Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia.  Learning to simulate complex physics with graph networks.  In International Conference on Machine Learning, pages 8459–8468. PMLR, 2020.
  * [86] Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis.  Learning nonlinear operators via deeponet based on the universal approximation theorem of operators.  Nature Machine Intelligence, 3(3):218–229, 2021.
  * [87] S Beetham and J Capecelatro.  Formulating turbulence closures using sparse regression with embedded form invariance.  Physical Review Fluids, 5(8):084611, 2020.
  * [88] Sarah Beetham, Rodney O Fox, and Jesse Capecelatro.  Sparse identification of multiphase turbulence closures for coupled fluid–particle flows.  Journal of Fluid Mechanics, 914, 2021.
  * [89] Martin Schmelzer, Richard P Dwight, and Paola Cinnella.  Discovery of algebraic reynolds-stress models using sparse symbolic regression.  Flow, Turbulence and Combustion, 104(2):579–603, 2020.
  * [90] Jiaqing Kou and Weiwei Zhang.  A hybrid reduced-order framework for complex aeroelastic simulations.  Aerospace science and technology, 84:880–894, 2019.
  * [91] Maziar Raissi, Alireza Yazdani, and George Em Karniadakis.  Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations.  Science, 367(6481):1026–1030, 2020.
  * [92] Xuan Zhao, Lin Du, Xuhao Peng, Zichen Deng, and Weiwei Zhang.  Research on refined reconstruction method of airfoil pressure based on compressed sensing.  Theoretical and Applied Mechanics Letters, page 100223, 2021.
  * [93] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar.  Fourier neural operator for parametric partial differential equations.  arXiv preprint arXiv:2010.08895, 2020.
  * [94] Hugo Frezat, Guillaume Balarac, Julien Le Sommer, Ronan Fablet, and Redouane Lguensat.  Physical invariance in neural networks for subgrid-scale scalar flux modeling.  Physical Review Fluids, 6(2):024607, 2021.
  * [95] Kookjin Lee and Kevin T Carlberg.  Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders.  Journal of Computational Physics, 404:108973, 2020.
  * [96] B. R. Noack, K. Afanasiev, M. Morzynski, G. Tadmor, and F. Thiele.  A hierarchy of low-dimensional models for the transient and post-transient cylinder wake.  Journal of Fluid Mechanics, 497:335–363, 2003.
  * [97] Peter Benner, Serkan Gugercin, and Karen Willcox.  A survey of projection-based model reduction methods for parametric dynamical systems.  SIAM review, 57(4):483–531, 2015.
  * [98] Clarence W Rowley and Scott TM Dawson.  Model reduction for flow analysis and control.  Annual Review of Fluid Mechanics, 49:387–417, 2017.
  * [99] Alan A Kaptanoglu, Jared L Callaham, Christopher J Hansen, Aleksandr Aravkin, and Steven L Brunton.  Promoting global stability in data-driven models of quadratic nonlinear dynamics.  arXiv preprint arXiv:2105.01843, 2021.
  * [100] J.-C. Loiseau and S. L. Brunton.  Constrained sparse Galerkin regression.  Journal of Fluid Mechanics, 838:42–67, 2018.
  * [101] N Benjamin Erichson, Michael Muehlebach, and Michael W Mahoney.  Physics-informed autoencoders for lyapunov-stable fluid flow prediction.  arXiv preprint arXiv:1905.10866, 2019.
  * [102] J.-C. Loiseau, B. R. Noack, and S. L. Brunton.  Sparse reduced-order modeling: sensor-based dynamics to full-state estimation.  Journal of Fluid Mechanics, 844:459–490, 2018.
  * [103] Jean-Christophe Loiseau.  Data-driven modeling of the chaotic thermal convection in an annular thermosyphon.  Theoretical and Computational Fluid Dynamics, 34(4):339–365, 2020\.
  * [104] Nan Deng, Bernd R Noack, Marek Morzyński, and Luc R Pastur.  Low-order model for successive bifurcations of the fluidic pinball.  Journal of fluid mechanics, 884, 2020.
  * [105] Nan Deng, Bernd R Noack, Marek Morzyński, and Luc R Pastur.  Galerkin force model for transient and post-transient dynamics of the fluidic pinball.  Journal of Fluid Mechanics, 918, 2021.
  * [106] M. Schlegel and B. R. Noack.  On long-term boundedness of galerkin models.  Journal of Fluid Mechanics, 765:325–352, 2015.
  * [107] Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu.  Towards physics-informed deep learning for turbulent flow prediction.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1457–1466, 2020.
  * [108] Michael Grant, Stephen Boyd, and Yinyu Ye.  Cvx: Matlab software for disciplined convex programming, 2008.
  * [109] Stephen Boyd and Lieven Vandenberghe.  Convex optimization.  Cambridge university press, 2009.
  * [110] SBj Pope.  A more general effective-viscosity hypothesis.  Journal of Fluid Mechanics, 72(2):331–340, 1975.