# Efficient Visual Pretraining with Contrastive Detection

Olivier J. Hénaff Skanda Koppula Jean-Baptiste Alayrac
Aaron van den Oord Oriol Vinyals João Carreira
DeepMind, London, UK

###### Abstract

Self-supervised pretraining has been shown to yield powerful representations
for transfer learning. These performance gains come at a large computational
cost however, with state-of-the-art methods requiring an order of magnitude
more computation than supervised pretraining. We tackle this computational
bottleneck by introducing a new self-supervised objective, contrastive
detection, which tasks representations with identifying object-level features
across augmentations. This objective extracts a rich learning signal per
image, leading to state-of-the-art transfer accuracy on a variety of
downstream tasks, while requiring up to 10$\times$ less pretraining. In
particular, our strongest ImageNet-pretrained model performs on par with SEER,
one of the largest self-supervised systems to date, which uses 1000$\times$
more pretraining data. Finally, our objective seamlessly handles pretraining
on more complex images such as those in COCO, closing the gap with supervised
transfer learning from COCO to PASCAL.

##  1 Introduction

Since the AlexNet breakthrough on ImageNet, transfer learning from large
labeled datasets has become the dominant paradigm in computer vision [34, 50].
While recent advances in self-supervised learning have alleviated the
dependency on labels for pretraining, they have done so at a tremendous
computational cost, with state-of-the-art methods requiring an order of
magnitude more computation than supervised pretraining [7, 10, 21]. Yet the
promise of self-supervised learning is to harness massive unlabeled datasets,
making its computational cost a critical bottleneck.

Figure 1: Efficient self-supervised pretraining with DetCon. Self-supervised
pretraining with SimCLR [9] matches the transfer performance of supervised
pretraining only when given 10$\times$ more training iterations. Our proposed
DetCon objective surpasses both, while requiring 5$\times$ less computation
than SimCLR. Transfer performance is measured by fine-tuning the
representation on the COCO dataset for 12 epochs, using a Mask-RCNN.

In this work, we aim to alleviate the computational burden of self-supervised
pretraining. To that end we introduce contrastive detection, a new objective
which maximizes the similarity of object-level features across augmentations.
The benefits of this objective are threefold. First, it extracts separate
learning signals from all objects in an image, enriching the information
provided by each training example for free—object-level features are simply
obtained from intermediate feature arrays. Second, it provides a larger and
more diverse set of negative samples to contrast against, which also
accelerate learning. Finally, this objective is well suited to learning from
complex scenes with many objects, a pretraining domain that has proven
challenging for self-supervised methods.

We identify approximate object-based regions in the image through the use of
unsupervised segmentation algorithms. Perceptual grouping [32, 41]—the idea
that low and mid-level regularities in the data such as color, orientation and
texture allow for approximately parsing a scene into connected surfaces or
object parts—has long been theorized to be a powerful prior for vision [22,
40, 55]. We leverage these priors by grouping local feature vectors
accordingly, and applying our contrastive objective to each object-level
feature separately. We investigate the use of several unsupervised, image-
computable masks [17, 2], and find our objective to work well despite their
inaccuracies.

We test the ability of our objective to quickly learn transferable
representations by applying it to the ImageNet dataset and measuring its
transfer performance on challenging tasks such as COCO detection and instance
segmentation, semantic segmentation on PASCAL and Cityscapes, and NYU depth
estimation. Compared to representations obtained from recent self-supervised
objectives such as SimCLR and BYOL [9, 21], our representations are more
accurate and can be obtained with much less training time. We also find this
learning objective to better handle images of more complex scenes, bridging
the gap with supervised transfer from the COCO dataset. In summary, we make
the following contributions:

1\. We formulate a new contrastive objective which maximizes the similarity
across augmentations of all objects in a scene, where object regions are
provided by a simple, unsupervised heuristic.

2\. We find this objective to alleviate the computational burden of self-
supervised transfer learning, reducing by up to 10$\times$ the computation
required to match supervised transfer learning from ImageNet. Longer training
schedules lead to state-of-the-art transfer to COCO detection and instance
segmentation, and our best model matches the very recent state-of-the-art
self-supervised system SEER [20] which is trained on 1000$\times$ more—if less
curated—images.

3\. When transferring from complex scene datasets such as COCO, our method
closes the gap with a supervised model which learns from human-annotated
segmentations.

4\. Finally, we assess to what extent the existing contrastive learning
paradigm could be simplified in the presence of high quality image
segmentations, raising questions and opening avenues for future work.

##  2 Related work

Transferring the knowledge contained in one task and dataset to solve other
downstream tasks (i.e. transfer learning) has proven very successful in a
range of computer vision problems [19, 39]. While early work focused on
improving the pretraining architecture [27, 51] and data [52], recent work in
self-supervised learning has focused on the choice of pretraining objective
and task. Early self-supervised pretraining typically involved image
restoration, including denoising [59], inpainting [46], colorization [65, 36],
egomotion prediction [1], and more [15, 43, 66]. Higher-level pretext tasks
have also been studied, such as predicting context [13], orientation [18],
spatial layouts [44], temporal ordering [42], and cluster assignments [5].

Contrastive objectives, which maximize the similarity of a representation
across views, while minimizing its similarity with distracting negative
samples, have recently gained considerable traction [23]. These views have
been defined as local and global crops [29, 4, 56, 28] or different input
channels [53]. Instance-discrimination approaches generate global, stochastic
views of an image through data-augmentation, and maximize their similarity
relative to marginally sampled negatives [9, 14, 16, 24, 62], although the
need for negative samples has recently been questioned [12, 21]. While the
benefits of instance-discrimination approaches have mostly been limited to
pretraining from simple datasets such as ImageNet, clustering-based
pretraining has proven very successful in leveraging large amounts of
uncurated images for transfer learning [3, 6, 7, 20, 31].

While most work has focused on learning whole-image representations, there has
been increasing interest in learning local descriptors that are more relevant
for downstream tasks such as detection and segmentation. Examples of such work
include the addition of auxiliary losses [54], architectural components [48],
or both [63]. While perceptual grouping has long been used for representation
learning, often relying on coherent motion in videos [37, 45, 60], it has only
recently been combined with contrastive learning [30, 58, 67]. Most related to
our work are [58, 67] that also leverage image segmentations for self-
supervised learning, although both differ from ours in that they learn
backbones that are specialized for semantic segmentation and employ different
loss functions. Although these works arrive at impressive unsupervised
segmentation accuracy, neither report gains in pretraining efficiency for
transfer learning tasks such as COCO detection and instance segmentation,
which we study next.

##  3 Method

We introduce a new contrastive objective which maximizes the similarity across
views of local features which represent the same object (Figure 2). In order
to isolate the benefit of these changes, we make the deliberate choice of re-
using elements of existing contrastive learning frameworks where possible. To
test the generality of our approach, we derive two variants, DetConS and
DetConB, based on two recent self-supervised baselines, SimCLR [9] and BYOL
[21] respectively. We adopt the data augmentation procedure and network
architecture from these methods while applying our proposed contrastive
detection loss to each.

###  3.1 The contrastive detection framework

Figure 2: The contrastive detection method. We identify object-based regions
with approximate, image-computable segmentation algorithms (bottom). These
masks are carried through two stochastic data augmentations and a
convolutional feature extractor, creating groups of feature vectors in each
view (middle). The contrastive detection objective then pulls together pooled
feature vectors from the same mask (across views) and pushes apart features
from different masks and different images (top).

Data augmentation. Each image is randomly augmented twice, resulting in two
images: ${\bm{x}},{\bm{x}}^{\prime}$. DetConS and DetConB adopt the
augmentation pipelines of SimCLR and BYOL respectively, which roughly consist
of random cropping, flipping, blurring, and point-wise color transformations.
We refer the reader to appendix A.1 for more details. In all cases, images are
resized to 224$\times$224 pixel resolution.

In addition, we compute for each image a set of masks which segment the image
into different components. As described in Section 3.2, these masks can be
computed using efficient, off-the-shelf, unsupervised segmentation algorithms.
If available, human-annotated segmentations can also be used. In any case, we
transform each mask (represented as a binary image) using the same cropping
and resizing as used for the underlying RGB image, resulting in two sets of
masks $\\{{\bm{m}}\\},\\{{\bm{m}}^{\prime}\\}$ which are aligned with the
augmented images ${\bm{x}},{\bm{x}}^{\prime}$ (see Figure 2, augmented views).

Figure 3: Example masks used by the DetCon model. 1 column: random images from
the COCO training set. 2 column: masks based on spatial proximity only. Global
masks (top) are implicitly used by methods such as SimCLR, MoCo, and BYOL. 3
column: image-computable masks obtained from the Felzenszwalb-Huttenlocher
(FH, [17]) algorithm, with $s=500$. 4 column: image-computable masks inferred
using Multiscale Combinatorial Grouping (MCG) [2]. 5 column: “oracle” masks
used to assess potential improvements from higher-quality segmentations.

Architecture. We use a convolutional feature extractor $f$ to encode each
image with a spatial map of hidden vectors: ${\bm{h}}=f({\bm{x}})$ where
${\bm{h}}\in{\mathbb{R}}^{H\times W\times D}$. We use the output of a standard
ResNet-50 encoder [27], before the final mean-pooling layer, such that hiddens
form a 7$\times$7 grid of 2048-dimensional vectors ${\bm{h}}[i,j]$. For every
mask ${\bm{m}}$ associated with the image, we compute a mask-pooled hidden
vector

${\bm{h}}_{\bm{m}}=\frac{1}{\sum_{i,j}m_{i,j}}\sum_{i,j}m_{i,j}\
{\bm{h}}[i,j],$

having spatially downsampled the binary mask to a 7$\times$7 grid with average
pooling. We then transform each of these vectors with a two-layer MLP,
yielding non-linear projections
${\bm{z}}_{\bm{m}}=g({\bm{h}}_{\bm{m}})\in{\mathbb{R}}^{d}$.

For DetConS we process both views with the same encoder $f_{\theta}$ and
projection network $g_{\theta}$, where $\theta$ are the learned parameters.
For DetConB one view is processed with $f_{\theta}$ and $g_{\theta}$ and the
other with $f_{\xi}$ and $g_{\xi}$, where $\xi$ is an exponential moving
average of $\theta$. The first view is further transformed with a prediction
network $q_{\theta}$. Here again we reuse the details of SimCLR and BYOL for
DetConS and DetConB respectively in the definition of the projection and
prediction networks (see appendix A.2). In summary, we represent each view and
mask as latents ${\bm{v}}_{\bm{m}}$ and
${\bm{v}}^{\prime}_{{\bm{m}}^{\prime}}$ where

${\bm{v}}_{\bm{m}}=g_{\theta}({\bm{h}}_{{\bm{m}}}),\quad{\bm{v}}^{\prime}_{{\bm{m}}^{\prime}}=g_{\theta}({\bm{h}}^{\prime}_{{\bm{m}}^{\prime}})$

for DetConS and

${\bm{v}}_{\bm{m}}=q_{\theta}\circ
g_{\theta}({\bm{h}}_{{\bm{m}}}),\quad{\bm{v}}^{\prime}_{{\bm{m}}^{\prime}}=g_{\xi}({\bm{h}}^{\prime}_{{\bm{m}}^{\prime}})$

for DetConB. We rescale all latents with a temperature hyperparameter $\tau$,
such that their norm is equal to $1/\sqrt{\tau}$, with $\tau=0.1$. Note that
for downstream tasks, we retain only the feature extractor $f_{\theta}$ and
discard all other parts of the network (the prediction and projection heads,
as well as any exponential moving averages).

Objective: contrastive detection. Let
${\bm{v}}_{\bm{m}},{\bm{v}}^{\prime}_{{\bm{m}}^{\prime}}$ be the latents
representing masks ${\bm{m}},{\bm{m}}^{\prime}$ in the views
${\bm{x}},{\bm{x}}^{\prime}$. The contrastive loss function

$\ell_{{\bm{m}},{\bm{m}}^{\prime}}=-\log\frac{\exp({\bm{v}}_{\bm{m}}{\cdot}{\bm{v}}^{\prime}_{{\bm{m}}^{\prime}})}{\exp({\bm{v}}_{\bm{m}}{\cdot}{\bm{v}}^{\prime}_{{\bm{m}}^{\prime}})+\sum_{\bm{n}}\exp({\bm{v}}_{\bm{m}}{\cdot}{\bm{v}}_{{\bm{n}}})}$
(1)

defines a prediction task: having observed the projection ${\bm{v}}_{\bm{m}}$,
learn to recognize the latent ${\bm{v}}^{\prime}_{{\bm{m}}^{\prime}}$ in the
presence of negative samples $\\{{\bm{v}}_{\bm{n}}\\}$. We include negative
samples from different masks in the image and different images in the batch.
Note that we make no assumptions regarding these masks, allowing negative
masks to overlap with the positive one.

A natural extension of this loss would be to jointly sample paired masks
${\bm{m}},{\bm{m}}^{\prime}$ which correspond to the same region in the
original image, and maximize the similarity of features representing them

$\mathcal{L}=\mathbb{E}_{({\bm{m}},{\bm{m}}^{\prime})\sim\mathcal{M}}\
\ell_{{\bm{m}},{\bm{m}}^{\prime}}.$ (2)

We make a few practical changes to this objective. First, in order to
facilitate batched computation we randomly sample at each iteration a set of
16 (possibly redundant) masks from the variable-sized sets of masks
$\\{{\bm{m}}\\}$ and $\\{{\bm{m}}^{\prime}\\}$. Second, we densely evaluate
the similarity between all pairs of masks and all images, such that each image
contributes 16 negative samples to the set $\\{{\bm{v}}_{{\bm{n}}}\\}$ in
equation (1), rather than a single one. We aim to makes these negatives as
diverse as possible by choosing masks that roughly match different objects in
the scene (Section 3.2). Finally, we mask out the loss to only maximize the
similarity of paired locations, allowing us to handle cases where a mask is
present in one view but not another (see Figure 2). Together, these simple
modifications bring us to the DetCon objective:

$\mathcal{L}=\sum_{{\bm{m}}}\sum_{{\bm{m}}^{\prime}}\mathds{1}_{{\bm{m}},{\bm{m}}^{\prime}}\ell_{{\bm{m}},{\bm{m}}^{\prime}}$
(3)

where the binary variable $\mathds{1}_{{\bm{m}},{\bm{m}}^{\prime}}$ indicates
whether the masks ${\bm{m}},{\bm{m}}^{\prime}$ correspond to the same
underlying region.

Optimization. When pretraining on ImageNet we adopt the optimization details
of SimCLR and BYOL for training DetConS and DetConB respectively. When
pretraining on COCO we make minor changes to the learning schedule to
alleviate overfitting (see appendix A.3).

Computational cost. The computational requirements of self-supervised learning
are largely due to forward and backward passes through the convolutional
backbone. For the typical ResNet-50 architecture applied to
224$\times$224-resolution images, a single forward pass requires approximately
4B FLOPS. The additional projection head in SimCLR and DetConS requires an
additional 4M FLOPS. Since we forward 16 hidden vectors through the projection
head instead of 1, we increase the computational cost of the forward pass by
67M FLOPS, less than 2% of total. Together with the added complexity of the
contrastive loss, this increase is 5.3% for DetConS and 11.6% for DetConB (see
appendix A.2). Finally, the cost of computing image segmentations is
negligible because they can be computed once and reused throughout training.
Therefore the increase in complexity of our method relative to the baseline is
sufficiently small for us to interchangeably refer to “training iterations”
and “computational cost”.

###  3.2 Unsupervised mask generation

To produce masks required by the DetCon objective, we investigate several
segmentation procedures, from simple spatial heuristics to graph-based
algorithms from the literature.

Spatial heuristic. The simplest segmentation we consider groups locations
based on their spatial proximity only. Specifically, we divide the image into
an $n\times n$ grid of non-overlapping, square sub-regions (Figure 3, 2
column). Note that when using a single, global mask ($n=1$), the DetConS
objective reverts to SimCLR.

Image-computable masks: FH. We also consider the Felzenszwalb-Huttenlocher
algorithm [17], a classic segmentation procedure which iteratively merges
regions using pixel-based affinity (Figure 3, 3 column). We generate a diverse
set of masks by varying two hyperparameters, the scale $s$ and minimum cluster
size $c$, using $s\in\\{500,1000,1500\\}$ and $c=s$ when training on COCO and
$s=1000$ when training on ImageNet.

Figure 4: Efficient ImageNet pretraining with DetConS. We pretrain networks
with SimCLR, DetConS, or supervised learning on ImageNet for different numbers
of epochs, and fine-tune them for COCO detection and instance segmentation
(for 12 epochs), semantic segmentation on PASCAL or Cityscapes, or depth
estimation on NYU v2. DetConS outperforms SimCLR, with up to 10$\times$ less
pretraining.

Image-computable masks: MCG. Multiscale Combinatorial Grouping [2] is a more
sophisticated algorithm which groups superpixels into many overlapping object
proposal regions [8], guided by mid-level classifiers (Figure 3, 4 column).
For each image we use 16 MCG masks with the highest scores. Note that the fact
that these masks can overlap is supported by our formulation.

Human annotated masks. Throughout this work we consider the benefits afforded
by the use of the unsupervised masks detailed above. In the final section, we
ask whether higher quality masks (provided by human annotators; Figure 3, 5
column) can improve our results.

| Detection | Instance Segmentation | Semantic Segmentation | Depth Estimation
---|---|---|---|---
| COCO | COCO | PASCAL | Cityscapes | NYU v2
Pretrain epochs |  300 |  1000 |  300 |  1000 |  300 |  1000 |  300 |  1000 |  100 |  1000
BYOL |  41.2 |  41.6 |  37.1 |  37.2 |  74.7 |  75.7 |  73.4 |  74.6 |  83.7 |  84.2
DetConB |  42.0 |  42.7 |  37.8 |  38.2 |  75.6 |  77.3 |  75.1 |  77.0 |  85.1 |  86.3
Efficiency Gain | $>3\times$ | $>3\times$ | $\approx 3\times$ | $>3\times$ | $>10\times$
Table 1: Efficient ImageNet pretraining with DetConB. We pretrain networks on
ImageNet with BYOL or DetConB, and fine-tune them for COCO detection and
instance segmentation (for 12 epochs), semantic segmentation on PASCAL or
Cityscapes, or depth estimation on NYU v2. DetConB outperforms BYOL, with up
to 10$\times$ less pretraining (colors highlight gains in pretraining
efficiency).

###  3.3 Evaluation protocol

Having trained a feature extractor in an unsupervised manner, we evaluate the
quality of the representation by fine-tuning it for object detection and
instance segmentation on COCO, segmentatic segmentation on PASCAL and
Cityscapes, and depth estimation on NYU v2.

Object detection and instance segmentation. We use the pretrained network to
initialize the feature extractor of a Mask-RCNN [26] equipped with feature
pyramid networks [38] and cross-replica batch-norm [47]. We adopt the Cloud
TPU
implementation111https://github.com/tensorflow/tpu/tree/master/models/official/detection
and use it without modification. We fine-tune the entire model on the COCO
train2017 set, and report bounding-box AP (AP${}^{\text{bb}}_{\text{}}$) and
mask AP (AP${}^{\text{mk}}_{\text{}}$) on the val2017 set. We use two standard
training schedules: 12 epochs and 24 epochs [24].

Semantic segmentation. Following [24] we use our network to initialize the
backbone of a fully-convolutional network [39]. For PASCAL, we fine-tune on
the train_aug2012 set for 45 epochs and report the mean intersection over
union (mIoU) on the val2012 set. For Cityscapes, we finetune on the train_fine
set for 160 epochs and evaluate on the val_fine set.

Depth estimation. Following [21] we stack the deconvolutional network from
[35] on top of our feature extractor, and fine-tune on the NYU v2 dataset. We
report accuracy as the percentage of errors below 1.25 (pct$<$1.25).

##  4 Experiments

Our main self-supervised learning experiments employ FH masks, because as we
will show, with DetCon they outperform simple spatial heuristics and approach
the performance of MCG masks while being fast and easy to apply to large
datasets such as ImageNet, given their availability in scikit-image [57].

###  4.1 Transfer learning from ImageNet

We first study whether the DetCon objective improves the pretraining
efficiency of transfer learning from ImageNet.

Pretraining efficiency. We train SimCLR and DetConS models on ImageNet for
100, 200, 500 and 1000 epochs, and transfer them to several datasets and
tasks. Across all downstream tasks and pretraining regimes, DetConS
substantially outperforms SimCLR (Figures 1 and 4, blue and orange curves).
When fine-tuning on COCO, the performance afforded by 1,000 epochs of SimCLR
pretraining is surpassed by only 200 epochs of DetConS pretraining (i.e. a
5$\times$ gain in pretraining efficiency). We found similar results when
transferring to other downstream tasks: DetConS yields a 2$\times$ gain in
pretraining efficiency for PASCAL semantic segmentation (Figure 4, 2 column),
and 10$\times$ gains for Cityscapes semantic segmentation and NYU depth
prediction. (Figure 4, 3, and 4 columns).

We also evaluated the transfer performance of a supervised ResNet-50 trained
on ImageNet (Figures 1 and 4, red curve). While supervised pretraining
performs well with small computational budgets (e.g. 100 pretraining epochs),
it quickly saturates, indicating that ImageNet labels only partially inform
downstream tasks. This is emphasized for Cityscapes semantic segmentation and
NYU depth prediction, which represent a larger shift in the domain and task.

From BYOL to DetConB. How general is DetCon? We tested this by comparing
DetConB to the BYOL framework, upon which it is based. We adopt the underlying
framework details (e.g. data-augmentation, architecture, and optimization)
without modification, possibly putting the DetCon objective at a disadvantage.
Despite this, DetConB outperforms BYOL across pretaining budgets and
downstream tasks. In particular, DetConB yields a 3$\times$ gain in
pretraining efficiency when transferring to COCO, PASCAL, and Cityscapes
detection and segmentation, and a 10$\times$ gain when transferring to NYU
depth prediction (Table 1).

Comparison with prior art. We now compare to other works in self-supervised
transfer learning, and use fully-trained DetConS and DetConB models for the
comparison. Here we focus on transfer to COCO as it is more widely studied.
Note that other methods use a slightly different implementation of the Mask-
RCNN [61] however their results for supervised ImageNet pretraining and SimCLR
match our own [54, 63], enabling a fair comparison. Table 2 shows that DetCon
outperforms all other methods for supervised and self-supervised transfer
learning.

|  Fine-tune 1$\times$ |  Fine-tune 2$\times$
---|---|---
method |  AP${}^{\text{bb}}_{\text{~{}}}$ |  AP${}^{\text{mk}}_{\text{~{}}}$ |  AP${}^{\text{bb}}_{\text{~{}}}$ |  AP${}^{\text{mk}}_{\text{~{}}}$
Supervised |  39.6 |  35.6 |  41.6 |  37.6
VADeR [48] |  39.2 |  35.6 |  - |  -
MoCo [24] |  39.4 |  35.6 |  41.7 |  37.5
SimCLR [9] |  39.7 |  35.8 |  41.6 |  37.4
MoCo v2 [11] |  40.1 |  36.3 |  41.7 |  37.6
InfoMin [54] |  40.6 |  36.7 |  42.5 |  38.4
PixPro [63] |  41.4 |  - |  - |  -
BYOL [21] |  41.6 |  37.2 |  42.4 |  38.0
SwAV [7] |  41.6 |  37.8 |  - |  -
DetConS |  41.8 |  37.4 |  42.9 |  38.1
DetConB |  42.7 |  38.2 |  43.4 |  38.7
Table 2: Comparison to prior art: all methods are pretrained on ImageNet then
fined-tuned on COCO for 12 epochs (1$\times$ schedule) or 24 epochs (2$\times$
schedule).  Figure 5: Scaling DetCon to larger models. We pretrain ResNet-50,
ResNet-101, ResNet-152, and ResNet-200 feature extractors on ImageNet using
supervised learning, SimCLR, BYOL, or DetConB and fine-tune them on COCO for
12 epochs.

Scaling model capacity. Prior works in self-supervised learning have been
shown to scale very well with model capacity [14, 33, 9]. Could the gains
afforded by DetCon disappear with larger models? We trained SimCLR, BYOL, and
DetConB models on ImageNet, using ResNet-101, -152, and -200 feature
extractors instead of ResNet-50. Figure 5 and Table A.1 show that DetCon
continues to outperform other methods in this higher-capacity regime.

We went a step further and trained a ResNet-200 with a 2$\times$ width
multiplier [33], containing 250M parameters. Surprisingly, despite only being
trained on ImageNet, this model’s transfer performance matches that of a very
recently proposed large-scale self-supervised model, SEER [20], having 693M
parameters and trained on 1000$\times$ more data (Table 3). While the
comparison is imperfect (large-scale data is necessarily more noisy), it
highlights the potential of improvements from the self-supervised learning
objective alone.

pretrain |  Data |  Params |  AP${}^{\text{bb}}_{\text{~{}}}$ |  AP${}^{\text{mk}}_{\text{~{}}}$
---|---|---|---|---
Supervised [20] |  IN-1M |  250 M |  45.9 |  41.0
SEER [20] |  IG-1B |  693 M |  48.5 |  43.2
DetConB |  IN-1M |  250 M |  48.9 |  43.0
Table 3: Comparison to large-scale transfer learning: all methods pretrain a
backbone and transfer to COCO detection and instance segmentation using a
Mask-RCNN. SEER trains on a billion Instagram images whereas DetConS trains on
ImageNet (1.3 million images). SEER and the supervised baseline use the recent
RegNet architecture [49], whereas DetConS uses a generic ResNet-200 (2$\times$
width). Despite this, DetCon pretraining matches the performance of large-
scale SEER pretraining.

###  4.2 Transfer learning from COCO

We next investigate the ability of the DetCon objective to handle complex
scenes with multiple objects. For this we pretrain on the COCO dataset and
compare to SimCLR.

Training efficiency. We train SimCLR and DetConS for a range of schedules
(324–5184 epochs), and transfer all models to semantic segmentation on PASCAL.
We find DetConS to outperform SimCLR across training budgets (Figure 6). As
before, the maximum accuracy attained by SimCLR is reached with 4$\times$ less
pretraining time.

Surpassing supervised transfer from COCO. We also evaluated the transfer
performance of representations trained on COCO in a supervised manner.
Specifically, we trained a Mask-RCNN with a long schedule (108 epochs, a
“9$\times$” schedule), and use the learned feature extractor (a ResNet-50, as
for SimCLR and DetCon pretraining) as a representation for PASCAL
segmentation. Unlike SimCLR, DetCon pretraining surpasses the performance of
this fully supervised baseline (Figure 6).

Figure 6: Efficient transfer from COCO. We pretrain representations using
SimCLR or DetConS on COCO for different numbers of epochs, and transfer to
PASCAL semantic segmentation by fine-tuning them for 45 epochs. Figure 7:
Effect of type of masks used in DetCon objective. We train DetCon models on
COCO using unsupervised masks (blue), or the ground-truth COCO masks (grey).
Using a single, global mask (i.e. a “1$\times$1” grid) is equivalent to SimCLR
(orange). We compute the Average Best Overlap (ABO) by measuring the IoU
between each ground-truth mask and the closest pretraining mask, and averaging
over all ground truth instances and images (x-axis). We evaluate the accuracy
of each model on COCO detection using the frozen-feature paradigm (y-axis).

###  4.3 Ablations and analysis

We now dissect the components of the DetCon objective and assess the benefits
of each. For this we pretrain on COCO as it contains complex scenes with many
objects and associated ground-truth masks, allowing us to measure the impact
of segmenting them accurately. We evaluate learned representations with a
frozen-feature analysis, in which the feature extractor is kept fixed while we
train the other layers of a Mask-RCNN also on COCO. This controlled setting is
analogous222But note that the Mask-RCNN contains several non-linear layers due
to the additional complexity of the output space relative to classification.
to the linear classification protocol used to evaluate the quality of self-
supervised representations for image recognition [13, 15, 45, 65]

What makes good masks? The DetCon objective can be used with a variety of
different image segmentations, which ones lead to the best representation? We
first consider spatial heuristics which partition the image into 2$\times$2,
5$\times$5, or 10$\times$10 grids, a 1$\times$1 grid being equivalent to using
the SimCLR objective. We find downstream performance increases with finer
grids, a 5$\times$5 grid being optimal (Figure 7).

Next we consider image-computable FH and MCG masks, both of which outperform
the spatial heuristic masks, MCG masks leading to slightly better
representations. Interestingly, the quality of the representation correlates
very well with the overlap between pretraining masks and ground-truth—the
better each ground truth object is covered by some mask, the better DetCon
performs.

model |  masks |  #latents |  AP${}^{\text{bb}}_{\text{~{}}}$ |  AP${}^{\text{mk}}_{\text{~{}}}$
---|---|---|---|---
SimCLR |  global |  1 |  31.6 |  29.2
(a) |  global |  16 |  31.5 |  29.3
(b) |  FH |  1 |  31.2 |  28.8
DetConS |  FH |  16 |  33.4 |  30.6
Table 4: Ablation: from SimCLR to DetConS. We pretrain on COCO and evaluate
frozen feature accuracy also on COCO. masks: specifies whether hidden vectors
are pooled globally, or within individual FH masks. #latents: number of masks.

Contrastive detection vs contrastive recognition. How does the DetCon
objective benefit from these image segmentations? We assess the impact of each
of its components by incrementally adding them to the SimCLR framework. As
mentioned previously, we recover SimCLR when using a single, global mask in
the DetCon objective. As a sanity check, we verify that duplicating this mask
several times and including the resulting (identical) features in the DetCon
objective makes no difference in the quality of the representation (Table 4,
row a). Interestingly, using FH masks but only sampling a single mask per
image slightly deteriorates performance, presumably because the model only
learns from part of the image at every iteration (Table 4, row b). By densely
sampling object regions DetConS learns from the entire image, while also
benefiting from a diverse set of positive and negative samples, resulting in
increased detection and segmentation accuracy (Table 4, final row).

###  4.4 What if segmentation were solved?

The DetCon objective function leads to fast transfer learning and strong
performance despite using fairly approximate segmentation masks. In Section
4.3 we found higher quality segmentations (such as those computed using MCG,
or obtained from human annotators) to improve representational quality. How
might we improve the learning objective given more accurate segmentations? We
assessed this question by revisiting our design choices for the contrastive
objective, when given ground-truth masks from the COCO dataset as opposed to
the approximate FH masks.

Figure 8:  Better segmentations benefit from higher resolutions. We pretrain
backbone networks on COCO using SimCLR and DetConS (with FH or GT masks) at
various resolutions. We report frozen-feature performance with a fixed
resolution of 1024$\times$1024.

Scaling image resolution. We hypothesized that higher image resolutions might
enable the network to benefit more from these more informative segmentations.
To preserve fine-grained information we sample local features from within each
mask and optimize them using the DetCon objective. We pretrain SimCLR and
DetConS models equipped with FH or ground-truth (GT) masks, given
384$\times$384 or 512$\times$512 resolution images. While DetCon with FH masks
only modestly benefited and SimCLR’s performance deteriorated with high-
resolution images, DetCon with GT masks improves substantially (Figure 8).
Note that this is solely due to an improved representation quality; the image
resolution used for downstream evaluation is maintained at 1024$\times$1024
for all models.

Revisiting the contrastive framework. Finally, we asked whether the current
contrastive learning paradigm—which utilizes large numbers of negatives and
predictions across stochastic augmentations—remains optimal in the context of
the DetCon objective with high-quality segmentations.

Are large numbers of negative samples necessary? Not with high-quality masks.
When dividing the total number of negative samples by 128 (by only gathering
negatives from within a worker) the performance of DetCon with FH masks drops
(Table 5, row a), consistently with other contrastive learning frameworks [9,
24]. In contrast, DetConS using GT masks improves despite this limitation.

Are positive pairs sampled across augmented views necessary? Not with high-
quality masks. We run DetCon models while sampling a single augmentation for
each image and maximizing the similarity of mask-based features within this
view. Here again, the DetCon objective suffers from this handicap when using
approximate FH masks, but not with high-quality segmentations (Table 5, row
b).

How can this be? One interpretation is that other images give us clean
negative examples because images in COCO depict different scenes. However it
appears that negatives from the same image provide a stronger learning signal
(in that they share features such as lighting, background, etc) as long as
they are clean—i.e., we are not pushing features from the same object apart.
Positives from the same image are also at least as good as those across
augmentations if again they are clean—i.e., we are not pulling together
features from different objects.

|  |  | Masks
---|---|---|---
model |  all neg |  two views |  FH |  GT
DetCon |  ✓ |  ✓ |  33.6 |  37.0
(a) |  |  ✓ |  32.2 |  38.5
(b) |  |  |  27.7 |  38.8
Table 5: Simplifying the contrastive framework. We train DetConS models on
COCO using approximate FH masks or higher-quality ground-truth (GT) masks, and
evaluate them in the frozen-feature setting. “all neg”: Negative samples are
collected from the entire batch as opposed to only within a worker (out of 128
workers). “two views”: Contrastive predictions are made across augmentations,
as opposed to within a view.

##  5 Discussion

We have proposed DetCon, a simple but powerful self-supervised learning
algorithm. By exploiting low-level cues for organizing images into entities
such as objects and background regions, DetCon accelerates pretraining on
large datasets while improving accuracy on a variety of downstream tasks. Our
best models achieve state-of-the-art performance among self-supervised methods
pretrained on ImageNet and match recent state-of-the-art methods training
larger models on a much larger dataset [20].

We showed that the power of DetCon strongly correlates with how well the masks
used align with object boundaries. This seems intuitive—the DetCon objective
can only leverage independent learning signals from each image region if they
contain distinct content. Similarly, the resulting negative samples are
genuinely diverse only if they represent different objects. This opens
exciting prospects of research in jointly discovering objects and learning to
represent them. Given the improved performance of DetCon representations for
instance segmentation, a natural question is whether they could be used to
perform better unsupervised segmentations than the ones used during
pretraining. If so, these might be used to learn better representations still,
leading to a virtuous crescendo of unsupervised scene understanding.

## Acknowledgements

The authors are grateful to Carl Doersch, Raia Hadsell, and Evan Shelhamer for
insightful discussions and feedback on the manuscript.

##  Appendix A Appendix

###  A.1 Implementation: data augmentation

Self-supervised pretraining. Each image is randomly augmented twice, resulting
in two images: ${\bm{x}},{\bm{x}}^{\prime}$. The augmentations are constructed
as compositions of the following operations, each applied with a given
probability:

  1. 1.

random cropping: a random patch of the image is selected, whose area is
uniformly sampled in $[0.08\cdot\mathcal{A},\mathcal{A}]$, where $\mathcal{A}$
is the area of the original image, and whose aspect ratio is logarithmically
sampled in $[3/4,4/3]$. The patch is then resized to 224 $\times$224 pixels
using bicubic interpolation;

  2. 2.

horizontal flipping;

  3. 3.

color jittering: the brightness, contrast, saturation and hue are shifted by a
uniformly distributed offset;

  4. 4.

color dropping: the RGB image is replaced by its grey-scale values;

  5. 5.

gaussian blurring with a 23$\times$23 square kernel and a standard deviation
uniformly sampled from $[0.1,2.0]$;

  6. 6.

solarization: a point-wise color transformation $x\mapsto
x\cdot\mathds{1}_{x<0.5}+(1-x)\cdot\mathds{1}_{x\geq 0.5}$ with pixels $x$ in
$[0,1]$.

The augmented images ${\bm{x}},{\bm{x}}^{\prime}$ result from augmentations
sampled from distributions $\mathcal{T}$ and $\mathcal{T}^{\prime}$
respectively. These distributions apply the primitives described above with
different probabilities, and different magnitudes. The following table
specifies these parameters for the SimCLR [9] and BYOL frameworks [21], which
we adopt for DetConS and DetConB without modification.

|  DetConS |  DetConB
---|---|---
Parameter | $\mathcal{T}$ | $\mathcal{T}^{\prime}$ | $\mathcal{T}$ | $\mathcal{T}^{\prime}$
Random crop probability | 1.0
Flip probability | 0.5
Color jittering probability | 0.8
Color dropping probability | 0.2
Brightness adjustment max | 0.8 | 0.4
Contrast adjustment max | 0.8 | 0.4
Saturation adjustment max | 0.8 | 0.2
Hue adjustment max | 0.2 | 0.1
Gaussian blurring probability | $1.0$ | $0.0$ | $1.0$ | $0.1$
Solarization probability | $0.0$ | $0.0$ | $0.0$ | $0.2$

Transfer to COCO. When fine-tuning, image are randomly flipped and resized to
a resolution of $u\cdot 1024$ pixels on the longest side, where $u$ is
uniformly sampled in $[0.8,1.25]$, then cropped or padded to a
1024$\times$1024 image. The aspect ratio is kept the same as the original
image. During testing, images are resized to 1024 pixels on the longest side
then padded to 1024$\times$1024 pixels.

Transfer to PASCAL. During training, images are randomly flipped and scaled by
a factor in $[0.5,2.0]$. Training and testing are performed with
513$\times$513-resolution images.

Transfer to Cityscapes. During training, images are randomly horizontally
flipped and scaled by a factor in $[0.5,2.0]$, with minimum step size 0.25
within that range. Training is performed on 769$\times$769-resolution images
and testing is performed on 1025$\times$2049-resolution images.

Transfer to NYU-Depth v2. The original 640$\times$480 frames are down-sampled
by a factor of 2 and center-cropped to 304$\times$228 pixels. For training,
images are randomly flipped horizontally and color jittered with the same
grayscale, brightness, saturation, and hue settings as [21].

###  A.2 Implementation: architecture

Our default feature extractor is a ResNet-50 [27]. In Section 4.1 we also
investigate deeper architectures (ResNet-101, -152, and -200), and a wider
model (ResNet-200 $\times$2) obtained by scaling all channel dimensions by a
factor of 2.

As detailed in Section 3.1, this encoder yields a grid of hidden vectors which
we pool within masks to obtain a set of vectors ${\bm{h}}_{{\bm{m}}}$
representing each mask. These are then transformed by a projection head $g$
(and optionally a prediction head $q$) before entering the contrastive loss.

DetConS. Following SimCLR, the projection head is a two-layer MLP whose hidden
and output dimensions are 2048 and 128. The network uses the learned
parameters $\theta$ for both views.

DetConB. Following BYOL, the projection head is a two-layer MLP whose hidden
and output dimensions are 4096 and 256. The network uses the learned
parameters $\theta$ for processing one view, and an exponential moving average
of these parameters $\xi$ for processing the second. Specifically, $\xi$ is
updated using $\xi\leftarrow\lambda\cdot\xi+(1-\lambda)\cdot\theta$, where the
decay rate $\lambda$ is annealed over the course of training from
$\lambda_{0}$ to 1 using a cosine schedule [21]. $\lambda_{0}$ is set to 0.996
when training for 1000 epochs and 0.99 when training for 300 epochs. The
projection of the first view is further transformed with a prediction head,
whose architecture is identical to that of the projection head.

Computational cost. The forward pass through a ResNet-50 encoder requires
roughly 4B FLOPS. Ignoring the cost of bias terms and point-wise
nonlinearities, the projection head in DetConS requires 4.4M FLOPS (i.e.
2048$\times$2048 $+$ 2048$\times$128). Since this is calculated 16 times
rather than once, it results in an overhead of 67M FLOPS compared to SimCLR.
For DetConB the combined cost of evaluating the projection and prediction
heads results in an additional 173M FLOPS compared to BYOL. Finally, the cost
of evaluating the contrastive loss is 134M FLOPS for DetConS (i.e.
128$\times$4096$\times$162) and 268M FLOPS for DetConB. In total DetConS
requires 201M additional FLOPS and DetConB 441M which represent 5.3% and 11.6%
of the cost of evaluating the backbone. This overhead is sufficiently small
compared to the gain in training iterations required to reach a given transfer
performance (e.g. a 500% gain for DetConS over SimCLR, and a 333% for DetConB
over DetCon) for us not further distinguish between gains in computation and
training time.

###  A.3 Implementation: optimization

Self-supervised pretraining. We train using the LARS optimizer [64] with a
batch size of 4096 split across 128 Cloud TPU v3 workers. When training on
ImageNet we again adopt the optimization details of SimCLR and BYOL for
DetConS and DetConB, scaling the learning rate linearly with the batch size
and decaying it according to a cosine schedule. For DetConS the base learning
rate is 0.3 and the weight decay is $10^{-6}$. DetConB also uses these values
when training for 300 epochs; when training for 1000 epochs they are 0.2 and
$1.5\cdot 10^{-6}$.

When pretraining on COCO, we replace the cosine learning rate schedule with a
piecewise constant, which has been found to alleviate overfitting [25],
dropping the learning rate by a factor of 10 at the 96 and 98 percentiles. For
fair comparison we use the same schedules when applying SimCLR to the COCO
dataset, which we also find to perform better than the more aggressive cosine
schedule.

Transfer to COCO. We fine-tune with stochastic gradient descent, increasing
the learning rate linearly for the first 500 iterations and dropping twice by
a factor of 10, after $\frac{2}{3}$ and $\frac{8}{9}$ of the total training
time, following [61]. We use a base learning rate of 0.3 for ResNet-50 models
and 0.2 for larger ones, a momentum of 0.9, a weight decay of 4$\cdot$10-5,
and a batch size of 64 images split across 16 workers.

Transfer to PASCAL. We fine-tune for 45 epochs with stochastic gradient
descent, with a batch size of 16 and weight decay of $10^{-4}$. The learning
rate is 0.02 and dropped by a factor of 10 at the 70 and 90 percentiles.

Transfer to Cityscapes. We fine-tune for 160 epochs with stochastic gradient
descent and a Nesterov momentum of 0.9, using a batch size of 2 and weight
decay of $10^{-4}$. The initial learning rate is 0.005 and dropped by a factor
of 10 at the 70 and 90 percentiles.

Transfer to NYU-Depth v2. We fine-tune for 7500 steps with a batch size of
256, weight decay of 5$\cdot 10^{-4}$, and a learning rate of 0.16 scaled
linearly with the batch size [21].

###  A.4 Results: larger models

In Table 2 we compare to prior works on self-supervised learning which
transfer to COCO. Here we provide additional comparisons which use larger
models (ResNet-101, -152, and -200). We find DetCon to continue to outperform
prior work in this higher capacity regime (Table A.1).

|  Fine-tune 1$\times$ |  Fine-tune 2$\times$
---|---|---
method |  AP${}^{\text{bb}}_{\text{~{}}}$ |  AP${}^{\text{mk}}_{\text{~{}}}$ |  AP${}^{\text{bb}}_{\text{~{}}}$ |  AP${}^{\text{mk}}_{\text{~{}}}$
Supervised |  42.0 |  37.3 |  43.4 |  38.4
SimCLR [9] |  42.0 |  37.9 |  43.8 |  39.3
InfoMin [54] |  42.9 |  38.6 |  44.5 |  39.9
BYOL [21] |  43.7 |  38.8 |  44.3 |  39.4
DetConB |  45.2 |  40.0 |  45.7 |  40.4
(a) ResNet-101 feature extractor

|  Fine-tune 1$\times$ |  Fine-tune 2$\times$
---|---|---
method |  AP${}^{\text{bb}}_{\text{~{}}}$ |  AP${}^{\text{mk}}_{\text{~{}}}$ |  AP${}^{\text{bb}}_{\text{~{}}}$ |  AP${}^{\text{mk}}_{\text{~{}}}$
Supervised |  43.4 |  38.5 |  43.4 |  38.5
SimCLR [9] |  43.6 |  39.1 |  44.9 |  40.0
BYOL [21] |  44.9 |  40.0 |  45.7 |  40.6
DetConB |  46.0 |  40.6 |  46.4 |  40.7
(b) ResNet-152 feature extractor

|  Fine-tune 1$\times$ |  Fine-tune 2$\times$
---|---|---
method |  AP${}^{\text{bb}}_{\text{~{}}}$ |  AP${}^{\text{mk}}_{\text{~{}}}$ |  AP${}^{\text{bb}}_{\text{~{}}}$ |  AP${}^{\text{mk}}_{\text{~{}}}$
Supervised |  43.2 |  38.3 |  43.5 |  38.5
SimCLR [9] |  44.3 |  39.6 |  45.3 |  40.3
BYOL [21] |  45.6 |  40.5 |  45.9 |  40.5
DetConB |  47.1 |  41.3 |  47.2 |  41.5
(c) ResNet-200 feature extractor

Table A.1: Comparison to prior art: all methods are pretrained on ImageNet
then fined-tuned on COCO for 12 epochs (1$\times$ schedule) or 24 epochs
(2$\times$ schedule).

## References

  * [1] Pulkit Agrawal, Joao Carreira, and Jitendra Malik.  Learning to see by moving.  In ICCV, 2015.
  * [2] Pablo Arbeláez, Jordi Pont-Tuset, Jonathan T Barron, Ferran Marques, and Jitendra Malik.  Multiscale combinatorial grouping.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 328–335, 2014.
  * [3] Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi.  Self-labelling via simultaneous clustering and representation learning.  arXiv preprint arXiv:1911.05371, 2019.
  * [4] Philip Bachman, R Devon Hjelm, and William Buchwalter.  Learning representations by maximizing mutual information across views.  arXiv preprint arXiv:1906.00910, 2019.
  * [5] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  In The European Conference on Computer Vision (ECCV), September 2018\.
  * [6] Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin.  Leveraging large-scale uncurated data for unsupervised pre-training of visual features.  In Leveraging Large-Scale Uncurated Data for Unsupervised Pre-training of Visual Features, 2019.
  * [7] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  arXiv preprint arXiv:2006.09882, 2020.
  * [8] Joao Carreira and Cristian Sminchisescu.  Cpmc: Automatic object segmentation using constrained parametric min-cuts.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(7):1312–1328, 2011.
  * [9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  arXiv preprint arXiv:2002.05709, 2020.
  * [10] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton.  Big self-supervised models are strong semi-supervised learners.  arXiv preprint arXiv:2006.10029, 2020.
  * [11] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [12] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  arXiv preprint arXiv:2011.10566, 2020.
  * [13] Carl Doersch, Abhinav Gupta, and Alexei A Efros.  Unsupervised visual representation learning by context prediction.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1422–1430, 2015.
  * [14] Carl Doersch and Andrew Zisserman.  Multi-task self-supervised visual learning.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2051–2060, 2017.
  * [15] Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell.  Adversarial feature learning.  arXiv preprint arXiv:1605.09782, 2016.
  * [16] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox.  Discriminative unsupervised feature learning with convolutional neural networks.  In NIPS, 2014.
  * [17] Pedro F Felzenszwalb and Daniel P Huttenlocher.  Efficient graph-based image segmentation.  International journal of computer vision, 59(2):167–181, 2004.
  * [18] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  arXiv preprint arXiv:1803.07728, 2018.
  * [19] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014.
  * [20] Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al.  Self-supervised pretraining of visual features in the wild.  arXiv preprint arXiv:2103.01988, 2021.
  * [21] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  Advances in Neural Information Processing Systems, 33, 2020.
  * [22] Jose A Guerrero-Colón, Eero P Simoncelli, and Javier Portilla.  Image denoising using mixtures of gaussian scale mixtures.  In 2008 15th IEEE International Conference on Image Processing, pages 565–568. IEEE, 2008.
  * [23] Raia Hadsell, Sumit Chopra, and Yann LeCun.  Dimensionality reduction by learning an invariant mapping.  In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pages 1735–1742. IEEE, 2006.
  * [24] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  arXiv preprint arXiv:1911.05722, 2019.
  * [25] Kaiming He, Ross Girshick, and Piotr Dollár.  Rethinking imagenet pre-training.  In Proceedings of the IEEE international conference on computer vision, pages 4918–4927, 2019.
  * [26] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [28] Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord.  Data-efficient image recognition with contrastive predictive coding.  arXiv preprint arXiv:1905.09272, 2019.
  * [29] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Adam Trischler, and Yoshua Bengio.  Learning deep representations by mutual information estimation and maximization.  arXiv preprint arXiv:1808.06670, 2018.
  * [30] Allan Jabri, Andrew Owens, and Alexei A Efros.  Space-time correspondence as a contrastive random walk.  arXiv preprint arXiv:2006.14613, 2020.
  * [31] Xu Ji, João F Henriques, and Andrea Vedaldi.  Invariant information clustering for unsupervised image classification and segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9865–9874, 2019.
  * [32] Kurt Koffka.  Principles of Gestalt psychology, volume 44.  Routledge, 2013.
  * [33] Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer.  Revisiting self-supervised visual representation learning.  CoRR, abs/1901.09005, 2019.
  * [34] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in neural information processing systems, pages 1097–1105, 2012.
  * [35] Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab.  Deeper depth prediction with fully convolutional residual networks.  In 2016 Fourth international conference on 3D vision (3DV), pages 239–248. IEEE, 2016.
  * [36] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich.  Colorization as a proxy task for visual understanding.  In CVPR, pages 6874–6883, 2017.
  * [37] Yin Li, Manohar Paluri, James M Rehg, and Piotr Dollár.  Unsupervised learning of edges.  In CVPR, 2016.
  * [38] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [39] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [40] David Lowe.  Perceptual organization and visual recognition, volume 5.  Springer Science & Business Media, 2012.
  * [41] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.  In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pages 416–423. IEEE, 2001.
  * [42] Ishan Misra, C Lawrence Zitnick, and Martial Hebert.  Shuffle and learn: unsupervised learning using temporal order verification.  In ECCV, 2016.
  * [43] T Nathan Mundhenk, Daniel Ho, and Barry Y Chen.  Improvements to context based self-supervised learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9339–9348, 2018.
  * [44] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In European Conference on Computer Vision, pages 69–84. Springer, 2016.
  * [45] Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan.  Learning features by watching objects move.  arXiv preprint arXiv:1612.06370, 2016.
  * [46] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
  * [47] Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, and Jian Sun.  Megdet: A large mini-batch object detector.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6181–6189, 2018.
  * [48] Pedro O Pinheiro, Amjad Almahairi, Ryan Y Benmaleck, Florian Golemo, and Aaron Courville.  Unsupervised learning of dense visual representations.  arXiv preprint arXiv:2011.05499, 2020.
  * [49] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár.  Designing network design spaces.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10428–10436, 2020.
  * [50] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [51] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [52] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta.  Revisiting unreasonable effectiveness of data in deep learning era.  In Proceedings of the IEEE international conference on computer vision, pages 843–852, 2017.
  * [53] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  arXiv preprint arXiv:1906.05849, 2019.
  * [54] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola.  What makes for good views for contrastive learning.  arXiv preprint arXiv:2005.10243, 2020.
  * [55] Shimon Ullman et al.  High-level vision: Object recognition and visual cognition, volume 2.  MIT press Cambridge, MA, 1996.
  * [56] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [57] Stéfan van der Walt, Johannes L. Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D. Warner, Neil Yager, Emmanuelle Gouillart, Tony Yu, and the scikit-image contributors.  scikit-image: image processing in Python.  PeerJ, 2014.
  * [58] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool.  Unsupervised semantic segmentation by contrasting object mask proposals.  arXiv preprint arXiv:2102.06191, 2021.
  * [59] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol.  Extracting and composing robust features with denoising autoencoders.  In Proceedings of the 25th international conference on Machine learning, pages 1096–1103, 2008.
  * [60] Xiaolong Wang and Abhinav Gupta.  Unsupervised learning of visual representations using videos.  In ICCV, 2015.
  * [61] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [62] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733–3742, 2018.
  * [63] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu.  Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning.  arXiv preprint arXiv:2011.10043, 2020.
  * [64] Yang You, Igor Gitman, and Boris Ginsburg.  Large batch training of convolutional networks.  arXiv preprint arXiv:1708.03888, 2017.
  * [65] Richard Zhang, Phillip Isola, and Alexei A Efros.  Colorful image colorization.  In European conference on computer vision, pages 649–666. Springer, 2016.
  * [66] Richard Zhang, Phillip Isola, and Alexei A Efros.  Split-brain autoencoders: Unsupervised learning by cross-channel prediction.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1058–1067, 2017.
  * [67] Xiao Zhang and Michael Maire.  Self-supervised visual representation learning from hierarchical grouping.  arXiv preprint arXiv:2012.03044, 2020.


# Perspectives on predicting and controlling turbulent flows through deep
learning

Ricardo Vinuesa1,2∗
1 FLOW, Engineering Mechanics, KTH Royal Institute of Technology, Stockholm,
Sweden
2 Swedish e-Science Research Centre (SeRC), Stockholm, Sweden

###### Abstract

The current revolution in the field of machine learning (ML) is leading to
many interesting developments in a wide range of areas, including fluid
mechanics. Here we review recent and emerging possibilities in the context of
predictions, simulations and control of fluid flows, focusing on wall-bounded
turbulence. A number of important areas are benefiting from ML, and it is
important to identify the synergies with the existing pillars of scientific
discovery, i.e. theory, experiments and simulations. It is essential to adopt
a balanced approach as a community in order to harness all the positive
potential of these novel methods.

Impact statement. Fluid mechanics, and more concretely turbulence, is an
ubiquitous problem in science and engineering. Being able to understand and
predict the evolution of turbulent flows can have a critical impact on our
possibilities to tackle a wide range of sustainability problems (including the
current climate emergency) and industrial applications. In this piece the
potential of novel deep-learning methods to study and predict turbulence is
discussed. Another important area where deep learning can help is control,
i.e. the active manipulation of the fluid flow to improve the efficiency of
processes (e.g. reduced drag, increased mixing, etc.).

_Keywords:_ machine learning (ML); deep learning (DL); artificial intelligence
(AI); computational fluid dynamics (CFD); experiments; turbulent flows; flow
control

††∗ Corresponding author: Ricardo Vinuesa<EMAIL_ADDRESS>
##  1 Predictions in turbulence

Recent years have witnessed a renewed interest in machine-learning (ML)
methods applied to the study of fluid mechanics [7]. The availability of
massive amounts of data, together with the possibility of employing more
powerful graphics-processing-unit (GPU)-based machines and widely available ML
libraries [1], are enabling significant progress at a fast pace. Following
some applications to chaotic dynamical systems [8], recent studies start to
focus on low-dimensional representations of turbulence [61] and even wall-
bounded turbulent flows [6]. In these studies the focus is on predicting the
temporal dynamics of the flow, a task that can be achieved with quite some
success with data-driven methods involving long-short-term-memory (LSTM)
networks [27] (which are deep-learning architectures capable of exploiting the
temporal patterns in the data to perform time-series predictions) and also
Koopman-based frameworks with nonlinear forcing [15]. Other promising
approaches for such temporal predictions include reservoir computing [11]
(which has been shown to effectively capture extreme events in the time
series) and transformers [75] (which have the potential to perform accurate
instantaneous predictions over longer time horizons than other data-driven
methods). Here it is important to note that, although after a certain time
horizon the various data-driven approaches start to deviate with respect to
the original time series, the temporal dynamics of the system is well
represented as illustrated via e.g. Poincaré maps and Lyapunov exponents [61].

Besides temporal predictions, spatial predictions in turbulence can
significantly benefit from machine learning. The early work by Milano &
Koumoutsakos [41] showed the feasibility of using deep neural networks to make
meaningful predictions in the near-wall region of turbulent channels.
Interestingly, these authors showed that by restricting to linear activation
functions one can recover the well-known POD (proper-orthogonal decomposition)
modes [38]. Another relevant application is non-intrusive sensing, i.e. the
predictions of the flow above the wall based on measurements at the wall. In
this spirit, Guastoni et al. [25] showed that it is possible to use
convolutional neural networks (CNNs) [34] to effectively exploit the spatial
correlations in the turbulence data to perform predictions of the velocity
fluctuations based on the two wall-shear-stress fields and the wall pressure.
More advanced computer-vision architectures, namely generative adversarial
networks (GANs) [23], have also been used to obtain robust off-wall
predictions based on sparse measurements. This paves the way to obtain
methodologies which can be deployed in the context of experiments. Other
super-resolution approaches have been proposed by Fukami et al. [20] and by
Yousif et al. [74]. By exploiting the similarity between the wall-shear stress
and the heat flux at the wall, Kim & Lee [29] could leverage the potential of
CNNs to predict turbulent heat transfer. Another interesting application of
CNNs is the prediction of the flow close to the wall based on information
farther away from it [4], an approach that could potentially be used for wall
modeling.

##  2 Simulations of fluid flows

When it comes to performing computations of fluid flows, ML can help to
accelerate direct numerical simulations (DNS), it can improve turbulence
modeling and it can help to develop more robust reduced-order models (ROMs)
[66]. In the first category, the interesting work by Kochkov et al. [31]
suggests that it may be possible to accurately perform accurate simulations in
coarse meshes. Some more recent work indicates that such an approach to
accelerate DNS may also be applied to spectral methods [13]. Other approaches
to perform turbulence simulations at a reduced cost may be possible thanks to
physics-informed neural networks (PINNs) [51, 52], for instance by effectively
solving the Reynolds-averaged Navier–Stokes (RANS) equations [17]. In
particular, Eivazi et al. [17] showed that, given adequate boundary conditions
(including those of the Reynolds stresses), the incompressible RANS equations
can be efficiently solved, with very good results for the Reynolds stresses,
via PINNs. This data-driven approach has also been able to predict the flow
from sparse measurements, as described by Sitte & Doan [57]. An alternative
way to perform fluid-flow simulations at a reduced computational cost is to
reduce the size of the computational domain through data-driven inflow
conditions [47, 21, 75] and far-field distributions, which can be obtained
e.g. via Gaussian-process regression [43].

Turbulence modeling is also benefiting from the capabilities enabled by ML.
Large-eddy simulations (LES) can rely on accurate subgrid-scale models (SGS)
developed by convolutional neural networks (CNNs) together with the eddy-
viscosity assumption [5]. In this sense, a promising approach proposed by
Novati et al. [45] relies on using reinforcement learning to determine the
coefficient in the Smagorinski model [58]. This has led to some success in
simulations of turbulent channel. Similar ideas, based on reinforcement
learning, have been proposed to develop wall models [2]; this is an area with
interesting potential applications to high-Reynolds-number wall-bounded
turbulent flows. Furthermore, RANS simulations have also been improved via ML,
more concretely through the development of accurate and robust RANS models.
Some of the earlier deep-learning-enabled RANS studies include those by Ling
et al. [35] and Wu et al. [72], and as anticipated by several perspective
papers [32, 14], these data-driven techniques have expanded their application
to turbulence modeling. Another data-driven approach to RANS modeling was
proposed by Weatheritt & Sandberg [71, 70], who relied on genetic programming
to obtain expressions for the Reynolds stresses. The advantage of their work
is the fact that they produce interpretable [54, 67] models, i.e. it is
possible to establish an equation relating the inputs and the outputs. In this
spirit, Jiang et al. [28] have proposed a deep-learning-based RANS model which
is also interpretable, a very important feature of turbulence simulations.
Their method, which is shown schematically in Figure 1, is based on two
different neural networks to model the structural and parametric aspects of
the turbulent flow.

Figure 1: Schematic representation of the interpretable machine-learning
framework proposed by Jiang et al. [28] for RANS modeling, which includes the
three following phases: (i) design of the framework based on the domain
knowledge, (ii) training strategy and (iii) performance assessment. Reprinted
from Ref. [28], with permission of the publisher (AIP Publishing).

The last aspect of fluid-flow simulation is the development of ROMs, which can
be very helpful in the context of flow control, estimation and optimization.
Besides the classical approaches to obtain ROMs, namely proper-orthogonal
decomposition (POD) [38, 63] and dynamic-mode decomposition (DMD) [56] (which
both rely on linear algebra), deep learning has been recently used for ROM
development by leveraging the non-linearity introduced by the activation
functions. Murata et al. [44] proposed the usage of CNNs, which are capable of
exploiting the spatial features in the data, to produce non-linear modal
decompositions of fluid flows. Moreover, Fukami et al. [22] developed an
interesting approach for non-linear ROMs based on hierarchical autoencoders.
The autoencoder (AE) is a deep-learning model which, through successive
application of convolutional filters, can provide a compressed representation
of the original data in the latent space. AEs can produce significantly
compressed representations of the original data thanks to the employed
nonlinearities, but they do not yield modes sorted by energy contribution, and
these modes are not orthogonal. These are important properties to provide
interpretable and parsimonious ROMs. The hierarchical autoencoder (HAE) [22]
relies on the following process:

  1. 1.

An autoencoder with dimension $d=1$ in the latent space is trained, producing
one latent vector $\boldsymbol{r}_{1}$.

  2. 2.

Then, another autoencoder with $d=2$ is trained such that $\boldsymbol{r}_{1}$
is fixed, and a new latent vector $\boldsymbol{r}_{2}$ is obtained. The
contribution of the new vector towards the reconstruction of the original data
is smaller than that of the initial one.

  3. 3.

Subsequently, a new autoencoder with $d=3$ is trained fixing
$\boldsymbol{r}_{1}$ and $\boldsymbol{r}_{2}$, producing a new latent vector
$\boldsymbol{r}_{3}$.

  4. 4.

This process is repeated until the desired size of the latent space is
completed.

This method produces a series of non-linear AE modes with progressively
smaller contribution towards the reconstruction, and Fukami et al. [22]
illustrated its usage in the flow around a two-dimensional cylinder at a
Reynolds number $Re=100$ based on cylinder diameter and incoming velocity. The
HAE technique was assessed in the turbulent flow between two wall-mounded
obstacles [33] by Eivazi et al. [16]. While the method works well even in
complex turbulent flows, and the modes progressively contribute less towards
the reconstruction, their lack of orthogonality affects the interpretability
of the ROM. An alternative approach was proposed by Eivazi et al. [16], and it
focused on developing a non-linear orthogonal modal decomposition of the flow
while also being able to rank the modes by their contribution towards the
reconstruction of the turbulent data. Their method relies on
$\beta$-variational autoencoders ($\beta$VAEs), i.e. a modification of the VAE
[30], an architecture that is receiving attention in the fluid-mechanics
community [40]. In this approach, stochasticity is introduced in the latent
space, and a penalization is added to the loss function with the goal of
promoting learning the minimum number of nonzero latent variables which are
statistically independent. In this way, it is possible to learn a parsimonious
and disentangled latent representation of the original data. The $\beta$VAE
approach enables recovering almost $90\%$ of the energy from the original data
with only 5 modes, whereas the same number of POD modes leads to a recovery of
around $30\%$. Furthermore, the $\beta$VAE modes exhibit $99.2\%$
orthogonality, as measured by the cross-correlation matrix, a fact that shows
the potential of this method for compact and interpretable ROM development.
This point is illustrated in Figure 2, which shows the first 5 modes from the
$\beta$VAE, POD, HAE and a standard AE based on CNNs. The first interesting
observation comes from the HAE and standard AE approaches, which exhibit
levels of orthogonality below $90\%$, where no distinct patterns or physical
phenomena can be really noticed. The first POD modes show the presence of
large-scale shedding, which is of course expected in this case, and
interestingly this is also identified by the $\beta$VAE modes. In fact, the
$\beta$VAE modes also exhibit higher-frequency content in the modes associated
with turbulent fluctuations consistent with such shedding, a fact that shows
the potential of this approach to significantly compress the data while
retaining physical interpretability. Autoencoder-based methods have great
potential to develop compact ROMs for turbulent flows, being able to exploit
the nonlinear modal reconstruction, as well as potentially interpreting the
latent space [36, 12, 3]. Recent work has leveraged transformers [64] to
predict the temporal dynamics of the latent space [59, 55], effectively
producing a ROM to advance the solution in time.

Figure 2: The ranked spatial modes obtained from the various approaches for
$d=5$. Adapted from Ref. [16], with permission of the publisher (Elsevier).

##  3 Control of turbulence

Data-driven methods for flow control have been extensively used in the
literature, for instance exploiting the linear relations in the flow to
control transition to turbulence [18] or forcing at the natural frequency of
the shear layer [73] to control separation, a method that can be complented
with harmonic resolvent analysis [46]. There are however other approaches to
flow control based on ML, which may exhibit interesting potential in a number
of applications. For instance, Bayesian regression based on Gaussian processes
[53] has been used in the context of turbulent-bounday-layer control by
Mahfoze et al. [39]. Another data-driven control approach which has been
proved successful in controlling external flows is genetic programming [48,
42], which in principle enables exploring various terms in the control law
through evolutionary algorithms, and has led to very promising results. One
interesting aspect of these approaches is that they allow to assess a larger
space of control laws by leveraging the access to large-scale databases,
potentially leading to more sophisticated control strategies than the
classical ones.

Another very promising approach to discover novel control strategies is deep
reinforcement learning (DRL). In this framework, an agent (the neural network)
interacts with an environment (the flow simulator) through actions (the
control), thus changing its state. The goal of DRL is to decide the set of
actions to take, given the state of the system, in order to maximize a certain
reward [19]. This goal is achieved through iterative interaction with the
system by gathering experience on the effect of the actions, a fact that has
the potential of producing novel and unexpected control mechanisms in a wide
variety of flows. In this sense, one of the first studies to apply DRL for
flow control was that of Rabault et al. [49], who obtained significant drag
reduction in the flow around a two-dimensional (2D) cylinder through active
control via jets. In this context, Guastoni et al. [24] have shown that it is
possible to use DRL to reduce the length of a 2D separation bubble. One of
their results is the documentation of an improved result from DRL compared
with classical periodic control, due to the richer range of frequencies
discovered by DRL. Other interesting applications include DRL for control in
Couette flow, where the policy learned in a ROM is extended to the full domain
[37], the work on turbulent channels by Sonoda et al. [60] and the work on
three-dimensional cylinders by Suárez et al. [62]. Guastoni et al. [26] have
recently documented a complete framework based on multi-agent reinforcement
learning (MARL), which is illustrated in Figure 3, to perform active flow
control in turbulent channels. They document a higher drag reduction through
MARL ($30\%$) than that obtained through the classical opposition control
($20\%$) [9]. An extensive account of the potential for flow control from DRL,
including turbulent wings, can be found in Refs. [68, 65].

Figure 3: Schematic representation of the multi-agent reinforcement learning
(MARL) framework proposed by Guastoni et al. [26] to control turbulence in
channels. The state is defined in terms of the streamwise and wall-normal
fluctuations at the sensing plane, the reward is the reduction of wall-shear
stress and the actions are blowing/suction at the wall. Reprinted from Ref.
[26], with permission of the publisher (Springer).

##  4 Outlook

The current revolution in ML, particularly in deep learning, is leading to a
number of interesting developments in the context of predicting and
controlling fluid flows. It is important to highlight that these methods are
not meant to replace any of the current pillars of scientific discovery, i.e.
theory, experiments or simulations; rather, these techniques are enabling to
solve concrete problems thanks to an improved predictive power from the
available data, and therefore ML can be considered as a fourth complementing
pillar instead of a competing approach. In this sense, while very interesting
progress is being observed in chaotic systems and simplified fluid flows, the
applications to fully-turbulent flows are still scarce. It is in the
application of ML methods to large-scale turbulent cases where the most
relevant and impactful applications will be found. However, in order to obtain
ML models that can effectively be trained on the large datasets from wall-
bounded turbulence (on the order of many terabytes of data), it will be
necessary to scale up the current algorithms. This will require to address
important computer-science questions having to do with the scaling of the
algorithms, and their possibility to run concurrently, in situ, with very
large simulations. The current trend in high-performance-computing (HPC)
facilities, combining both central-processing and graphics-processing units
(CPUs and GPUs respectively), further increases the complexity of applying
these methods in the future large-scale flow simulations. Despite the
challenges, the possibilities in terms of prediction and control are numerous,
including in experimental cases [69]. Furthermore, already-trained deep-
learning models may help to learn novel physical phenomena through
interpretability methods [10, 67].

The development of progressively larger databases and models, and the high
sensitivity of the results to small hyper parameter choices in deep learning
(and more crucially in DRL), make it essential to develop, as a community,
better practices in terms of benchmark cases, open-source code development and
data sharing. This will enable reproducibility of the results and a quicker
(and more robust) advancement of the field of data-driven scientific discovery
in fluid mechanics. An important implication of this is the possibility to
develop deep-learning models that can generalize to new cases; transfer
learning (for example from low to high Reynolds numbers [25]) is critical in
this sense, as well as the framework of continual learning [50], which may
enable training a deep-learning model which performs equally well in a wide
range of tasks. Moreover, novel experimental design aided by ML is another
very promising area of research, including data assimilation.

To conclude, there is great potential in ML for fluid mechanics in many areas,
and this paradigm can complement the other existing pillars of scientific
discovery. In order to properly develop the field and harness all its
potential, it is important to maintain the scientific rigor as a community,
keeping an informed optimism about the new developments, and avoiding
exaggerated claims on the achievements of these techniques.

## Competing Interest

The author declares no competing interests.

## Funding

RV acknowledges the financial support from the Lundeqvist foundation and the
ERC Grant No. “2021-CoG-101043998, DEEPCONTROL”. Views and opinions expressed
are however those of the author only and do not necessarily reflect those of
the European Union or the European Research Council. Neither the European
Union nor the granting authority can be held responsible for them.

## Data Availability Statement

Data Availability not applicable to this position paper.

## Acknowledgements

An initial version of this paper is available via the ERCOFTAC Bulletin 134,
March 2023, after invitation by Dr. Anh Khoa Doan.

## References

  * Abadi et al [2016] M. Abadi et al.  Tensorflow: A system for large-scale machine learning.  _Proceedings of the 12thUSENIX Symposium on Operating Systems Design and Implementation (OSDI 16) (USENIX Association, 2016)_ , 16:265–283, 2016.
  * Bae and Koumoutsakos [2022] H. J. Bae and P. Koumoutsakos.  Scientific multi-agent reinforcement learning for wall-models of turbulent flows.  _Nature Communications_ , 13:1443, 2022.
  * Bakarji et al. [2023] J. Bakarji, K. Champion, J. N. Kutz, and S. L. Brunton.  Discovering governing equations from partial measurements with deep delay autoencoders.  _Proceedings of the Royal Society A_ , 479:20230422, 2023\.
  * Balasubramanian et al. [2023] A. G. Balasubramanian, L. Guastoni, A. Güemes, A. Ianiro, S. Discetti, P. Schlatter, H. Azizpour, and R. Vinuesa.  Predicting the near-wall region of turbulence through convolutional neural networks.  _International Journal of Heat and Fluid Flow_ , 103:109200, 2023.
  * Beck et al. [2019] A. D. Beck, D. G. Flad, and C.-D. Munz.  Deep neural networks for data-driven LES closure models.  _Journal of Computational Physics_ , 398:108910, 2019.
  * Borrelli et al. [2022] G. Borrelli, L. Guastoni, H. Eivazi, P. Schlatter, and R. Vinuesa.  Predicting the temporal dynamics of turbulent channels through deep learning.  _International Journal of Heat and Fluid Flow_ , 96:109010, 2022.
  * Brunton et al. [2020] S. L. Brunton, B. R. Noack, and P. Koumoutsakos.  Machine learning for fluid mechanics.  _Annual Review of Fluid Mechanics_ , 52:477–508, 2020.
  * Champion et al. [2019] K. Champion, B. Lusch, J. N. Kutz, and S. L. Brunton.  Data-driven discovery of coordinates and governing equations.  _Proceedings of the National Academy of Sciences_ , 116(45):22445–22451, 2019.
  * Choi et al. [1994] H. Choi, P. Moin, and J. Kim.  Active turbulence control for drag reduction in wall-bounded flows.  _Journal of Fluid Mechanics_ , 262:75–110, 1994.
  * Cranmer et al. [2020] M. Cranmer, A. Sanchez-Gonzalez, P. Battaglia, R. Xu, K. Cranmer, D. Spergel, and S. Ho.  Discovering symbolic models from deep learning with inductive biases.  _34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada. Preprint arXiv:2006.11287_ , 2020.
  * Doan et al. [2021a] N. A. K. Doan, W. Polifke, and L. Magri.  Short-and long-term predictions of chaotic flows and extreme events: a physics-constrained reservoir computing approach.  _Proceedings of the Royal Society A_ , 477:20210135, 2021a.
  * Doan et al. [2021b] N. A. K. Doan, W. Polifke, and L. Magri.  Auto-encoded reservoir computing for turbulence learning.  _International Conference on Computational Science. Springer, Cham_ , pages 344–351, 2021b.
  * Dresdner et al. [2022] G. Dresdner, D. Kochkov, P. Norgaard, L. Zepeda-Núñez, L. Jamie, A. Smith, M. P. Brenner, and S. Hoyer.  Learning to correct spectral methods for simulating turbulent flows.  _Preprint arXiv:2207.00556_ , 2022.
  * Duraisamy et al. [2019] K. Duraisamy, G. Iaccarino, and H. Xiao.  Turbulence modeling in the age of data.  _Annual Review of Fluid Mechanics_ , 51:357–377, 2019.
  * Eivazi et al. [2021] H. Eivazi, L. Guastoni, P. Schlatter, H. Azizpour, and R. Vinuesa.  Recurrent neural networks and Koopman-based frameworks for temporal predictions in a low-order model of turbulence.  _International Journal of Heat and Fluid Flow_ , 90:108816, 2021.
  * Eivazi et al. [2022a] H. Eivazi, S. Le Clainche, S. Hoyas, and R. Vinuesa.  Towards extraction of orthogonal and parsimonious non-linear modes from turbulent flows.  _Expert Systems with Applications_ , 202:117038, 2022a.
  * Eivazi et al. [2022b] H. Eivazi, M. Tahani, P. Schlatter, and R. Vinuesa.  Physics-informed neural networks for solving Reynolds-averaged Navier–Stokes equations.  _Physics of Fluids_ , 34:075117, 2022b.
  * Fabbiane et al. [2014] N. Fabbiane, O. Semeraro, S. Bagheri, and D. S. Henningson.  Adaptive and model-based control theory applied to convectively unstable flows.  _Applied Mechanics Reviews_ , 66:060801, 2014.
  * François-Lavet et al. [2018] V. François-Lavet, P. Henderson, R. Islam, and M. Bellemare.  An introduction to deep reinforcement learning.  _Foundations and Trends in Machine Learning_ , 11:219–354, 2018.
  * Fukami et al. [2019a] K. Fukami, K. Fukagata, and K. Taira.  Super-resolution reconstruction of turbulent flows with machine learning.  _Journal of Fluid Mechanics_ , 870:106–120, 2019a.
  * Fukami et al. [2019b] K. Fukami, Y. Nabae, K. Kawai, and K. Fukagata.  Synthetic turbulent inflow generator using machine learning.  _Physical Review Fluids_ , 4:064603, 2019b.
  * Fukami et al. [2020] K. Fukami, T. Nakamura, and K. Fukagata.  Convolutional neural network based hierarchical autoencoder for nonlinear mode decomposition of fluid field data.  _Physics of Fluids_ , 32:095110, 2020.
  * Goodfellow et al. [2014] J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Advances in neural information processing systems.  _Journal of Fluid Mechanics_ , 27, 2014.
  * Guastoni et al. [2021a] L. Guastoni, A. Ghadirzadeh, J. Rabault, P. Schlatter, H. Azizpour, and R. Vinuesa.  Deep reinforcement learning for active drag reduction in wall turbulence.  _APS Division of Fluid Dynamics Meeting Abstracts_ , A19:007, 2021a.
  * Guastoni et al. [2021b] L. Guastoni, A. Güemes, A. Ianiro, S. Discetti, P. Schlatter, H. Azizpour, and R. Vinuesa.  Convolutional-network models to predict wall-bounded turbulence from wall quantities.  _Journal of Fluid Mechanics_ , 928:A27, 2021b.
  * Guastoni et al. [2023] L. Guastoni, J. Rabault, P. Schlatter, H. Azizpour, and R. Vinuesa.  Deep reinforcement learning for turbulent drag reduction in channel flows.  _European Physical Journal E_ , 46:27, 2023.
  * Hochreiter and Schmidhuber [1997] S. Hochreiter and J. Schmidhuber.  Deep learning.  _Neural Computation_ , 9:1735, 1997.
  * Jiang et al. [2021] C. Jiang, R. Vinuesa, R. Chen, J. Mi, S. Laima, and H. Li.  An interpretable framework of data-driven turbulence modeling using deep neural networks.  _Physics of Fluids_ , 33:055133, 2021.
  * Kim and Lee [2020] J. Kim and C. Lee.  Prediction of turbulent heat transfer using convolutional neural networks.  _Journal of Fluid Mechanics_ , 882:A18, 2020.
  * Kingma and Welling [2013] D. P. Kingma and M. Welling.  Auto-encoding variational Bayes.  _Preprint arXiv:1312.6114_ , 2013.
  * Kochkov et al. [2021] D. Kochkov, J. A. Smith, A. Alieva, Q. Wang, M. P. Brenner, and S. Hoyer.  Machine learning-accelerated computational fluid dynamics.  _Proceedings of the National Academy of Sciences_ , 118:e2101784118, 2021.
  * Kutz [2017] J. N. Kutz.  Deep learning in fluid dynamics.  _Journal of Fluid Mechanics_ , 814:1–4, 2017.
  * Lazpita et al. [2022] E. Lazpita, A. Martínez-Sánchez, A. Corrochano, S. Hoyas, S. Le Clainche, and R. Vinuesa.  On the generation and destruction mechanisms of arch vortices in urban fluid flows.  _Physics of Fluids_ , 34:051702, 2022.
  * LeCun et al. [2015] Y. LeCun, Y. Bengio, and G. Hinton.  Deep learning.  _Nature_ , 521:436–444, 2015.
  * Ling et al. [2016] J. Ling, A. Kurzawski, and J. Templeton.  Reynolds averaged turbulence modelling using deep neural networks with embedded invariance.  _Journal of Fluid Mechanics_ , 807:155–166, 2016.
  * Linot and Graham [2022] A. J. Linot and M. D. Graham.  Data-driven reduced-order modeling of spatiotemporal chaos with neural ordinary differential equations.  _Chaos_ , 32:073110, 2022.
  * Linot et al. [2023] A. J. Linot, K. Zeng, and M. D. Graham.  Turbulence control in plane Couette flow using low-dimensional neural ODE-based models and deep reinforcement learning.  _International Journal of Heat and Fluid Flow_ , 101:109139, 2023.
  * Lumley [1967] J. L. Lumley.  The structure of inhomogeneous turbulence.  _Atmospheric turbulence and wave propagation, A. M. Yaglom and V. I. Tatarski (eds). Nauka, Moscow_ , pages 166–178, 1967.
  * Mahfoze et al. [2019] O. A. Mahfoze, A. Moody, A. Wynn, R. D. Whalley, and S. Laizet.  Reducing the skin-friction drag of a turbulent boundary-layer flow with low-amplitude wall-normal blowing within a Bayesian optimization framework.  _Physical Review Fluids_ , 4:094601, 2019.
  * Maulik et al. [2020] R. Maulik, K. Fukami, N. Ramachandra, K. Fukagata, and K. Taira.  Probabilistic neural networks for fluid flow surrogate modeling and data recovery.  _Physical Review Fluids_ , 5:104401, 2020.
  * Milano and Koumoutsakos [2002] M. Milano and P. Koumoutsakos.  Neural network modeling for near wall turbulent flow.  _Journal of Computational Physics_ , 182:1–26, 2002.
  * Minelli et al. [2020] G. Minelli, T. Dong, B. Noack, and S. Krajnović.  Upstream actuation for bluff-body wake control driven by a genetically inspired optimization.  _Journal of Fluid Mechanics_ , 893:A1, 2020.
  * Morita et al. [2022] Y. Morita, S. Rezaeiravesh, N. Tabatabaei, R. Vinuesa, K. Fukagata, and P. Schlatter.  Applying Bayesian optimization with Gaussian-process regression to computational fluid dynamics problems.  _Journal of Computational Physics_ , 449:110788, 2022.
  * Murata et al. [2020] T. Murata, K. Fukami, and K. Fukagata.  Nonlinear mode decomposition with convolutional neural networks for fluid dynamics.  _Journal of Fluid Mechanics_ , 882:A13, 2020.
  * Novati et al. [2021] G. Novati, H. L. de Laroussilhe, and P. Koumoutsakos.  Automating turbulence modelling by multi-agent reinforcement learning.  _Nature Machine Intelligence_ , 3:87–96, 2021.
  * Padovan et al. [2020] A. Padovan, S. E. Otto, and C. W. Rowley.  Analysis of amplification mechanisms and cross-frequency interactions in nonlinear flows via the harmonic resolvent.  _Journal of Fluid Mechanics_ , 900:A14, 2020.
  * Poletto et al. [2013] R. Poletto, T. Craft, and A. Revell.  A new divergence free synthetic eddy method for the reproduction of inlet flow conditions for LES.  _Flow Turbulence and Combustion_ , 91:519–539, 2013.
  * R. et al. [2017] L. R., B. R. Noack, L. Cordier, J. Borée, and F. Harambat.  Drag reduction of a car model by linear genetic programming control.  _Experiments in Fluids_ , 58:103, 2017.
  * Rabault et al. [2019] J. Rabault, M. Kuchta, A. Jensen, U. Réglade, and N. Cerardi.  Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control.  _Journal of Fluid Mechanics_ , 7:62, 2019.
  * Raghavan and Balaprakash [2021] K. Raghavan and P. Balaprakash.  Formalizing the generalization-forgetting trade-off in continual learning.  _35th Conference on Neural Information Processing Systems (NeurIPS 2021)_ , 2021.
  * Raissi et al. [2019] M. Raissi, P. Perdikaris, and G. E. Karniadakis.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  _Journal of Computational Physics_ , 378:686–707, 2019\.
  * Raissi et al. [2020] M. Raissi, A. Yazdani, and G. E. Karniadakis.  Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations.  _Science_ , 367:1026–1030, 2020.
  * Rasmussen [2003] C. E. Rasmussen.  Gaussian processes for machine learning.  _In Journal fur Urologie und Urogynakologie. Berlin: Springer International Publishing. Berlin: Springer International Publishing_ , 7, 2003\.
  * Rudin [2019] C. Rudin.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ , 1:206–215, 2019.
  * Sanchis-Agudo et al. [2023] M. Sanchis-Agudo, Y. Wang, L. Guastoni, K. Duraisamy, and R. Vinuesa.  Easy attention: A simple self-attention mechanism for transformer-based time-series reconstruction and prediction.  _Preprint arXiv:2308.12874_ , 2023.
  * Schmid [2010] P. J. Schmid.  Dynamic mode decomposition of numerical and experimental data.  _Journal of Fluid Mechanics_ , 656:5–28, 2010.
  * Sitte and Doan [2022] M. P. Sitte and N. A. K. Doan.  Velocity reconstruction in puffing pool fires with physics-informed neural networks.  _Physics of Fluids_ , 34:087124, 2022.
  * Smagorinsky [1963] J. Smagorinsky.  General circulation experiments with the primitive equations: I. The basic experiment.  _Monthly Weather Review_ , 91:99–164, 1963.
  * Solera-Rico et al. [2023] A. Solera-Rico, C. S. Vila, M. A. Gómez, Y. Wang, A. Almashjary, S. T. M. Dawson, and R. Vinuesa.  $\beta$-variational autoencoders and transformers for reduced-order modelling of fluid flows.  _Preprint arXiv:2304.03571_ , 2023.
  * Sonoda et al. [2023] T. Sonoda, Z. Liu, T. Itoh, and Y. Hasegawa.  Reinforcement learning of control strategies for reducing skin friction drag in a fully developed turbulent channel flow.  _Journal of Fluid Mechanics_ , 960:A30, 2023.
  * Srinivasan et al. [2019] P. A. Srinivasan, L. Guastoni, H. Azizpour, P. Schlatter, and R. Vinuesa.  Predictions of turbulent shear flows using deep neural networks.  _Physical Review Fluids_ , 4:054603, 2019.
  * Suárez et al. [2023] P. Suárez, F. Alcántara-Ávila, A. Miró, J. Rabault, B. Font, O. Lehmkuhl, and R. Vinuesa.  Active flow control for three-dimensional cylinders through deep reinforcement learning.  _Preprint arXiv:2309.02462_ , 2023.
  * Taira et al. [2017] K. Taira, S. L. Brunton, S. Dawson, C. W. Rowley, T. Colonius, B. J. McKeon, O. T. Schmidt, S. Gordeyev, V. Theofilis, and L. S. Ukeiley.  Modal analysis of fluid flows: An overview.  _AIAA Journal_ , 55(12):4013–4041, 2017.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.  Attention is all you need.  _Advances in Neural Information Processing Systems_ , 30, 2017.
  * Vignon et al. [2023] C. Vignon, J. Rabault, and R. Vinuesa.  Recent advances in applying deep reinforcement learning for flow control: perspectives and future directions.  _Physics of Fluids_ , 35:031301, 2023.
  * Vinuesa and Brunton [2022] R. Vinuesa and S. L. Brunton.  Enhancing computational fluid dynamics with machine learning.  _Nature Computational Science_ , 2:358–366, 2022.
  * Vinuesa and Sirmacek [2021] R. Vinuesa and B. Sirmacek.  Interpretable deep-learning models to help achieve the Sustainable Development Goals.  _Nature Machine Intelligence_ , 3:926, 2021.
  * Vinuesa et al. [2022] R. Vinuesa, O. Lehmkuhl, A. Lozano-Durán, and J. Rabault.  Flow control in wings and discovery of novel approaches via deep reinforcement learning.  _Fluids_ , 865:281–302, 2022.
  * Vinuesa et al. [2023] R. Vinuesa, S. L. Brunton, and B. J. McKeon.  The transformative potential of machine learning for experiments in fluid mechanics.  _Nature Reviews Physics_ , 5:536–545, 2023.
  * Weatheritt and Sandberg [2016] J. Weatheritt and R. D. Sandberg.  A novel evolutionary algorithm applied to algebraic modifications of the RANS stress-strain relationship.  _Journal of Computational Physics_ , 325:22–37, 2016.
  * Weatheritt and Sandberg [2017] J. Weatheritt and R. D. Sandberg.  The development of algebraic stress models using a novel evolutionary algorithm.  _International Journal of Heat and Fluid Flow_ , 68:298–318, 2017.
  * Wu et al. [2019] J. Wu, H. Xiao, R. Sun, and Q. Wang.  Reynolds-averaged Navier–Stokes equations with explicit data-driven Reynolds stress closure can be ill-conditioned.  _Journal of Fluid Mechanics_ , 869:553–586, 2019.
  * Wu et al. [2022] W. Wu, C. Meneveau, R. Mittal, A. Padovan, C. W. Rowley, and L. Cattafesta.  Response of a turbulent separation bubble to zero-net-mass-flux jet perturbations.  _Physical Review Fluids_ , 7:084601, 2022.
  * Yousif et al. [2022] M. Z. Yousif, L. Yu, and H.-C. Lim.  Super-resolution reconstruction of turbulent flow fields at various Reynolds numbers based on generative adversarial networks.  _Physics of Fluids_ , 34:015130, 2022.
  * Yousif et al. [2023] M. Z. Yousif, M. Zhang, L. Yu, R. Vinuesa, and H. Lim.  A transformer-based synthetic-inflow generator for spatially-developing turbulent boundary layers.  _Journal of Fluid Mechanics_ , 957:A6, 2023.