# Bootstrapping SparseFormers from Vision Foundation Models

Ziteng Gao Zhan Tong Kevin Qinghong Lin Joya Chen Mike Zheng Shou✉
Show Lab, National University of Singapore

###### Abstract

The recently proposed SparseFormer architecture provides an alternative
approach to visual understanding by utilizing a significantly lower number of
visual tokens via adjusting RoIs, greatly reducing computational costs while
still achieving promising performance. However, training SparseFormers from
scratch is still expensive, and scaling up the number of parameters can be
challenging. In this paper, we propose to bootstrap SparseFormers from ViT-
based vision foundation models in a simple and efficient way. Since the
majority of SparseFormer blocks are the standard transformer ones, we can
inherit weights from large-scale pre-trained vision transformers and freeze
them as much as possible. Therefore, we only need to train the SparseFormer-
specific lightweight focusing transformer to adjust token RoIs and fine-tune a
few early pre-trained blocks to align the final token representation. In such
a way, we can bootstrap SparseFormer architectures from various large-scale
pre-trained models (e.g., IN-21K pre-trained AugRegs or CLIPs) using a rather
smaller amount of training samples (e.g., IN-1K) and without labels or
captions within just a few hours. As a result, the bootstrapped unimodal
SparseFormer (from AugReg-ViT-L/16-384) can reach $84.9\%$ accuracy on IN-1K
with only $49$ tokens, and the multimodal SparseFormer from CLIPs also
demonstrates notable zero-shot performance with highly reduced computational
cost without seeing any caption during the bootstrapping procedure. In
addition, CLIP-bootstrapped SparseFormers, which align the output space with
language without seeing a word, can serve as efficient vision encoders in
multimodal large language models. Code will be publicly available at
https://github.com/showlab/sparseformer

†† ✉: Corresponding Author.

##  1 Introduction

Figure 1: SparseFormer bootstrapping procedure and task evaluation. a) With
only images as inputs, we bootstrap SparseFormers from vision foundation
models by inheriting weights and aligning final representations with much
fewer tokens (_e.g_., $0.25\times$). b) Bootstrapped SparseFormers can serve
as the efficient vision encoder in either off-the-shelf or fine-tuning manner
for both unimodal and multimodal tasks.

Large-scale pre-trained vision models [76, 13, 28, 49, 59], or vision
foundation models, exhibit strong transferable or zero-shot capabilities after
pre-training. Most vision foundation models, e.g., ViTs [16], are simply based
on the standard transformer encoder architecture [15], which is known for its
high capacity and proven effectiveness in modeling massive natural language
corpora [4, 64]. This large capacity is also observed on the vision domain.
For instance, the large-scale ViT pre-trained on the JFT-300M dataset [60]
delivers notable performance on various down-streaming tasks and scaling up
such models and data still consistently leads to consistent improvements [76,
13, 49]. Despite their improved performance, these vision foundation
transformers require significantly more computational resources and memory,
both in the training and inference stage, especially when working with high-
resolution images, due to the transformer architecture. As an example,
processing a single $384\times 384$ resolution image with ViT-L/16 requires
handling 576 visual tokens, and attention operators between these tokens take
up memory and computation quadratically with the number of tokens.

Recently, SparseFormer [20] has been proposed as an alternative vision
transformer architecture with much fewer visual tokens in the latent space
rather than the original image space. Each token is associated with a region
of interest (RoI) descriptor, and SparseFormer exploits the recurring focusing
transformer to adjust token RoIs and sample image features sparsely according
to these RoIs. This design allows tokens to be deformable and adjustable in
terms of their spatial locations. By training with only classification labels,
these latent tokens can focus on foreground objects and exclude non-
informative backgrounds. Therefore, the number of tokens in SparseFormer can
be greatly reduced compared to conventional vision transformers, as well as
the computational cost and memory footprint. Though being effective in the low
compute region, SparseFormer is found to be hard to scale up, and the largest
SparseFormer variant in only achieves 84.8% top-1 acc on ImageNet 1K [20].
Moreover, despite tokens being reduced, training SparseFormers from scratch is
also time-costing on the ImageNet benchmark, _e.g_., $\sim$12 A100 GPU days
for a base variant, not to mention the large-scale vision language training.
Therefore, it is interesting to explore how to efficiently train SparseFormers
to serve as efficient visual transformer alternatives but with capabilities as
strong as possible.

In this paper, we present a simple solution to this question: to “bootstrap”
SparseFormers from large-scale pre-trained vision foundation models, _e.g_.,
AugRegs [59] and CLIPs [49]. By “bootstrapping”, we mean to firstly inherit
large-scale pre-trained weights from foundation models into the standard
transformer encoder blocks in SparseFormers. And then, we randomly initialize
the focusing transformer in SparseFormers and train the combined SparseFormers
to align the final representation with foundation models with much fewer
tokens, as depicted in Figure 1. For preserving the structure of the output
space, we only tune a moderate early blocks with pre-trained weights. Thanks
to the SparseFormer efficiency and reuse of pre-trained parameters, we can
quickly bootstrap scaled-up SparseFormer variants in few hours. For instance,
bootstrapping from AugReg-ViT-L/16 ($85.8\%$ on IN-1K with $388$ imgs/s) only
takes just 6 hours on 8 A5000s, and the resulted SparseFormer achieves
$84.5\%$ IN-1K accuracy, using only 49 visual tokens, with $1557$ imgs/s
throughput. Continuing bootstrapping with 80 tokens leads to $85.5\%$
accuracy, only $0.3\%$ lagging behind AugReg-ViT-L/16 but with $2.4\times$
throughput. Bootstrapped SparseFormers can also serve as backbones for
semantic segmentation, reaching $51+$ mIoU on ADE20k [78] via 256 tokens for a
512$\times$512 input.

Moreover, since the bootstrapping procedure only needs images as inputs
without any labels, we can also bootstrap SparseFormers from CLIP models to
output the visual embedding in the language-aligned space just on ImageNet-1K.
Without seeing any caption, SparseFormer bootstrapping from CLIP ViT-L/14-336
demonstrates $75.9\%$ ImageNet-1K zero-shot accuracy with only 0.25$\times$
tokens, as well as the $57.0\%$ I$\to$T@1 retrieval score on the out-of-domain
MS COCO [37]. In addition to that, we can also incorporate our bootstrapped
SparseFormers into multimodal large language models (MLLMs) seamlessly without
further fine-tuning and obtain promising results on the multimodal question
answering ScienceQA dataset [41].

##  2 Related work

###  2.1 Vision Foundation Models

The term “foundation model” was first introduced in Bommasani et al. [3] to
refer to highly capable language transformers that were pre-trained on massive
data samples and can be easily adapted to various downstream tasks. Examples
of such models include the seminal BERT [15] and autoregressive GPT series
[47, 48, 4]. The concept “foundation model” also applies to computer vision
area as well, dating back to the previous cornerstone vision backbones [57,
22] pre-trained on the ImageNet. Recent work [16, 39, 30, 59] on computer
vison focuses on pre-training models on large-scale datasets, such as
ImageNet-21K [14] and JFT-300M [60], by the classification supervision with
annotations. The reliance on human annotations of these datasets somewhat
imposes a constraint on pre-training such foundation models and several work
explores contrastive learning [7, 21, 23], masked modeling [24, 62, 58, 18] or
other self-supervised learning techniques [5, 45] to alleviate this issue.
Despite the hunger for annotations, training time and hardware demands make
training a vision foundation model a difficult and expensive thing.

Data in various domains, such as web images [55], is naturally multimodal in
language and vision, which has led to the interest in language vision pre-
training for vision models [32, 33, 77, 72, 68, 73, 61]. Among these, CLIPs
[49] as the most canonical multimodal models align the output embedding space
of the language transformer and the vision transformer, showing remarkable
zero-shot and transfering capabilities on various vision-language
understanding tasks. The success of CLIP has also benefited many downstream
tasks, including text-conditioned image generation [51, 53, 54] and open-
vocabulary detection [42, 35, 79, 31]. Furthermore, given the growing
influence and availability of large language models (LLMs) [44, 64], recent
studies [33, 34, 80, 38, 12] aim to build multimodal models with image inputs
by using CLIP pre-trained vision transformers. But still, training these
vision encoders from scratch can be extremely expensive, and such dense
transformers can be computationally demanding during inference, particularly
for larger inputs.

###  2.2 Efficient Vision Foundation Models

It is always an appealing research topic to build efficient vision
transformers, _e.g_., with efficient attention mechanisms [65, 29, 10], or
using compact transformer architectures [39, 25, 66, 17]. Beside these, a
research line aims to investigate the redundancy of tokens and thus reduce the
token number [67, 52, 36, 74, 19, 2, 6] to expedite the inference phase.
Knowledge distillation on [26, 43, 63, 1, 69] have also been widely used to
transfer knowledge from large vision transformers to small ones. Apart from
unimodal models, there has been an increasing emphasis on efficient multimodal
transformers [46, 56].

The recently proposed SparseFormer [20] as a vision transformer variant
exploits the token adjusting mechanism with much fewer tokens to expedite the
visual understanding task. SparseFormer can be also seen as a vision
transformer with the token reduction right from the start, but lengthy
training from scratch is required. In this paper, we mainly discuss how to
bootstrap SparseFormers from vision foundation models with limited training
time and data samples, and use the bootstrapped SparseFormers as the efficient
vision foundation models with remarkable performance in both unimodal and
multimodal settings.

##  3 Method

In this section, we will first briefly revisit the SparseFormer architecture
[20]. Next, we will describe in detail bootstrapping SparseFormers from vision
foundation models with the limited training time, hardware budgets, and data
samples.

Figure 2: The detailed bootstrapping procedure. We typically set the number of
sparse latent tokens in SparseFormers to $1/4$ those in vision transformers.
The starting index of tunable blocks $i$ is $N/3$ and the frozen is $2N/3$ for
all bootstrapping settings. Note that [CLS] represents the extra token in
vision transformers besides visual tokens and there is no classification
supervision on [CLS] in bootstrapping.

###  3.1 Prerequisites: the SparseFormer Architecture

The SparseFormer architecture, as a vision transformer variant, aims to
represent an image by a highly reduced number of tokens and their
corresponding adjustable RoIs. SparseFormer consists up of two components: the
focusing transformer and the cortex transformer. The focusing transformer,
which is designed to be with minimal parameters and computational costs,
iteratively adjusts token RoIs to focus on foregrounds. The cortex transformer
is exactly a plain vision transformer encoder block, similar to [16], and with
the majority of parameters and computation used by SparseFormer. Specifically,
given the latent token embedding set
$\mathbf{T}=[\mathbf{t}_{1},\mathbf{t}_{2},\cdots,\mathbf{t}_{N}]$ and the
token RoI set
$\mathbf{B}=[\mathbf{b}_{1},\mathbf{b}_{2},\cdots,\mathbf{b}_{N}]$, where $N$
is the number of latent tokens, the focusing transformer operates as follows:

$\displaystyle\mathbf{T}^{i+1},\mathbf{B}^{i+1}=\mathrm{FocusingTransformer}(\mathbf{T}^{i},\mathbf{B}^{i}),$
(1)

where $\mathbf{T}^{1}$ and $\mathbf{B}^{1}$ are initial token embedding set
and initial RoI set, and they are learnable parameters of models, independent
of input images. In the typical setting, the focusing transformer is executed
in 4 iterations. In each iteration, the focusing transformer extracts image
features according the current token RoI $\mathbf{b}_{(\cdot)}=[x,y,w,h]$ into
token embedding $\mathbf{t}_{(\cdot)}$, perform self-attention and feed
forward network on embeddings, and adjust token RoIs. The RoI adjusting
mechanism uses the normalized delta form:

$\displaystyle x^{\prime}=x+\Delta_{x}w$
$\displaystyle,y^{\prime}=y+\Delta_{y}h,$ (2) $\displaystyle
w^{\prime}=w\cdot\exp(\Delta_{w})$
$\displaystyle,h^{\prime}=h\cdot\exp(\Delta_{h}),$ (3)

where the tuple $(\Delta_{x},\Delta_{y},\Delta_{w},\Delta_{h})$ is produced by
the token embedding in the focusing transformer. All operations are applied
individually to each token, except for self-attention which involves token
interaction.

After the focusing transformer, these token RoIs tend to cluster around
foreground objects [20]. Then, the so-called cortex transformer, whose
architecture is exactly a standard transformer encoder, operates on the token
embedding as:

$\displaystyle\mathbf{T}^{\mathrm{final}}=\mathrm{CortexTransformer}(\mathbf{T}^{\prime}),$
(4)

where we can see that the cortex transformer just processes token embeddings
and does not rely on the token RoIs anymore. With the RoI adjusting mechanism,
SparseFormer utilizes a highly reduced number of tokens (e.g., $0.25\times$)
for visual understanding by only attending to foreground objects and excluding
backgrounds.

Feasibility to bootstrap SparseFormers from ViTs. Since the cortex transformer
strictly follows the standard transformer encoder architecture, it is feasible
to inherit pre-trained vision transformer weights. Once the weights of the
cortex transformer are inherited, we only need to train the focusing
transformer from scratch to align its representation into the cortex
transformer to be compatible with inherited cortex transformer weights. Thus,
when the input representation well aligned, the output representation of
SparseFormer will also align with the original pre-trained vision
transformers, which is especially beneficial for language-aligned vision
representations, such as CLIPs [49]. We term this procedure as “bootstrapping”
SparseFormers from large-scale pre-trained vision foundation models.

###  3.2 Bootstrapping from Vision Foundation Models

As discussed above, we need to align the output representation of the focusing
transformer with the input representation of inherited vision transformers.
However, this straightforward idea is infeasible since the shape of these two
representations typically does not match as the number of SparseFormer tokens
is much lower than ones in most vision transformers.

Therefore, we resort to directly aligning the final representation of
SparseFormer to one from pre-trained vision transformers. Specifically, we
propose a simple method to bootstrap SparseFormers from vision foundation
models, which are mostly based on transformers. First, we design SparseFormer
variants to match the dimension and the number of their cortex transformer
blocks with these pre-trained vision transformers. We inherit these large-
scale pre-trained weights of these transformers and load them into cortex
transformer blocks in SparseFormers. Then, we initialize the focusing
transformer from random and concatenate these two to build a complete
SparseFormer.

Once a SparseFormer is constructed, we align the final representation from
SparseFormer with that of the pre-trained vision transformer given the same
image input, as depicted in Figure 2. We simply use the cosine loss to align
these two final token representations:

$\displaystyle\ell=1-\frac{\mathbf{t}_{\mathrm{sparseformer}}^{T}\mathbf{t}_{\mathrm{pretrained}}}{\|\mathbf{t}_{\mathrm{sparseformer}}\|\|\mathbf{t}_{\mathrm{pretrained}}\|},$
(5)

where $\mathbf{t}_{\mathrm{sparseformer}}$ and
$\mathbf{t}_{\mathrm{pretrained}}$ are the final token embeddings in the last
embedding space of models, which are [CLS] tokens before the classifier layer
in classification models or after the projection layer in CLIP models. We can
bootstrap SparseFormers from large-scale pre-trained vision foundation models
without labels, since the alignment target from pre-trained models,
$\mathbf{t}_{\mathrm{pretrained}}$, already has rich semantics. All we need to
do is to seek for a set of unlabeled supporting images. We find that the
medium-sized ImageNet-1K training set [14], which is most publicly available
to its scale, with minimal augmentation is a sufficient supporting image set
for bootstrapping competitive SparseFormers from pre-trained models.

Difference with distillation methods. It is important to note that the cosine
loss between two token embeddings is not dependent on pre-defined label sets,
and we do not exploit any labels or captions during the bootstrapping process.
This differs a lot from existing distillation methods, which typically make
use of specified label sets [26, 63] for classification models or vision-
language pairs for multimodal models [70]. Also, distillation methods do not
reuse teacher model weights since the student architecture usually differs
from the teacher, and thus require a lengthy schedule to transfer knowledge
effectively. In comparison, our method can bootstrap a 200M SparseFormer
variant in a few hours ($20$ epochs) with only 1.2M images from ImageNet as
inputs.

Truncate the leading, tune the middle, and freeze the ending blocks. The
leading blocks of pre-trained vision transformers are usually specialized for
low-level visual modeling [50], which overlaps with the role of the focusing
transformer in SparseFormers. Thus, we choose to discard a moderate number,
$1/3$, of leading transformer blocks when constructing our SparseFormer
variants, which also help reduce the compute need. Since the bootstrapping
goal is to well align the final representation, we set a few middle
transformer blocks tunable starting from inherited weights to adapt the output
of the focusing transformer. We leave frozen the rest transformer blocks in
the ending to preserve the structure of the output space by pre-trained
weights as much as possible.

##  4 Experiments

We start experiments on bootstrapping SparseFormers from large-scale pre-
trained unimodal models and then discuss vision-language pre-trained ones.

###  4.1 Bootstrapping from Unimodal Models

We choose the well-established AugReg transformers [59], which strictly follow
the original ViT architecture, as the unimodal classification models to
bootstrap from. The AugReg models are pre-trained by the supervised
classification on the ImageNet-21K dataset with well-curated data augmentation
and regularization techniques.

#### Model configurations and experimental settings.

We design two SparseFormer variants (SF-B${}_{\textrm{AugReg}}$ and
SF-L${}_{\textrm{AugReg}}$) according to the base and large variant
(ViT-B/16-224${}_{\textrm{AugReg}}$ and ViT-L/16-224${}_{\textrm{AugReg}}$) in
Table 1. We set the token dimension in the focusing transformers half those of
the cortex transformers except that the last focusing transformer block
performs sampling operations on the full dimension, in accord with [20].
Unlike the leading one, the last focusing transformer block does not involve
FFNs and self-attention layers. We append a learnable [CLS] to latent tokens
after the focusing stage in SparseFormer and use its final representation for
bootstrapping and classification. To be more consistent with AugReg models
(CLIPs below as well) which add positional encoding to visual tokens, we
inject positional encoding based on token RoIs in the sinusoidal form in each
focusing iteration. To further reduce compute and make SparseFormer even more
sparse, we reduce the channel of the early convolution from $96$ to $64$, and
shrink the number of sampling points for a token in the sparse sampling
procedure from $36$ to $16$.

model | #tokens |  focusing iterations |  focusing dimension |  #truncate blocks |  #tunable blocks | FLOPs | #Params
---|---|---|---|---|---|---|---
ViT-B/16-224${}_{\textrm{AugReg}}$ | 196 | - | - | - | - | 17.5G | 64M
SF-B${}_{\textrm{AugReg}}$ | 49 | 4+1 | 384 | $4$ | $4$ | 3.8G | 86M
ViT-L/16-224${}_{\textrm{AugReg}}$ | 196 | - | - | - | - | 61.6G | 304M
SF-L${}_{\textrm{AugReg}}$ | 49 | 4+1 | 512 | $8$ | $8$ | 11.4G | 213M
Table 1: Configurations of SparseFormers.

We choose the publicly-available medium-sized ImageNet-1K training set as our
bootstrapping image set. The data augmentation is set to be minimal: random
horizontal flipping with the probability $0.5$ and random resized cropping
with from the scale $(0.5,1.0)$ and the aspect ratio $(3/4,4/3)$. We use a
budget of $20$ epochs as our bootstrapping schedule, where learning rate
begins at $2\times 10^{-4}$ and follows a half-cosine decay schedule after the
first warming-up epoch. In the warming-up epoch, the inherited pre-trained
weights keep frozen for training stability. In bootstrapping, only the
focusing transformer and tunable blocks of cortex transformer are learnable.
All the rest parameters inherited from AugReg models, including the final
layer normalization and classification layer, remain frozen.

model | top-1 acc. | #Params | FLOPs | img/s
---|---|---|---|---
ViT-B/16-224${}_{\textrm{AugReg}}$, 196 tokens | 84.6 | 86M | 17.5G | 1126
SF-B${}_{\textrm{AugReg}}$, 49 tokens | 82.5 | 64M |  3.8G↓78.3% |  3001↑166%
SF-B${}_{\textrm{AugReg}}$, 80 tokens$\uparrow$ | 83.4 | 64M |  6.2G↓64.6% |  2080↑85%
SF-B${}_{\textrm{AugReg}}$, 80 tokens$\uparrow$, 320px$\uparrow$ | 83.7 | 64M |  6.3G↓64.0% |  1898↑67%
ViT-L/16-224${}_{\textrm{AugReg}}$, 196 tokens | 85.8 | 304M | 61.6G | 388
SF-L${}_{\textrm{AugReg}}$, 49 tokens | 84.5 | 213M |  11.4G↓81.5% |  1557↑301%
SF-L${}_{\textrm{AugReg}}$, 80 tokens$\uparrow$ | 85.2 | 213M |  18.6G↓69.8% |  1008↑160%
SF-L${}_{\textrm{AugReg}}$, 80 tokens$\uparrow$, 320px$\uparrow$ | 85.5 | 213M |  18.7G↓69.6% |  928↑139%
ViT-L/16-384${}_{\textrm{AugReg}}$, 576 tokens | 86.9 | 304M | 191G | 105
SF-L${}_{\textrm{AugReg}}$, 49 tokens | 84.9 | 213M |  11.6G↓93.9% |  1353↑1189%
SF-L${}_{\textrm{AugReg}}$, 144 tokens$\uparrow$ | 86.5 | 213M |  33.9G↓82.3% |  560↑433%
SF-L${}_{\textrm{AugReg}}$, 196 tokens$\uparrow$ | 86.7 | 213M |  46.5G↓75.7% |  378↑260%
Table 2:  Results of SparseFormers bootstrapped from AugReg models. The
default SparseFormer token number is 49. Throughputs are measured with half-
precision data type and batch size of 128 on one A5000 GPU.

#### Bootstrapped SparseFormers with 49 tokens.

We show results of bootstrapped SparseFormer from AugReg classification models
in Table 2. The default token number is 49 for SparseFormer models. From the
results, we can see that SparseFormer models with 49 tokens have achieved
promising results, _e.g_., 84.9% top-1 accuracy for SF-L${}_{\textrm{AugReg}}$
bootstrapping from ViT-L/16-384${}_{\textrm{AugReg}}$. Note that the number of
tokens, _e.g_. 49, is fixed from the beginning to the end in SparseFormer
models, which differs from recent other methods progressively dropping tokens
through blocks [2, 6]. To our best knowledge, we are the first to obtain
$\sim$85% top-1 accuracy on ImageNet-1K with only 49 tokens in all blocks for
vision transformers.

#### Continue to bootstrapping with more tokens.

Although SparseFormers with 49 tokens yield promising results, they are still
lagging behind those models to bootstrap from with a non-negligible margin. To
minimize the gap, we further continue to bootstrapping SF models with the
increased number of tokens from default SF models. Note that these increased
numbers are still much lower than original vision transformers, especially for
high resolution one. The continued bootstrap lasts for 5 epochs with LR
starting from $5\times 10^{-5}$, also following a half-cosine decay scheme. We
denote these models with an extra notation $\uparrow$. From the table, the
gaps between SparseFormers and AugRegs become closer when original models
become larger, especially for SF-L${}_{\textrm{AugReg}}$ with 196 tokens,
which is on par with ViT-L/16-384${}_{\textrm{AugReg}}$ but at about 3$\times$
real-time throughput. When combined with the higher input resolution,
SparseFormer models still yield improvements on the same number of tokens at a
marginal additional burden. Note that as a sparse architecture, the input
resolution has little impact on computational costs of the most components in
SparseFormers.

bootstrapping |  with cls loss |  w/o reusing weights |  KL distill $\tau=3$ |  KL distill $\tau=5$ |  supervised cls from scratch
---|---|---|---|---|---
82.5 | 77.4 | 69.5 | 81.0 | 81.5 | 80.0
Table 3:  Ablations on bootstrapping SF-B${}_{\textrm{AugReg}}$ with 49
tokens. Except for ‘from scratch’ (supervised classification, from scratch,
300ep), we keep the bootstrapping training settings the same (_e.g_.,
trainable parameters, epochs & augmentation).

#### Ablation studies on bootstrapping.

Our bootstrapping procedure is simple as it just aligns final representations
of SparseFormers and large-scale pre-trained models given the same image
input. Labels are not needed through the bootstrapping procedure except for
the evaluation on benchmarks. Here we ablate the bootstrapping design in Table
3. Interestingly, aligning the final representations along with the
classification loss (‘with cls loss’ in the table) actually impairs the
performance. We suspect that the supervised classification requires strong
augmentation and regularization to be stable with the alignment. We also
replace our token representation alignment with KL distillation loss with
different temperatures on the ImageNet labels. This simple distillation yields
promising results but still with $>1\%$ lagging behind our bootstrapping.

We also include the bootstrapping result without the reusage of pre-trained
weights (but the final linear classifier remained) and train the entire
transformer to align the final representation for 20 epochs. As expected, it
yields the inferior result, and more training epochs may be required to well
align from scratch.

#### Comparison with token reduction methods.

model | top-1 acc. | FLOPs | img/s
---|---|---|---
ViT-B/16-224${}_{\textrm{AugReg}}$ | 84.6 | 17.5G | 1126
ToMe@16, off-the-shelf | 80.4 | 8.8G↓49.7% | 1980↑75%
ToMe@16, fine-tuned | 82.8 | 8.8G↓49.7% | 1980↑75%
DiffRate, fine-tuned${\dagger}$ | 83.2 | 11.5G↓34.3% | 1400↑24%
SF-B${}_{\textrm{AugReg}}$, 80 tokens$\uparrow$ | 83.4 |  6.2G↓64.6% |  2080↑84%
SF-B${}_{\textrm{AugReg}}$, 80 tokens$\uparrow$, 320px$\uparrow$ | 83.7 |  6.3G↓64.0% |  1898↑68%
SF-B${}_{\textrm{AugReg}}$, 100 tokens$\uparrow$, 320px$\uparrow$ | 84.0 |  7.8G↓55.4% |  1579↑40%
ViT-L/16-224${}_{\textrm{AugReg}}$ | 85.8 | 61.6G | 388
ToMe@12, off-the-shelf | 20.6${\ddagger}$ | 20.8G↓66.2% | 1006↑159%
ToMe@8, off-the-shelf | 83.5 | 31.0G↓49.7% | 700↑80%
ToMe@8, fine-tuned | 84.8 | 31.0G↓49.7% | 700↑80%
DiffRate, fine-tuned${\dagger}$ | 85.7 | 42.3G↓31.3% | 452↑16%
SF-L${}_{\textrm{AugReg}}$, 80 tokens$\uparrow$ | 85.2 |  18.6G↓69.8% |  1008↑160%
SF-L${}_{\textrm{AugReg}}$, 80 tokens$\uparrow$, 320px$\uparrow$ | 85.5 |  18.7G↓69.6% |  928↑139%
SF-L${}_{\textrm{AugReg}}$, 100 tokens$\uparrow$, 320px$\uparrow$ | 85.7 |  23.4G↓62.0% |  752↑94%
ViT-L/16-384${}_{\textrm{AugReg}}$ | 86.9 | 191G | 105
ToMe@40, off-the-shelf | 8.8${\ddagger}$ | 56.0G↓70.7% | 315↑200%
ToMe@23, off-the-shelf | 86.2 | 96.4G↓49.5% | 187↑78%
ToMe@23, fine-tuned | 86.8 | 96.4G↓49.5% | 187↑78%
SF-L${}_{\textrm{AugReg}}$, 196 tokens$\uparrow$ | 86.7 |  46.5G↓75.7% |  378↑260%
Table 4:  Comparisons on SparseFormers with token reduction methods, ToMe [2]
and DiffRate [6]. ${\dagger}$ use MAE [24] pre-trained weights. ${\ddagger}$
ToMe is not unfriendly to large token merging rates.

SparseFormers can be considered as a method to reduce the number of visual
tokens in vision transformers after pre-training, similar to token pruning or
merging approaches [36, 2, 19, 6]. Here, we investigate the effectiveness of
SparseFormers from the token reduction perspective and compare them to state-
of-the-art reduction methods, ToMe [2] and DiffRate [6], in Table 4. In
addition to using off-the-shelf AugReg models with ToMe, we also fine-tune
AugRegs with ToMe using the fine-tuning recipe described in [24] to ensure a
fair comparison with our methods, as additional fine-tuning is required. We
sweep to find best fine-tuning LRs for these ToMe AugRegs respectively.
DiffRate on MAE models are also included, where the
ViT-L/16-224${}_{\textrm{MAE}}$ baseline reports a slightly better result than
ViT-L/16-224${}_{\textrm{AugReg}}$. As shown in the table, SparseFormer models
achieve the best trade-off between the accuracy and the actual throughput,
especially for the large models where many tokens may be redundant. In fact,
tokens in bootstrapped SparseFormers can be further applied with these token
reduction methods but this is beyond this paper scope.

#### Dense prediction task.

model | mIoU | #Params | FLOPs | train mem. | img/s
---|---|---|---|---|---
Swin-L [39] \+ UPerNet [71]${\dagger}$ | 51.1 | 234M | 408G | 11G, 2bs | 20
ViT-Adapter-B${}_{\textrm{AugReg}}$ [8] \+ UPerNet [71] | 51.9 | 134M | 376G | 50G, 2bs | 19
ViT-Adapter-L${}_{\textrm{AugReg}}$ [8] \+ UPerNet [71] | 53.4 | 364M | 667G |  49G, 2bs${\ddagger}$ | 12
SF-B${}_{\textrm{AugReg}}$ segmentation (ours) | 49.3 | 68M | 36G | 10G, 4bs | 88
SF-L${}_{\textrm{AugReg}}$ segmentation (ours) | 51.5 | 216M | 77G | 13G, 4bs | 46
Table 5:  Segmentation performance on ADE20k val set [78]. ${\dagger}$ is reported by mmsegmentation [11]. ${\ddagger}$ is with gradient checkpointing. Throughput is measured with batch size 4 on A5000.  method |  visual encoder |  visual FLOPs |  IN-1K zero-shot@1 |  MS COCO I$\rightarrow$T@1 |  MS COCO T$\rightarrow$I@1 |  Flickr30k I$\rightarrow$T@1 |  Flickr30k T$\rightarrow$I@1
---|---|---|---|---|---|---|---
CLIP | ViT-B/16-224 | 17.5G | 68.3 | 52.4 | 33.0 | 81.9 | 62.0
CLIP | ViT-B/32-224 | 4.4G | 63.3 | 50.1 | 30.4 | 77.5 | 58.8
TinyCLIP [70]${\dagger}$ | ViT-63M/32-224 | 2.0G | 64.5 | 56.9 | 38.5 | 84.9 | 66.0
CLIP + SF (ours) |  SF-B${}_{\textrm{CLIP}}$ w/ 49 tokens | 3.8G | 66.0 | 47.7 | 31.5 | 76.3 | 59.6
CLIP | ViT-L/14-224 | 81.1G | 75.4 | 56.3 | 36.5 | 85.1 | 65.2
CLIP + SF (ours) |  SF-L${}_{\textrm{CLIP}}$ w/ 64 tokens | 14.8G | 73.6 | 52.3 | 34.8 | 80.3 | 62.8
CLIP + SF (ours) |  SF-L${}_{\textrm{CLIP}}$ w/ 128 tokens | 30.0G | 74.9 | 54.7 | 36.4 | 82.4 | 65.2
CLIP | ViT-L/14-336 | 191.0G | 76.6 | 57.9 | 37.1 | 87.4 | 67.3
CLIP + SF (ours) |  SF-L${}_{\textrm{CLIP}}$ w/ 144 tokens | 33.9G | 75.9 | 57.0 | 38.3 | 84.6 | 67.7
Table 6:  Zero-shot and retrieval evaluation results. We individually
bootstrap SparseFormers from CLIP visual encoders in the grayed rows with
$<$2M ImageNet-1K images as inputs. ${\dagger}$ needs training on text-image
pairs of LAION-400M [55].

We here further investigate the ability of bootstrapped SparseFormers as pre-
trained backbones to perform dense per-pixel prediction task, _e.g_., semantic
segmentation. As latent tokens of SparseFormer are not structured in a grid-
like map like conventional vision transformers [16, 39, 66], it requires some
workarounds to obtain dense feature maps from these latent tokens. We
generally follow the idea of the original SparseFormer on semantic
segmentation to project latent tokens back into the original dense pixel space
according to their RoIs [20]. Here, we find that it is more effective to
straightforward project prediction logits of tokens, instead of embeddings,
back into the dense pixel space using simple single-head attention
aggregation:

$\displaystyle\mathbf{P}_{\textrm{token}}$
$\displaystyle=\textrm{classifier}(\mathbf{T})\in\mathbb{R}^{N\times L},$ (6)
$\displaystyle\mathbf{P}_{\textrm{dense}}$
$\displaystyle=\textrm{softmax}(\mathbf{Q}_{\textrm{dense}}\mathbf{K}_{\textrm{token}}^{T}/\sqrt{d}+\mathbf{B})\mathbf{P}_{\textrm{token}},$
(7)

where $\textrm{classifier}(\cdot)$ just classify the latent token into
prediction logits, $N$ is the number of latent tokens, and $L$ is the number
of classes. In the current implementation, we first use a linear layer to
reduce the latent token dimension to $d=256$ and introduce two small
transformer encoder blocks ($d=256$) to process latent tokens, and we use a
linear layer as our latent classifier.
$\mathbf{Q}_{\textrm{dense}}\in\mathbb{R}^{HW/4^{2}\times d}$ is the flattened
feature map transformed by one $3\times 3$ convolution from the early
convolution feature in SparseFormer with shape $H/4\times W/4$. We bias the
attention score by
$\mathbf{B}=\mathbf{B}_{\textrm{geometric}}+\mathbf{B}_{\textrm{predictive}}$,
where the geometric bias $\mathbf{B}_{\textrm{geometric}}$ is in the Gaussian-
like form w.r.t. token RoIs as described in [20]. The predictive bias is
produced by a linear layer whose input is the latent token embedding,
$\mathbf{B}_{\textrm{predictive}}=\mathrm{linear}(\mathbf{T})\in\mathbb{R}^{1\times
N}$, indicating the overall importance of token logits into the dense map. In
simple words, we map the latent token predictions into the dense grid with the
consideration of their RoIs and semantic importance, similar to [9]. The
classification supervision is imposed on the mapped dense predictions.

We generally follow the training recipe of [8], including learning rate and
drop path [27] settings, and finetune our bootstrapped SparseFormers (default
ones w/ 49 tokens) on the ADE20k dataset [78] with the increased token number
of 256. As our segementation models based on SparseFormer are memory friendly,
the batch size is increased to 4 and the total training iteration is reduced
to 80K. The results are reported in Table 5. We can see that the performance
of SparseFormer on the segmentation task is promising, with much fewer FLOPs
and much higher throughput. This efficiency is credited to fewer computation
over limited latent tokens in SparseFormers compared to conventional dense
vision transformers (_e.g_., 256 versus 1024 for ViT-Adapter [8] for
512$\times$512 inputs).

###  4.2 Bootstrapping from Multimodal Models

Till now, we have bootstrapped SparseFormers from unimodal classification
models and apply them to the downstreaming dense prediction task, showing
their efficiency and effectiveness. Since our bootstrap procedure directly
aligns the final representations to the pre-trained transformers, it can be
easily adopted for the fundamental vision-language CLIP [49] models as well.

#### Experimental setups.

The bootstrapping procedure for CLIP models exactly follows the same recipe in
unimodal classification AugRegs, as described in Sec 4.1, including the
training budget, the data augmentation, and the learning rate setting. We use
the official OpenAI CLIP pre-trained vision transformers to bootstrap from,
_i.e_., ViT-B/16-224, ViT-L/14-224, and ViT-L/14-336. We only bootstrap
SparseFormers from the CLIP visual encoders and leave the text encoders
untouched. It is worth noting that we still only use the ImageNet-1K training
set as the supporting image dataset, and we do not leverage any text-image
pairs to align the SparseFormer image embeddings with CLIP models. This
differs from TinyCLIP [70], which also inherits pre-trained weights but
requires at least 15M text-image pairs to distill smaller models.

#### Results.

We use the frozen CLIP text encoders in combination with our bootstrapped
SparseFormers as visual encoders to perform the zero-shot classification task
on ImageNet-1K validation set, as well as the retrieval task on MS COCO val
set [37] and the Flickr30k benchmark [75]. Results are presented in Table 6.
With only images as samples for aligning in the bootstrapped procedure,
SparseFormers as the visual encoders achieve decent zero-shot accuracies with
greatly reduced compute, especially for large and high resolution models. For
instance, our SparseFormer-L with 144 tokens is just 0.7 point behind the
ViT-L/14-336 with 576 tokens, and surpass the ViT-L/14-224 by 0.5 margin with
less than 0.5$\times$ computational cost. Interestingly, the zero-shot
retrieval performance on MS COCO and Flickr30k, which have more complicated
data distributions compared to the ImageNet samples for aligning, is also
quite effective.

###  4.3 Multimodal Large Language Models

vision encoder | acc | multi-modal only acc
---|---|---
ViT-L/16-224 | 91.2 | 88.6
SF-L${}_{\textrm{CLIP}}$ w/ 64 tokens | 89.2 | 84.5
SF-L${}_{\textrm{CLIP}}$ w/ 128 tokens | 89.8 | 85.8
Table 7:  In-place vision encoder replacement in LLaVa on the ScienceQA test
set.  Figure 3:  Visualization on the original SparseFormer-Tiny [20] and our
bootstrapped ViT-B/16-224${}_{\textrm{AugReg}}$. For each image, there are an
input image, token RoIs in the {first, third, last} stage, and sampling points
in the last stage in the focusing transformer from left to right.

Building large language models with integrated vision capabilities is an
emerging and appealing topic [38, 33, 34] within the computer vision
community. Since we have bootstrapped SparseFormers from CLIP models, a
natural question arises: can we seamlessly incorporate SparseFormer as the
vision encoder in the multimodal large language model without re-finetuning
LLMs. The answer is yes. We experiment with LLaVa [38], a recent large
language model that is fine-tuned for instructions and can accept images and
texts as the prompt. The vision encoder in LLaVA is typically the CLIP ViT-L
and it directly outputs all visual tokens (typical number is 256 for a 2242
image) as input tokens for autoregressive language modeling, meaning that
there are already 256 “word tokens” for image modeling before user input
prompt. In comparison, SparseFormers reduce this number of tokens to typically
$1/4\times$, which is more friendly for LLM text modeling.

To demonstrate quantitative multimodal LLM results, we benchmark the fine-
tuned LLaVa variant, llava-lcs558k-scienceqa-vicuna-13b-v1.3, for the
instruction following on the ScienceQA multi-choice question answering task
[41]. We find that the data distribution is unique on the ScienceQA dataset
and directly using bootstrapped SparseFormers in Sec 4.2 leads to inferior
results. Therefore, we continue bootstrapping SparseFormers from CLIP
ViT-L/14-224 using the ScienceQA training images for $\sim$1000 iterations to
adapt to this image domain for about 20 minutes. Then we replace the CLIP
vision encoder in LLaVa with SparseFormers without any further fine-tuning
LLM. Results are shown in Table 7. The bootstrapped SparseFormer as the vision
encoder preserves most language-vision abilities ($84.5$ vs $88.6$ in the
multimodal only entry) with only $1/4\times$ visual tokens as inputs to the
following autoregressive LLM. The performance gap between SparseFormers and
CLIP even narrows further when scaling to 128 tokens, which is still half the
size of CLIP tokens.

###  4.4 Visualization

We show the visualization of our bootstrapped SparseFormer and compare it to
the original SparseFormer [20] in Figure 3. While the bootstrapped
ViT-B/16-224${}_{\textrm{AugReg}}$ and the original SparseFormer-Tiny use
exactly the same number of visual tokens (_i.e_., 49), our bootstrapped model
demonstrates the better capability to focus on foregrounds and exclude non-
informative regions with less sampling points (36$\to$16 per token, as
described in Section 4.1). As a result, the bootstrapped SparseFormer from
large-scale pre-trained models can be even sparser in both focal ability and
sampling efficiency compared with the original one.

##  5 Conclusion

In this paper, we have proposed a simple but effective method to bootstrap
SparseFormers from large-scale pre-trained vision foundation transformers by
inheriting most pre-trained weights and explicitly aligning the final
representations. With the short training time (a few hours) and limited data
samples (only image samples from ImageNet-1K), the SparseFormers bootstrapped
from unimodal classification models can maintain the pre-trained capabilities
as much as possible with much fewer tokens and higher real-time throughput. In
particular, we can bootstrap a SparseFormer with only 49 tokens throughout all
transformer blocks that obtains 84.9 top-1 accuracy on ImageNet. Also,
SparseFormers can serve as backbone networks in the dense per-pixel semantic
segmentation task with decent results on the ADE20k benchmark but at 2$\times$
throughput. In addition to classification models, we can also bootstrap
SparseFormers from multimodal pre-trained transformer CLIPs by aligning their
visual representations, without seeing any text caption in the bootstrapping
procedure. With the CLIP text encoders reused and frozen, the bootstrapped
SparseFormers as vision encoders demonstrate strong text-based zero-shot
classification as well as image-to-text and text-to-image retrieval
capabilities but with down to 0.2$\times$ FLOPs. We also evaluate the
multimodal LLM performance with bootstrapped SparseFormers as the efficient
vision encoder.

#### Limitation

The limitation of our proposed bootstrapping procedure is that it presumes the
underlying transformer architecture of vision foundation models and thus can
only deal with transformer-based foundation models. This is mostly the case,
but there are some exceptions, _e.g_., [40]. Also, it necessitates the
availability of weights of vision foundation models, while that may not be
possible for some large-scale proprietary ones.

##  Appendix A Appendix

## A.1. More Ablations on Bootstrapping Settings

In Section 3, We have proposed to truncate the leading, tune the middle, and
freeze the ending pre-trained transformer blocks to further reduce the compute
and preserve the output embedding space of the foundation transformer to
bootstrap from. Here, we investigate the effect of this bootstrapping paradigm
in Table 8.

model |  #truncate blocks |  #tunable blocks |  IN-1K top-1 acc. | FLOPs | #Params
---|---|---|---|---|---
SF-B${}_{\textrm{AugReg}}$, default | $4$ |  $4$ of $8$ | 82.5 | 3.8G | 86M
SF-B${}_{\textrm{AugReg}}$, all frozen | $4$ |  $0$ of $8$ | 81.8 | 3.8G | 86M
SF-B${}_{\textrm{AugReg}}$, all tunable | $4$ |  $8$ of $8$ | 82.4 | 3.8G | 86M
SF-B${}_{\textrm{AugReg}}$, w/o truncation | $0$ |  $4$ of $12$ | 82.7 | 5.2G | 92M
SF-L${}_{\textrm{AugReg}}$, default | $8$ |  $8$ of $16$ | 84.5 | 11.4G | 213M
SF-L${}_{\textrm{AugReg}}$, all tunable | $8$ |  $16$ of $16$ | 84.4 | 11.4G | 213M
SF-L${}_{\textrm{AugReg}}$, w/o truncation | $0$ |  $8$ of $24$ | 84.3 | 16.4G | 314M
Table 8: Ablation on truncating, tuning, and freezing settings.

As shown in the table, bootstrapping SparseFormers without tuning pre-trained
transformer blocks (“all frozen”) leads to inferior results compared to ones
that do tune. This is expected since frozen pre-trained transformer blocks can
not adapt to the output of the focusing transformer during the bootstrapping
procedure. However, going to the opposite extreme of making all pre-trained
blocks tunable (“all tunable”) can also be lagging. This may be because the
frozen classifier relies on the structure of the well-preserved output
embedding space in our bootstrapping setting. We believe that this is also
true for vision language models. Besides that, we observe that bootstrapping
without truncating leading blocks can be very unstable, and lead to different
effects on SF-B${}_{\textrm{AugReg}}$ and SF-L${}_{\textrm{AugReg}}$ but with
much more FLOPs and parameters. Therefore, we choose our truncating the
leading, tuning the middle, and freezing the ending paradigm as our
bootstrapping design due to the reduced computation and minimal tunable
parameters.

## A.2. Experiment Settings in Details

We here describe more experiment details in the bootstrapping procedure. The
learning rate for tuning pre-trained transformer blocks is set to $0.1\times$
that of the focusing transformer to make the training more stable after the
warm-up. The focusing transformer in our designed SparseFormer variant
performs the feature sampling first, then self attention between tokens, the
feed-forward network, and then the RoI adjustment for each iteration, in
contrast to the original SparseFormer which the self attention is performed
first and the feature sampling then. We use this reversed order to prioritize
the self-attention interaction between different tokens with sampled features.
We use two-layered MLP to produce RoI adjusting deltas in the focusing
transformer.

Different from the original SparseFormers that do not inject positional
information into latent tokens, we inject RoI-based position encoding into
tokens after every feature sampling operation in the focusing transformer to
align with typical vision transformers. Our adopted positional encoding is
also sinusoidal but in a continuous form:

$\displaystyle\textsc{PE}_{v}=\left[\sin(\pi f_{0}v),\cos(\pi f_{0}v),\sin(\pi
f_{1}v),...\right]\in\mathbb{R}^{d/4},$

where $f_{i}$ is the frequency term that evenly lies in the exponential space
from $1$ to $f_{\textrm{max}}=128$ (there are $d/8$ frequency terms),
$v\in[v_{\textrm{left}},v_{\textrm{top}},v_{\textrm{right}},v_{\textrm{bottom}}]$
where each component is the normalized coordinate of a token RoI that lies in
$[0,1]$. The final positional encoding is these four positional encoding parts
concatenated:

$\displaystyle\textsc{PE}=\left[\textsc{PE}_{\textrm{left}}|\textsc{PE}_{\textrm{top}}|\textsc{PE}_{\textrm{right}}|\textsc{PE}_{\textrm{bottom}}\right]\in\mathbb{R}^{d}.$

## A.2. More Visualizations

Here, we visualize the detailed RoI adjustments in each iteration in the
focusing transformer of our bootstrapped SparseFormer
SF-B${}_{\textrm{AugReg}}$ in Figure 4 and 5.

Figure 4: RoI adjustments in each iteration in SF-B${}_{\textrm{AugReg}}$.

In addition to that, we perform comprehensive visualizations on assorted
bootstrapped SparseFormers (SF-B${}_{\textrm{AugReg}}$,
SF-L${}_{\textrm{AugReg}}$, SF-B${}_{\textrm{CLIP}}$, and
SF-L${}_{\textrm{CLIP}}$) on ImageNet-1K validation image samples in Figure 6
and 7. Bootstrapped SparseFormers exhibit better sparsity and localization on
foregrounds than the original SparseFormer.

Figure 5: RoI adjustments (cont’d). Figure 6: Visualizations on the original
SparseFormer and our bootstrapped SparseFormers. For each image, there are an
input image, token RoIs in the {first, third, last} stage, and sampling points
in the last stage in the focusing transformer from left to right. Figure 7:
Visualizations (cont’d).

## References

  * Beyer et al. [2022] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov.  Knowledge distillation: A good teacher is patient and consistent.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10925–10934, 2022.
  * Bolya et al. [2023] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman.  Token merging: Your ViT but faster.  In _International Conference on Learning Representations_ , 2023.
  * Bommasani et al. [2021] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _NeurIPS_ , pp. 1877–1901, 2020.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _ICCV_ , pp. 9630–9640. IEEE, 2021.
  * Chen et al. [2023a] Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo.  Diffrate : Differentiable compression rate for efficient vision transformers.  _CoRR_ , abs/2305.17997, 2023a.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International Conference on Machine Learning_ , 2020.
  * Chen et al. [2023b] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao.  Vision transformer adapter for dense predictions.  In _ICLR_. OpenReview.net, 2023b.
  * Cheng et al. [2021] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov.  Per-pixel classification is not all you need for semantic segmentation.  2021\.
  * Choromanski et al. [2020] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al.  Rethinking attention with performers.  _arXiv preprint arXiv:2009.14794_ , 2020.
  * Contributors [2020] MMSegmentation Contributors.  MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark.  https://github.com/open-mmlab/mmsegmentation, 2020.
  * Dai et al. [2023] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  _arXiv preprint arXiv:2305.06500_ , 2023.
  * Dehghani et al. [2023] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al.  Scaling vision transformers to 22 billion parameters.  In _International Conference on Machine Learning_ , pp. 7480–7512, 2023.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pp. 248–255. Ieee, 2009.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Fan et al. [2021] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer.  Multiscale vision transformers.  _arXiv:2104.11227_ , 2021.
  * Fang et al. [2023] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.  EVA: exploring the limits of masked visual representation learning at scale.  In _CVPR_ , pp. 19358–19369. IEEE, 2023.
  * Fayyaz et al. [2022] Mohsen Fayyaz, Soroush Abbasi Koohpayegani, Farnoush Rezaei Jafari, Sunando Sengupta, Hamid Reza Vaezi Joze, Eric Sommerlade, Hamed Pirsiavash, and Jürgen Gall.  Adaptive token sampling for efficient vision transformers.  In _European Conference on Computer Vision_ , pp. 396–414, 2022\.
  * Gao et al. [2023] Ziteng Gao, Zhan Tong, Limin Wang, and Mike Zheng Shou.  Sparseformer: Sparse visual recognition via limited latent tokens.  _arXiv preprint arXiv:2304.03768_ , 2023.
  * Grill et al. [2020] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent: A new approach to self-supervised learning.  In _Advances in Neural Information Processing Systems_ , 2020.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2016.  doi: 10.1109/CVPR.2016.90.
  * He et al. [2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Heo et al. [2021] Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh.  Rethinking spatial dimensions of vision transformers.  _arXiv preprint arXiv:2103.16302_ , 2021.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Huang et al. [2016] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger.  Deep networks with stochastic depth.  In _ECCV (4)_ , volume 9908 of _Lecture Notes in Computer Science_ , pp. 646–661. Springer, 2016.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International Conference on Machine Learning_ , pp. 4904–4916, 2021.
  * Kitaev et al. [2020] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya.  Reformer: The efficient transformer.  _arXiv preprint arXiv:2001.04451_ , 2020.
  * Kolesnikov et al. [2020] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby.  Big transfer (bit): General visual representation learning.  In _European Conference on Computer Vision_ , 2020.
  * Kuo et al. [2023] Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova.  Open-vocabulary object detection upon frozen vision and language models.  In _International Conference on Learning Representations_ , 2023.
  * Li et al. [2021] Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Li et al. [2022a] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _ICML_ , pp. 12888–12900, 2022a.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023.
  * Li et al. [2022b] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao.  Grounded language-image pre-training.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022b.
  * Liang et al. [2022] Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie.  Not all patches are what you need: Expediting vision transformers via token reorganizations.  In _International Conference on Learning Representations_ , 2022.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  In _ECCV (5)_ , volume 8693 of _Lecture Notes in Computer Science_ , pp. 740–755. Springer, 2014.
  * Liu et al. [2023] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Liu et al. [2022] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Lu et al. [2022] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan.  Learn to explain: Multimodal reasoning via thought chains for science question answering.  In _The 36th Conference on Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Minderer et al. [2022] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al.  Simple open-vocabulary object detection.  In _European Conference on Computer Vision_ , 2022.
  * Mirzadeh et al. [2020] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh.  Improved knowledge distillation via teacher assistant.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pp. 5191–5198, 2020.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report, 2023.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.  Dinov2: Learning robust visual features without supervision.  _CoRR_ , abs/2304.07193, 2023.
  * Pei et al. [2023] Renjing Pei, Jianzhuang Liu, Weimian Li, Bin Shao, Songcen Xu, Peng Dai, Juwei Lu, and Youliang Yan.  Clipping: Distilling clip-based models with a student base for video-language retrieval.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Radford et al. [2018] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , pp. 9, 2019.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Raghu et al. [2021] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy.  Do vision transformers see like convolutional neural networks?  In _NeurIPS_ , pp. 12116–12128, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Rao et al. [2021] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh.  Dynamicvit: Efficient vision transformers with dynamic token sparsification.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 2022.
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  LAION-400M: open dataset of clip-filtered 400 million image-text pairs.  _CoRR_ , abs/2111.02114, 2021.
  * Shi et al. [2023] Dachuan Shi, Chaofan Tao, Ying Jin, Zhendong Yang, Chun Yuan, and Jiaqi Wang.  UPop: Unified and progressive pruning for compressing vision-language transformers.  In _International Conference on Machine Learning_ , 2023.
  * Simonyan & Zisserman [2015] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In _ICLR_ , 2015.
  * Singh et al. [2023] Mannat Singh, Quentin Duval, Kalyan Vasudev Alwala, Haoqi Fan, Vaibhav Aggarwal, Aaron Adcock, Armand Joulin, Piotr Dollár, Christoph Feichtenhofer, Ross Girshick, et al.  The effectiveness of mae pre-pretraining for billion-scale pretraining.  _arXiv preprint arXiv:2303.13496_ , 2023.
  * Steiner et al. [2022] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer.  How to train your vit? data, augmentation, and regularization in vision transformers.  _Trans. Mach. Learn. Res._ , 2022, 2022.
  * Sun et al. [2017] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta.  Revisiting unreasonable effectiveness of data in deep learning era.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.
  * Sun et al. [2023] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao.  EVA-CLIP: improved training techniques for CLIP at scale.  _CoRR_ , abs/2303.15389, 2023.
  * Tong et al. [2022] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.  Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  _Advances in Neural Information Processing Systems_ , 2022.
  * Touvron et al. [2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In _International conference on machine learning_ , pp. 10347–10357. PMLR, 2021.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Wang et al. [2020] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma.  Linformer: Self-attention with linear complexity.  _arXiv preprint arXiv:2006.04768_ , 2020.
  * Wang et al. [2021a] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  In _IEEE/CVF International Conference on Computer Vision_ , 2021a.
  * Wang et al. [2021b] Yulin Wang, Rui Huang, Shiji Song, Zeyi Huang, and Gao Huang.  Not all images are worth 16x16 words: Dynamic vision transformers with adaptive sequence length.  _arXiv preprint arXiv:2105.15075_ , 2021b.
  * Wei et al. [2022] Yixuan Wei, Yue Cao, Zheng Zhang, Zhuliang Yao, Zhenda Xie, Han Hu, and Baining Guo.  icar: Bridging image classification and image-text alignment for visual recognition.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Wu et al. [2022] Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan.  Tinyvit: Fast pretraining distillation for small vision transformers.  In _European Conference on Computer Vision_ , pp. 68–85. Springer, 2022.
  * Wu et al. [2023] Kan Wu, Houwen Peng, Zhenghong Zhou, Bin Xiao, Mengchen Liu, Lu Yuan, Hong Xuan, Michael Valenzuela, Xi (Stephen) Chen, Xinggang Wang, Hongyang Chao, and Han Hu.  Tinyclip: Clip distillation via affinity mimicking and weight inheritance.  In _IEEE/CVF International Conference on Computer Vision_ , 2023.
  * Xiao et al. [2018] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun.  Unified perceptual parsing for scene understanding.  In _ECCV (5)_ , volume 11209 of _Lecture Notes in Computer Science_ , pp. 432–448. Springer, 2018.
  * Yang et al. [2022a] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao.  Unified contrastive learning in image-text-label space.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022a.
  * Yang et al. [2022b] Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang.  Vision-language pre-training with triple contrastive learning.  2022b.
  * Yin et al. [2022] Hongxu Yin, Arash Vahdat, Jose M. Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov.  A-vit: Adaptive tokens for efficient vision transformer.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10799–10808, 2022.
  * Young et al. [2014] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  _Trans. Assoc. Comput. Linguistics_ , 2:67–78, 2014.
  * Zhai et al. [2022] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.  Scaling vision transformers.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 12104–12113, 2022.
  * Zhong et al. [2022] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al.  Regionclip: Region-based language-image pretraining.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Zhou et al. [2017] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ADE20K dataset.  In _CVPR_ , pp. 5122–5130. IEEE Computer Society, 2017.
  * Zhou et al. [2022] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra.  Detecting twenty-thousand classes using image-level supervision.  In _European Conference on Computer Vision_ , 2022.
  * Zhu et al. [2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_ , 2023.


# Experimental evidence of nonlinear focusing in standing water waves

Yuchen He1<EMAIL_ADDRESS>Alexey Slunyaev2,3 Nobuhito Mori4 Amin
Chabchoub5,4,1<EMAIL_ADDRESS>1 Centre for Wind, Waves and
Water, School of Civil Engineering, The University of Sydney, Sydney, NSW
2006, Australia  2 National Research University-Higher School of Economics, 25
Bol’shaya Pechorskaya Street, Nizhny Novgorod 603950, Russia  3Institute of
Applied Physics RAS, 46 Ulyanova Street, Nizhny Novgorod 603950, Russia  4
Disaster Prevention Research Institute, Kyoto University, Uji, Kyoto 611-0011,
Japan  5 Hakubi Center for Advanced Research, Kyoto University, Yoshida-
Honmachi, Kyoto 606-8501, Japan

###### Abstract

Nonlinear wave focusing originating from the universal modulation instability
(MI) is responsible for the formation of strong wave localizations on the
water surface and in nonlinear wave guides, such as optical Kerr media and
plasma. Such extreme wave dynamics can be described by breather solutions of
the nonlinear Schrödinger equation (NLSE) like by way of example the famed
doubly-localized Peregrine breathers (PB), which typify particular cases of
MI. On the other hand, it has been suggested that the MI relevance weakens
when the wave field becomes broadband or directional. Here, we provide
experimental evidence of nonlinear and distinct PB-type focusing in standing
water waves describing the scenario of two counter-propagating wave trains.
The collected collinear wave measurements are in excellent agreement with the
hydrodynamic coupled NLSE (CNLSE) and suggest that MI can undisturbedly
prevail during the interplay of several wave systems and emphasize the
potential role of exact NLSE solutions in extreme wave formation beyond the
formal narrowband and uni-directional limits. Our work may inspire further
experimental investigations in various nonlinear wave guides governed by CNLSE
frameworks as well as theoretical progress to predict strong wave coherence in
directional fields.

††preprint: APS/123-QED

The emergence of strongly localized waves in nonlinear dispersive media is an
actively investigated field of research across wave physics [1, 2, 3, 4, 5].
While excluding any external effects on a wave system, extreme wave formations
in uni-directional wave guides can be explained either as a result of unstable
nonlinear wave interaction [6, 7], or linear superposition principle [8, 9].
Both mechanisms have been intensively studied in laboratory environments and
the ocean [10, 11]. The modulation instability (MI) is a nonlinear wave
focusing mechanism [12, 13], which has been proven to be present in complex
sea states, such as crossing seas [14, 15, 16], however being less dominant
compared to wave systems with a single wave vector due to the violation of
both critical assumptions, namely uni-directionality and narrowband energy
level [17, 18].

Then again, it has been recently conjectured that there is an increase of
probability of rogue wave formation in coupled two-wave systems compared to an
uncoupled directional wave field [19]. A coupled nonlinear Schrödinger (CNLSE)
framework has been already derived for water waves in the 80s [20] and is
nowadays considered as a fundamental dynamical system for the study of
complex, coherent, directional, and rogue wave dynamics in various physical
media [21, 22, 23, 24, 25].

In this letter, we provide experimental evidence of nonlinear wave focusing in
standing wave fields using the Peregrine breather (PB) [26] as referenced
nonlinear rogue wave and special MI model evolving in one of two counter-
propagating wave systems. It is shown that Peregrine-type unsteady packets on
finite and zero background can distinctly evolve in the presence of a counter-
propagative regular wave field without any noticeable disturbance nor
disintegration of coherence during wave focusing and defocusing. The
experimental results are in excellent agreement with the hydrodynamic CNLSE.
Moreover, we can confirm nonlinear focusing in such simplified but
representative configuration by means of direct numerical simulations of the
governing water wave equations using the high-order spectral method (HOSM) for
potential flows.

The nonlinear interaction of standing waves involving an incident wave field
characterized by the complex amplitude $\psi^{(1)}(x,t)$ with wavenumber
$\kappa^{(1)}=(k,0)$ interacting with the reflected waves $\psi^{(2)}(x,t)$
with wavenumber $\kappa^{(2)}=(-k,0)$, and propagating along the space co-
ordinate $x$, can be described by the CNLSE [20, 21]

$\begin{split}&i(\psi^{(1)}_{x}+\frac{1}{c_{g}}\psi^{(1)}_{t})+\delta\psi^{(1)}_{tt}+\nu(|\psi^{(1)}|^{2}-2|\psi^{(2)}|^{2})\psi^{(1)}=0.\\\
&i(\psi^{(2)}_{x}-\frac{1}{c_{g}}\psi^{(2)}_{t})+\delta\psi^{(2)}_{tt}+\nu(|\psi^{(2)}|^{2}-2|\psi^{(1)}|^{2})\psi^{(2)}=0,\end{split}$
(1)

The angular frequency $\omega^{(i)}$ and wavenumber $\kappa^{(i)}$ are
connected through the deep-water linear dispersion relation
$\omega^{(i)}=\sqrt{g\lvert\kappa^{(i)}\rvert}$, $i=1,2$ and $g$ being the
gravitational acceleration, while the CNLSE parameters read

$c_{g}=\frac{\omega}{2k},\ \delta=-\frac{k}{\omega^{2}},\ \nu=-k^{3}.$ (2)

The standing water surface elevation describing the interaction between the
incident wave field $\psi^{(1)}$ and opposing wave system $\psi^{(2)}$ to
first-order of approximation is calculated by means of the complex envelope
following the expression

$\begin{split}\eta(x,t)=\frac{1}{2}(&\psi^{(1)}(x,t)\exp[i(kx-\omega t)]\\\
&+\psi^{(2)}(x,t)\exp[-i(kx+\omega t)]+c.c.).\end{split}$ (3)

Here, $c.c.$ denotes the complex conjugation and we emphasize the weakening of
the bound waves contributions in standing wave systems in contrast to the uni-
directional wave evolution. Our wave modeling will be restricted to the CNLSE,
described by the set of equations in (1), nonetheless, we would like to also
highlight that a higher- and fourth-order CNLSE can be applied to two wave
systems with different wave frequencies or directions [27].

We investigate the possibility of nonlinear wave focusing in a standing wave
field, which is a special case of crossing wave systems with a crossing angle
of $\pi$, by using the PB model as incident wave field $\psi^{(1)}$ and a
regular wave envelope of same amplitude for $\psi^{(2)}$. We recall that the
PB is a doubly-localized solution of the uni-directional nonlinear Schrödinger
equation (NLSE) [26], which has been so far observed in optics, water waves
and plasma [28, 29, 30]. That said, this solution describes the nonlinear
stage of modulation instability in the case of infinitely long perturbation
[31, 32] and has a rational growth. In fact, it can be approached by either
Akhmediev- [33, 34] or Kuznetsov breathers [35], and so far not discussed in
the context of collinear waves. Its counterpart on zero background is also
known as the degenerate soliton [36].

Since most of the state of the art wave flumes do not have two wave generators
on opposing sides, one possibility to allow for wave train collisions is to
make use of the wall at the opposite side of the wave generator as a
reflective mirror for long-crested waves [37]. Indeed, the absence of wave
energy absorbing beach in a perfectly aligned flume permits a full reflection
and thus, the formation of standing waves. Such adjusted and adopted set-up
configuration can be viewed in Fig. 1.

Figure 1: Sketch of the wave flume setup as installed at the University of
Sydney after removing the artificial beach installation to enable full wave
reflection. The piston-type wave generator on the right end generates long-
crested uni-directional waves. The glass wall and wave mirror is at the
opposing end.

The facility operates a piston-type wave paddle to generate boundary
conditions in a form of surface elevation time-series. The tank has the
dimensions $30\times 1\times 1$ m3, the water depth is set to be $0.75$ m, and
eight resistive wave gauges with a sampling rate of 32 Hz are used to measure
the water surface elevation. The accurate controlled generation of the wave
field allows the reiteration of the experiments with different wave gauge
placements to ensure a high resolution also along the waves’ propagation
direction [4].

Since we are interested in the interaction of an incident PB wave field with a
regular wave train with opposing wave vector, a dynamic head-on interaction
which we will refer to as standing PB, we simply select a sufficiently long
realization of the PB solution in time to guarantee the propagation of the
unperturbed regular wave to the wall and back, before the wave maker launches
the small and localized Peregrine-specific modulation. We refer to [29, 38]
for the construction of the dimensional Peregrine boundary conditions. In our
experiments we chose the amplitude to be $a=0.01$ m for two carrier steepness
values $ak=0.09$ and $ak=0.10$. The respective values of the wave frequency
can be computed using the linear dispersion relation $\omega=\sqrt{gk}$. The
boundary conditions have been defined to expect the maximal wave compression
16 m from the wave generator. The results of the experimental campaign,
accounting for 184 measurements along the longitudinal direction for each
realization, together with the associated CNLSE predictions are shown in Fig.
2.

Figure 2: Top (a) and (b): Experimental observation of two standing PBs.
Bottom (c) and (d): Numerical CNLSE simulations starting from the same
boundary conditions as the laboratory experiments. Left (a) and (c): Wave
parameters $a=0.01$ m and $ak=0.09$. Right (b) and (d): Wave parameters
$a=0.01$ m and $ak=0.10$.

The numerical CNLSE simulations have been carried out for the same wave
parameters and boundary conditions as the laboratory experiments for
validations purposes. A pseudo-spectral approach and the fourth-order
Runge–Kutta method have been adopted for the integration in space.

The wave tank measurements show an excellent agreement with the CNLSE
dynamics: the PB remarkably evolves in standing wave states without any signs
of disintegration while keeping its coherence. Moreover, the characteristic
amplitude amplification of four in this collinear case is also reached at the
expected location in the water wave tank. This wave interaction can be also
confirmed when studying the respective bi-spectral evolution, see Fig. 3.

Figure 3: Spectral evolution of the standing Peregrine breathers, reported in
Fig. 2. (a) Computed for the measured surface elevation for the carrier
parameters $a=0.01$ m and $ak=0.09$. (c) Incident PB wave dynamics isolated
from (a). (e) Opposing wave train as isolated from (a). (b) Computed for the
measured surface elevation for the carrier wave parameters $a=0.01$ m and
$ak=0.10$. (d) Incident PB wave dynamics isolated from (b). (f) Opposing wave
train as isolated from (b).

Here, the decomposition of incident and reflected wave constituents in the bi-
spectra, which are estimated from the Fourier components [39], prove that the
extreme wave focusing only occurs for incident waves, dominated by the PB
dynamics. To be more precise, the spectral broadening, which is an indicator
of physical wave focusing, occurs only in the incident wave field, as in Fig.
3 (c) and (d). On the other hand, the energy of the counter-propagating waves
remains steady, see Fig. 3 (e) and (f). We can also notice that the second
harmonic energy weakens when the standing wave field has been fully developed.

Note that the variations in surface wave profiles are very sensitive to the
single gauge location. On a node point, the amplitude of the standing regular
wave field is zero [40]. It is also worth mentioning that the PB did not have
any influence in the destabilization of the regular opposing wave field. This
persistence may be partly explained by the form of the CNLSE (1), which
supports conservation of integrals $\int_{-\infty}^{\infty}{|\psi_{j}|^{2}dx}$
for $j=1,2$, so that the nonlinear exchange terms in Eqs. (1) lead to
frequency corrections only. The impossibility of energy exchange between two
counter propagating planar deep-water wave systems due to nonlinear four-wave
interactions beyond the narrow-band approximation was recently emphasized [41,
42]. Nonetheless, the generation of collinear opposite waves was observed in
fully nonlinear hydrodynamic simulations in [43].

In the following we investigate the evolution of the PB counterpart on zero
background [44, 36], i.e. the degenerate soliton solution, in the presence of
a counter-propagative regular wave field. Because this particular NLSE
solution has finite length construction on a zero background, the standing
wave patterns appear only locally during the interaction with the opposing
wave field. Experimental data and respective CNLSE simulations are displayed
in Fig. 4.

Figure 4: Top (a) and (b): Evolution of two standing degenerate solitons.
Bottom (c) and (d): Corresponding CNLSE simulations. Left (a) and (c): Carrier
wave parameters $a=0.01$ m and $ak=0.11$. Right (b) and (d): Carrier wave
parameters $a=0.01$ m and $ak=0.12$.

Once again, an excellent agreement can be noticed between laboratory
experiments and numerical simulations, suggesting an elastic collision between
the regular envelope and degenerate soliton [45] and by that confirming once
again that the CNLSE framework may have an extended applicability range beyond
the physical limitations technically limiting its exploitation.

As next, we extend our proof of concept validation study in considering fully
nonlinear water wave framework by numerically solving the Euler equations
using the HOSM [46, 47], which resolves up to $7$-wave interactions in our
simulations. This approach does not only allow consideration of the wave
evolution for longer time and distance than at disposal in experimental wave
facilities, but also provides a higher reliability compared to weakly
nonlinear CNLSE approach. Fig. 5 shows a corresponding and particular case
study, which has been experimentally studied and reported in Fig. 2 (a) and
(c), i.e. the case of the Peregrine breather on finite background. The initial
condition at $t=0$ is specified in the form of two long trains with the
carrier wave steepness $ak=0.09$, and which travel towards each other. The
rightward moving train is produced from the analytic PB solution; the
inoculating perturbation is characterized by the steepness less than $0.11$.

Figure 5: Evolution of a PB in standing waves with the carrier steepness
$ak=0.09$ as simulated by the HOSM by arranging two dozen phases allowing for
the reconstruction of wave envelope. (a) The surface displacement envelope
$k\eta$ is shown by the pseudo-color. The surface displacement profile of the
highest focused wave is displayed in black while the dashed black lines point
the location of maximal envelope compression, both according to the analytic
PB solution. The magenta parallelogram shows the area where the maximum wave
amplitude is evaluated. The coordinate and time are normalized with the
dominant wave length $\lambda$ and period $T$, respectively. (b) Evolution of
the maximum wave amplitude in the HOSM simulations and according to the
analytic PB solution of the NLSE. The filled areas correspond to the intervals
between crest and trough amplitudes, while the solid curves depict their
simple means.

The pseudo-color in Fig. 5 (a) represents the evolution of the surface
displacement envelope, which is produced from two dozen simulations when
various phase combinations were assigned to the initial wave trains. Hence,
the envelope corresponds to the phase averaging of the upper enveloping
surface. Only a part of the simulated domain is shown. The target focusing
time according to the exact NLSE PB solution is about $90$ wave periods, see
the dashed black lines.

The snapshot of the maximum wave, which is very close to the breaking onset,
is given by the black curve. Note that the deviations of the maximal envelope
compression location are less significant in the wave flume because of the
shorter propagating distance considered, a constraint imposed by the wave
generator frequency range and the limited length of state of the art wave
facilities.

The strongly nonlinear simulations reproduce all the main features of the
localized standing wave patterns as observed in the laboratory environment,
particularly the clean and quasi-undisturbed extreme wave focusing within the
incident PB group. At the same time, the long evolution reveals some
distinctions. Differently than the weakly nonlinear framework, the strongly
nonlinear simulation results correspond to slightly faster movement of the
growing modulation and noticeably longer focusing time. Besides, a minor
asymmetry of the emerged large wave group is clearly seen before and after the
maximal wave focusing.

Evolution of the corresponding maximum wave amplitude is displayed in Fig. 5
(b) by the red color. It is evaluated within the area bounded by the magenta
contour in Fig. 5 (a), which has been introduced to reduce the effect of
emerging large modulations at the edges of the colliding wave trains. This
dependence is compared with the result of a similar strongly nonlinear
numerical simulations of the PB when the opposite wave train is absent (blue
color) and with the exact analytic PB solution (black curve). The differences
between the red and blue curves correspond well to the amplitude of the
opposite wave train at $ak=0.09$, confirming once again the non-influence of
the oppositely propagating regular wave train on the focusing dynamics of the
modulationally unstable wave packet. The strongly nonlinear localized focusing
of Peregrine breathers on top of progressive or standing waves occurs later
than predicted by CNLSE and results in significantly larger focused waves. A
behaviour which has been also quantified in the uni-directional NLSE case [48,
49].

Conclusively, we have reported experimental evidence of quasi-unperturbed PB
hydrodynamics in standing waves. The same has been observed for the degenerate
soliton, which is the counterpart of PB on zero background. Our results
confirm that breather solutions of the NLSE can be considered to model and
describe extreme wave localizations in non-integrable systems [50, 51]. Since
experiments are always subject to dissipative effects, we believe that the
role of NLSE solitons and breathers can be extended to a wider range of
complex systems [52, 53, 54]. We also anticipate that our experimental work
will motivate further studies to investigate the role of breathers for
standing wave conditions in nonlinear dispersive media as well as to determine
their function in crossing seas and vector nonlinear fiber-analog systems.

A.S. acknowledges the support from the RFBR grant No. 21-55-15008 and from
Lab. of Dynamical Systems and Applications NRU HSE (the Ministry of Science
and Higher Education of the Russian Federation Grant No. 075-15-2019-1931).
A.C. acknowledges support from Kyoto University’s Hakubi Center for Advanced
Research.

## References

  * Onorato  _et al._ [2013] M. Onorato, S. Residori, U. Bortolozzo, A. Montina, and F. Arecchi, Rogue waves and their generating mechanisms in different physical contexts, Physics Reports 528, 47 (2013).
  * Dudley  _et al._ [2014] J. M. Dudley, F. Dias, M. Erkintalo, and G. Genty, Instabilities, breathers and rogue waves in optics, Nature Photonics 8, 755 (2014).
  * Suret  _et al._ [2020] P. Suret, A. Tikan, F. Bonnefoy, F. Copie, G. Ducrozet, A. Gelash, G. Prabhudesai, G. Michel, A. Cazaubiel, E. Falcon, _et al._ , Nonlinear spectral synthesis of soliton gas in deep-water surface gravity waves, Physical Review Letters 125, 264101 (2020).
  * Vanderhaegen  _et al._ [2021] G. Vanderhaegen, C. Naveau, P. Szriftgiser, A. Kudlinski, M. Conforti, A. Mussot, M. Onorato, S. Trillo, A. Chabchoub, and N. Akhmediev, ”Extraordinary” modulation instability in optics and hydrodynamics, Proceedings of the National Academy of Sciences 118 (2021).
  * Weisman  _et al._ [2021] D. Weisman, M. Carmesin, G. G. Rozenman, M. A. Efremov, L. Shemer, W. P. Schleich, and A. Arie, Diffractive guiding of waves by a periodic array of slits, Proceedings of the National Academy of Sciences 127 (2021).
  * Zakharov [1968] V. E. Zakharov, Stability of periodic waves of finite amplitude on the surface of a deep fluid, Journal of Applied Mechanics and Technical Physics 9, 190 (1968).
  * Bonnefoy  _et al._ [2016] F. Bonnefoy, F. Haudin, G. Michel, B. Semin, T. Humbert, S. Aumaître, M. Berhanu, and E. Falcon, Observation of resonant interactions among surface gravity waves, Journal of Fluid Mechanics 805 (2016).
  * Longuet-Higgins [1974] M. Longuet-Higgins, Breaking waves in deep or shallow water, in  _Proc. 10th Conf. on Naval Hydrodynamics_ , Vol. 597 (MIT, 1974) p. 605.
  * McAllister  _et al._ [2019] M. L. McAllister, S. Draycott, T. Adcock, P. Taylor, and T. Van Den Bremer, Laboratory recreation of the draupner wave and the role of breaking in crossing seas, Journal of Fluid Mechanics 860, 767 (2019).
  * Dudley  _et al._ [2019] J. M. Dudley, G. Genty, A. Mussot, A. Chabchoub, and F. Dias, Rogue waves and analogies in optics and oceanography, Nature Reviews Physics 1, 675 (2019).
  * Waseda [2020] T. Waseda, Nonlinear processes, in  _Ocean Wave Dynamics_ (2020) pp. 103–161.
  * Bespalov and Talanov [1966] V. Bespalov and V. Talanov, Filamentary structure of light beams in nonlinear liquids, Soviet Journal of Experimental and Theoretical Physics Letters 3, 307 (1966).
  * Benjamin and Feir [1967] T. B. Benjamin and J. E. Feir, The disintegration of wave trains on deep water part 1. theory, Journal of Fluid Mechanics 27, 417 (1967).
  * Waseda  _et al._ [2009] T. Waseda, T. Kinoshita, and H. Tamura, Evolution of a random directional wave and freak wave occurrence, Journal of Physical Oceanography 39, 621 (2009).
  * Onorato  _et al._ [2010] M. Onorato, D. Proment, and A. Toffoli, Freak waves in crossing seas, The European Physical Journal Special Topics 185, 45 (2010).
  * Gramstad  _et al._ [2018] O. Gramstad, E. Bitner-Gregersen, K. Trulsen, and J. C. Nieto Borge, Modulational instability and rogue waves in crossing sea states, Journal of Physical Oceanography 48, 1317 (2018).
  * Janssen [2003] P. A. Janssen, Nonlinear four-wave interactions and freak waves, Journal of Physical Oceanography 33, 863 (2003).
  * Mori  _et al._ [2011] N. Mori, M. Onorato, and P. A. Janssen, On the estimation of the kurtosis in directional sea states for freak wave forecasting, Journal of Physical Oceanography 41, 1484 (2011).
  * Grönlund  _et al._ [2009] A. Grönlund, B. Eliasson, and M. Marklund, Evolution of rogue waves in interacting wave systems, EPL (Europhysics Letters) 86, 24001 (2009).
  * Okamura [1984] M. Okamura, Instabilities of weakly nonlinear standing gravity waves, Journal of the Physical Society of Japan 53, 3788 (1984).
  * Onorato  _et al._ [2006] M. Onorato, A. R. Osborne, and M. Serio, Modulational instability in crossing sea states: A possible mechanism for the formation of freak waves, Physical Review Letters 96, 014503 (2006).
  * Baronio  _et al._ [2012] F. Baronio, A. Degasperis, M. Conforti, and S. Wabnitz, Solutions of the vector nonlinear Schrödinger equations: evidence for deterministic rogue waves, Physical Review Letters 109, 044102 (2012).
  * Frisquet  _et al._ [2016] B. Frisquet, B. Kibler, P. Morin, F. Baronio, M. Conforti, G. Millot, and S. Wabnitz, Optical dark rogue wave, Scientific reports 6, 1 (2016).
  * Støle-Hentschel  _et al._ [2018] S. Støle-Hentschel, K. Trulsen, L. B. Rye, and A. Raustøl, Extreme wave statistics of counter-propagating, irregular, long-crested sea states, Physics of Fluids 30, 067102 (2018).
  * Steer  _et al._ [2019] J. N. Steer, M. L. McAllister, A. G. Borthwick, and T. S. Van Den Bremer, Experimental observation of modulational instability in crossing surface gravity wavetrains, Fluids 4, 105 (2019).
  * Peregrine [1983] D. H. Peregrine, Water waves, nonlinear Schrödinger equations and their solutions, J. Austral. Math. Soc. Ser. B 25, 16 (1983).
  * Gramstad and Trulsen [2011] O. Gramstad and K. Trulsen, Fourth-order coupled nonlinear Schrödinger equations for gravity waves on deep water, Physics of Fluids 23, 062102 (2011).
  * Kibler  _et al._ [2010] B. Kibler, J. Fatome, C. Finot, G. Millot, F. Dias, G. Genty, N. Akhmediev, and J. M. Dudley, The peregrine soliton in nonlinear fibre optics, Nature Physics 6, 790 (2010).
  * Chabchoub  _et al._ [2011] A. Chabchoub, N. Hoffmann, and N. Akhmediev, Rogue wave observation in a water wave tank, Physical Review Letters 106, 204502 (2011).
  * Bailung  _et al._ [2011] H. Bailung, S. Sharma, and Y. Nakamura, Observation of peregrine solitons in a multicomponent plasma with negative ions, Physical Review Letters 107, 255005 (2011).
  * Akhmediev and Ankiewicz [1997] N. N. Akhmediev and A. Ankiewicz,  _Solitons: nonlinear pulses and beams_ (Chapman & Hall, 1997).
  * Osborne [2010] A. Osborne,  _Nonlinear Ocean Waves and the Inverse Scattering Transform_ (Academic Press, 2010).
  * Akhmediev  _et al._ [1985] N. Akhmediev, V. Eleonskii, and N. Kulagin, Generation of periodic trains of picosecond pulses in an optical fiber: exact solutions, Sov. Phys. JETP 62, 894 (1985).
  * Cavaleri  _et al._ [2012] L. Cavaleri, L. Bertotti, L. Torrisi, E. Bitner-Gregersen, M. Serio, and M. Onorato, Rogue waves in crossing seas: The louis majesty accident, Journal of Geophysical Research: Oceans 117 (2012).
  * Kuznetsov [1977] E. A. Kuznetsov, Solitons in a parametrically unstable plasma, in  _Akademiia Nauk SSSR Doklady_ , Vol. 236 (1977) pp. 575–577.
  * Chabchoub  _et al._ [2021] A. Chabchoub, A. Slunyaev, N. Hoffmann, F. Dias, B. Kibler, G. Genty, J. M. Dudley, and N. Akhmediev, The peregrine breather on the zero-background limit as the two-soliton degenerate solution: An experimental study, Frontiers in Physics 9, 633549 (2021).
  * Tanter  _et al._ [1998] M. Tanter, J.-L. Thomas, and M. Fink, Influence of boundary conditions on time-reversal focusing through heterogeneous media, Applied Physics Letters 72, 2511 (1998).
  * Chabchoub and Grimshaw [2016] A. Chabchoub and R. H. Grimshaw, The hydrodynamic nonlinear Schrödinger equation: Space and time, Fluids 1, 23 (2016).
  * Goda and Suzuki [1977] Y. Goda and Y. Suzuki, Estimation of incident and reflected waves in random wave experiments, in  _Coastal Engineering 1976_ (1977) pp. 828–845.
  * Aguilera-Rojas  _et al._ [2021] P. Aguilera-Rojas, M. Clerc, G. Gonzalez-Cortes, and G. Jara-Schulz, Localized standing waves induced by spatiotemporal forcing, Physical Review E 104, 044209 (2021).
  * Dyachenko  _et al._ [2017] A. Dyachenko, D. Kachulin, and V. Zakharov, Canonical system of equations for 1d water waves, Journal of Fluid Mechanics 828, 661 (2017).
  * Dyachenko [2020a] A. Dyachenko, Canonical system of equations for 1d water waves, Studies in Applied Mathematics 144, 493 (2020a).
  * Slunyaev [2018] A. Slunyaev, Group-wave resonances in nonlinear dispersive media: The case of gravity water waves, Physical Review E 97, 010202 (2018).
  * Kedziora  _et al._ [2014] D. Kedziora, A. Ankiewicz, and N. Akhmediev, Rogue waves and solitons on a cnoidal background, The European Physical Journal Special Topics 223, 43 (2014).
  * Dyachenko [2020b] A. Dyachenko, Canonical system of equations for 1d water waves, Studies in Applied Mathematics 144, 493 (2020b).
  * West  _et al._ [1987] B. West, K. Brueckner, R. Janda, D. Milder, and R. Milton, A new numerical method for surface hydrodynamics, Journal of Geophysical Research 92, 11803 (1987).
  * Dommermuth and Yue [1987] D. G. Dommermuth and D. K. Yue, A high-order spectral method for the study of nonlinear gravity waves, Journal of Fluid Mechanics 184, 267 (1987).
  * Slunyaev  _et al._ [2013] A. Slunyaev, E. Pelinovsky, A. Sergeeva, A. Chabchoub, N. Hoffmann, M. Onorato, and N. Akhmediev, Super rogue waves in simulations based on weakly nonlinear and fully nonlinear hydrodynamic equations, Physical Review E 88, 012909 (2013).
  * Shemer and Alperovich [2013] L. Shemer and L. Alperovich, Peregrine breather revisited, Physics of Fluids 25, 051701 (2013).
  * Kivshar and Malomed [1989] Y. S. Kivshar and B. A. Malomed, Dynamics of solitons in nearly integrable systems, Reviews of Modern Physics 61, 763 (1989).
  * Chekhovskoy  _et al._ [2019] I. Chekhovskoy, O. V. Shtyrina, M. P. Fedoruk, S. B. Medvedev, and S. K. Turitsyn, Nonlinear Fourier transform for analysis of coherent structures in dissipative systems, Physical Review Letters 122, 153901 (2019).
  * Haelterman  _et al._ [1992] M. Haelterman, S. Trillo, and S. Wabnitz, Dissipative modulation instability in a nonlinear dispersive ring cavity, Optics communications 91, 401 (1992).
  * Akhmediev and Ankiewicz [2008] N. Akhmediev and A. Ankiewicz,  _Dissipative solitons: from optics to biology and medicine_ , Vol. 751 (Springer Science & Business Media, 2008).
  * Nielsen  _et al._ [2021] A. U. Nielsen, Y. Xu, C. Todd, M. Ferré, M. G. Clerc, S. Coen, S. G. Murdoch, and M. Erkintalo, Nonlinear localization of dissipative modulation instability, Physical Review Letters 127, 123901 (2021).