# Data Filtering Networks

Alex Fang1 Albin Madappally Jose 1 Amit Jain1
Ludwig Schmidt2 Alexander Toshev1 Vaishaal Shankar1
1Apple, 2University of Washington

###### Abstract

Large training sets have become a cornerstone of machine learning and are the
foundation for recent advances in language modeling and multimodal learning.
While data curation for pre-training is often still ad-hoc, one common
paradigm is to first collect a massive pool of data from the Web and then
filter this candidate pool down to an actual training set via various
heuristics. In this work, we study the problem of learning a _data filtering
network_ (DFN) for this second step of filtering a large uncurated dataset.
Our key finding is that the quality of a network for filtering is distinct
from its performance on downstream tasks: for instance, a model that performs
well on ImageNet can yield worse training sets than a model with low ImageNet
accuracy that is trained on a small amount of high-quality data. Based on our
insights, we construct new data filtering networks that induce state-of-the-
art image-text datasets. Specifically, our best performing dataset DFN-5B
enables us to train state-of-the-art models for their compute budgets: among
other improvements on a variety of tasks, a ViT-H trained on our dataset
achieves 83.0% zero-shot transfer accuracy on ImageNet, out-performing models
trained on other datasets such as LAION-2B, DataComp-1B, or OpenAI’s WIT. In
order to facilitate further research in dataset design, we also release a new
2 billion example dataset DFN-2B and show that high performance data filtering
networks can be trained from scratch using only publicly available data.

##  1 Introduction

Carefully curated datasets have driven progress in machine learning for
decades, from early pattern recognition experiments in Bell Labs to recent
developments like GPT-4, Stable Diffusion, and CLIP (Highleyman & Kamentsky,
1959; LeCun et al., 1989, 1998; Deng et al., 2009; Krizhevsky et al., 2009,
2012; Radford et al., 2019, 2021, 2022; OpenAI, 2023). Despite their crucial
role, datasets themselves are rarely the subject of active research
(Sambasivan et al., 2021).

Current approaches to improving performance on machine learning tasks have
focused on scaling model capacity or training data volume. While scaling laws
(Hestness et al., 2017; Kaplan et al., 2020; Aghajanyan et al., 2023; Cherti
et al., 2023) have elucidated the relationship between model size, data size,
and performance, little formal guidance exists on how to scale these
quantities. On the model side, experimentation is straightforward - with
enough compute, permutations of width, depth, normalization and training
hyperparameters can be rigorously evaluated, leading to consistent modeling
improvements over the years (Touvron et al., 2023a, b; Elsen et al., 2023).

The dataset side is unfortunately murkier. Most large-scale training sets are
not released, leaving the community to attempt open reproductions (Schuhmann
et al., 2021, 2022; Byeon et al., 2022; Gao et al., 2020); however, these are
often one-off efforts without the iterative refinement that models enjoy.
Recent efforts like DataPerf, DataComp and MetaCLIP (Mazumder et al., 2022;
Gadre et al., 2023; Xu et al., 2023) help bridge the gap by providing
consistent dataset evaluation and reproduction frameworks.

We argue dataset design can leverage the same tools as model design. Almost
all large-scale dataset construction can be broken down into two phases:
uncurated data collection and dataset filtering. We focus our work on the
latter, with the assumption that a large uncurated dataset exists. We show
data filtering networks (DFNs) - neural networks designed to filter data - can
induce massive, high-quality pre-training datasets. Unlike previous techniques
relying on domain-specific heuristics, DFNs paired with a large unfiltered
image-text pool produce billion-scale state-of-the-art datasets
algorithmically. We demonstrate DFNs can be efficiently trained from scratch
and improved with the same techniques as standard ML models.

Figure 1: Compute scaling behavior of training CLIP models on various
datasets. DFN-2B, the subset of CommonPool (DataComp-12.8B) chosen by our best
performing data filtering networks, out-performs all other datasets including
OpenAI’s WIT and the previous state-of-the-art CLIP training dataset
DataComp-1B. Our ViT-L outperforms a ViT-G trained on LAION with 18$\times$
more compute. Similarly, our ViT-B/16 trained on our dataset outperforms
OpenAI’s ViT-L/14 trained with 4$\times$ more compute. Our ViT-H/14 achieves
83.0% on ImageNet, out-performing any model in its compute class. All DFN-
trained models were trained on DFN-2B, except for the ViT-H which was trained
on DFN-5B. Both datasets were induced by the same DFN.

The contributions of this work are as follows. First, we characterize the
properties of data filtering networks that lead to high-quality datasets. We
ablate properties of data filtering networks from supervision signal to
training data quality. We find that a small contrastive image-text model
trained on _only_ high-quality data is sufficient to construct state-of-the-
art datasets.

Second, we use these properties to train DFNs and construct datasets that
induce Contrastive Image-Text Pre-trained (CLIP) models that achieve high
accuracy and present better compute accuracy tradeoff than any existing
dataset in the literature as show in Figure 1. In particular we train a
ViT-L/14 or 12.8B examples seen on our DFN induced dataset DFN-2B to 81.4
ImageNet zero-shot transfer accuracy, outperforming the previous best ViT-L
trained on DataComp-1B by over 2 percentage points. We further train a
ViT-H/14 for 39B samples seen on a larger DFN induced dataset DFN-5B to 83.0
ImageNet zero-shot transfer accuracy. We show that models trained on these
datasets show consistent improvements on many tasks, including zero-shot
classification, retrieval, and visual question answering, and maintain the
favorable robustness properties of CLIP models.

Lastly, the above insights can be used as a recipe to construct high-quality
datasets from scratch by using only public data 111Most large public Image-
Text datasets including LAION-5B and DataComp-1B are built using OpenAI’s CLIP
model thus making strides towards democratization of large high-quality
datasets. In addition, we release DFN-2B for the community to enable research
on large image-text models.

##  2 Background and Related Work

###  2.1 Contrastive Image Language Pre-training (CLIP)

CLIP has altered the use of cheaply available image-alt-text datasets by
demonstrating the practicality of large-scale training on web-scraped image-
text pairs to build state-of-the-art image representations. CLIP consists of
separate vision and text encoders, and uses contrastive loss during training
to push the representations of related images and text pairs together, and
unrelated pairs apart. Crucial to this process is a large dataset of _aligned_
image-text pairs - images paired with semantically relevant text. The release
of CLIP was followed by several other image-text models such as ALIGN, BASIC,
LiT and Open-CLIP all of which we will refer to in this work as CLIP models
(Jia et al., 2021; Pham et al., 2023; Zhai et al., 2022b; Ilharco et al.,
2021). CLIP models generally come in 3 canonical sizes of vision transformer:
ViT-B/32, ViT-B/16 and ViT-L/14; since then, the open source community has
extended these to 3 larger variants ViT-H/14, ViT-g/14 and ViT-G/14
(Dosovitskiy et al., 2020; Zhai et al., 2022a). Generally the larger models
exhibit better zero-shot generalization and transfer properties. CLIP models
have been trained on a variety of datasets from OpenAI’s WiT, Google’s WebLI
and JFT-3B, LAION, COYO and DataComp-1B.

Prior work has also studied how to fine-tune CLIP models to improve
performance in targeted directions. CLIP models can be fine-tuned on image
classification tasks by using templates to transform labels to text (Fang et
al., 2022; Goyal et al., 2022). Additionally, practitioners often use weight
ensembling to preserve robustness properties of the pre-trained model while
reaping the benefits of fine-tuning (Wortsman et al., 2022). We take advantage
of these techniques in order to improve the filtering models we train in this
work.

###  2.2 Dataset Construction

Prior to CLIP, datasets most commonly used in computer vision were
_supervised_ with human labels (Deng et al., 2009; Krizhevsky et al., 2009).
Though these older dataset construction pipelines were quite intricate and did
not scale beyond a few million examples, they share some similarity with
current constructions. Classical datasets such as ImageNet and CIFAR started
with a large _roughly curated_ pool of images paired with metadata, and used
humans to either label or filter the data.

Modern dataset pipelines have a similar procedure but at a much higher scale.
The initial pool of images can contain up to 100 billion images, and the
dataset filtering is purely automated, often with a set of rules and heuristic
filtering stages (Jia et al., 2021). Past work in natural language processing
has used binary filters as an initial step to remove low quality documents
(Wenzek et al., 2019; Brown et al., 2020), but contain multiple components to
their filtering pipelines.

One of the first publicly available web-scale image-text datasets is LAION.
LAION-400M and LAION-2B were constructed by collecting image-text pairs from
Common Crawl, filtering by English, and then keeping pairs whose image and
text are well _aligned_. This alignment is performed using a procedure known
as _CLIP filtering_. The procedure leverages an existing image-text model (in
LAION’s case OpenAI CLIP ViT-B/32), and removes samples whose cosine
similarity between image and text are below some threshold. We show pseudocode
of the basic CLIP filtering operation below.

⬇

def clip_filter(image, text, threshold=0.3):

# compute image and text representations

image_features = clip.encode_image(image_input)

text_features = clip.encode_text(text_input)

# compute alignment

dot_product = image_features.T @ text_features

norm_a = image_features.norm()

norm_b = text_features.norm()

similarity = dot_product / (norm_a * norm_b)

# filter by alignment

return similarity > threshold

While CLIP filtering is convenient it is dependent on a existing trained CLIP
model, and perhaps limited on the top-line performance of any model trained
using it as a filter. For example, despite LAION-2B being five times larger
than OpenAI’s dataset, models trained on it could only match OpenAI’s ImageNet
zero-shot performance with a significantly larger compute budget.

To better facilitate the study of image-text datasets, researchers created the
DataComp benchmark (Gadre et al., 2023). The benchmark provides 12.8 billion
image-text pairs from Common Crawl so that researchers can study the effect of
various data filtering techniques. DataComp fixes the computational budget
used to train the resulting models, fixing the compute budget of the largest
scale to match the cost of training OpenAI’s ViT-L/14 CLIP model. These models
are then evaluated on a suite of 38 downstream tasks, which includes ImageNet
and distribution shifts, VTAB, and retrieval tasks. We use this benchmark as
our primary method of evaluating the datasets created by our data filtering
networks.

The authors of DataComp also released a baseline dataset, DataComp-1B (DC-1B)
that improved upon LAION-5B, by combining CLIP filtering with an ImageNet
based clustering approach to improve dataset quality on a variety of
benchmarks. However this dataset still relies on the OpenAI CLIP model for
CLIP filtering and imposes a costly ImageNet specific clustering step in the
pipeline.

Recent work (Xu et al., 2023) has demystified the CLIP dataset construction
process and demonstracted high quality dataset construction is possible by
simple keyword based sampling and global balancing. While their work does
create competitive datasets, the reliance on sampling heuristics from the
original CLIP paper (Radford et al., 2021) allows for accurate dataset
reproduction, our work focuses on _improving_ model performance using dataset
construction.

##  3 Data Filtering Networks

The core object we study in this work is a data filtering network (DFN). In
this section we define DFNs and introduce their evaluation setup.

###  3.1 Definitions

Since our ultimate goal is to build functions that filter potentially
trillions of examples efficiently, we restrict the scope of our study to DFNs
that are only applied pointwise to elements of a larger data pool. Thus,
processing a data pool with a DFN, defined in pseudocode as follows

⬇

def apply_dfn(dfn, data_pool):

return [x for x in data_pool if dfn(x)]

lends itself to parallelization and thus efficient application. For a given
DFN and pool, we refer to the data pool we train the DFN on as a filter
dataset. Furthermore, we refer to the dataset constructed by filtering the
pool with the DFN the _induced dataset_. We then refer to a model trained
(only) on that dataset the _induced model_.

As introduced in Section 2.2 a common choice for a DFN is a CLIP trained
image-text model. Thus, a DFN can not only be used to induce a dataset but
also be applied to common evaluation problems such as zero-shot ImageNet
classification. Inversely, a CLIP model can be both used for general
recognition as well as as a DFN. When we use a CLIP model as a DFN, we define
its filtering performance as the performance of the induced model, as
evaluated on standard benchmarks, e.g. ImageNet top-1.

Figure 2: A high level overview of our pipeline for constructing datasets
using DFNs

###  3.2 Data Filtering Networks Evaluation Setup

With these definitions in place, we now address how we evaluate DFNs. In our
context, the quality of a DFN is determined by the strength of models it can
induce. We build on the evaluation framework proposed by DataComp (Gadre et
al., 2023). DataComp provides a multi-scale evaluation framework for datasets
by measuring CLIP model zero-shot performance. It provides 4 nested unfiltered
image-text pair pools of increasing size. In this work, we use the medium
(128M datapoints), large (1.28B datapoints) and xlarge(12.8B datapoints)
pools. We also follow the DataComp guidelines of model hyperparameters for
each of these pools, which are ViT-B/32 for medium, ViT-B/16 for large and
ViT-L/14 for XL. Exact hyperparameters can be found in Table 7. We
additionally expand our DFN to a larger pool of 42B images by combining 30B
non-DataComp web-scraped images with the DataComp XL pool. We denote the
dataset induced using this pool and our DFN as DFN-5B, which we use to train a
ViT-H/14 model.

For evaluation we use 38 zero-shot classification and retrieval tasks in the
DataComp benchmark. We denote the average performance on these benchmarks
simply as “Average” performance, but we also track various subsets: ImageNet
performance (IN), ImageNet distribution shift performance (IN shifts), Visual
Task Adapation Benchmark (VTAB), Retrieval performance (COCO, Flickr,
WinoGAViL).

Our actual training runs on both Nvidia A100s and TPU v4s. We use OpenClip and
AXlearn to train our CLIP models on GPUs and TPUs respectively (Ilharco et
al., 2021; Apple, 2023) .

###  3.3 Understanding Data Filtering Networks

Figure 3: Filtering strength is uncorrelated with image task performance. The
models are trained using CLIP, and the number of samples seen and the training
data are displayed on the right hand side. Filtering performance is measured
by filtering on DataComp medium.

As open source CLIP-models improve on standard vision metrics such as
ImageNet, the question arises whether we can replace the OpenAI CLIP model
used in the dataset construction process with one of these better models. We
can even hope of recursively applying this process to continuously train
better models that can be used as better filtering models that once again
yield even better models. Unfortunately, this does not seem to be true. Figure
3 shows that ImageNet performance of CLIP models is not correlated with
filtering performance. To measure filtering performance, we create a dataset
by using the CLIP model to apply CLIP filtering on DataComp’s medium raw pool,
and measure ImageNet performance of models trained on the induced dataset. It
is especially striking that a model with 30% less ImageNet performance than
OpenAI’s CLIP models can be as good when used as a filtering model.

We find that data quality is key to training good filtering models. To
demonstrate this, we start with a high-quality pool of 10 million samples from
Conceptual 12M (CC12M), and gradually replace it with unfiltered data from
Common Crawl until this pool only contains Common Crawl. We train DFNs on
these data mixes, and use these DFNs to CLIP filter a separate pool of 128
million Common Crawl samples from DataComp’s medium scale. In Figure 4, we
measure the ImageNet performance of both the DFNs and the induced models
trained on datasets generated by each of the DFNs. While the ImageNet
performance of the DFNs degrade steadily as they are trained on larger
fractions of unfiltered data, their performance as filtering networks
decreases immediately when the high-quality pool is “poisoned” with even a
small portion of unfiltered data. Once the filtering training pool is
poisoned, the dataset induced by the DFN is only slightly better than
unfiltered data.

Figure 4: Data quality determines the filtering performance of models. We create these filter training datasets of various quality by having a set pool size of 10 million samples, and interpolating between CC-12M (high quality) and CommonPool (low quality). We then train models induced by the DFN filtering DataComp medium. Table 1: Filtering Performance of various filtering models, after filtering DataComp medium scale (ViT-B/32, 128M samples seen). We present results on ImageNet top-1 as well as “Average” set of tasks (see Sec. 3.2 for details.)  DFN Type | Filter Dataset | ImageNet | Average
---|---|---|---
No Filter Baseline | None | 0.176 | 0.258
ResNet-34 Image Binary Filter | ImageNet | 0.242 | 0.292
OpenAI ViT-B/32 Image Binary Filter | ImageNet | 0.266 | 0.295
ResNet-34 Image Binary Filter | CC12M | 0.203 | 0.257
OpenAI ViT-B/32 Image Binary Filter | CC12M | 0.218 | 0.276
M3AE ViT-B/16 | CC12M | 0.237 | 0.297
CLIP ViT-B/32 | CC12M | 0.289 | 0.335

Next, we explore using filtering models beyond CLIP models. While DFNs can use
any model that can be reduced to a binary function, intuitively it makes sense
to use CLIP models. By filtering with a similarity score between the image and
text, we encourage keeping samples where the image and text are aligned.

In order to verify this intuition we consider a few other options to produce a
DFN. One is to train a binary classifier that can distinguish between ImageNet
or CC12M data as positives and Common Crawl as negatives. We consider both
ResNet (He et al., 2016) as well as frozen OpenAI CLIP embeddings for this
filter. Another option is to use M3AE (Geng et al., 2022) trained on CC12M as
a DFN that takes into account both images and text. We can use reconstruction
loss as the filtering criterion, as it is a reasonable proxy for how similar
samples are to the high-quality data used to train the filtering model.

The filtering performance of all these options, including CLIP models, are
summarized in Table 1, where the CLIP model outperform the other backbones. A
key difference between the binary classifier and CLIP filters is that the
binary filter makes an explicit assumption on what qualifies as a good
distribution, while CLIP filters are more flexible. Although the M3AE and CLIP
filtering models both are trained on CC12M and examine both modalities, M3AE
performs much worse, potentially due to a combination of CLIP encouraging
image-text alignment and the difficulty of text reconstruction from just
CC12M. We conclude that CLIP models are the most practical and performant
models for image-text DFNs.

##  4 Creating Better Data Filtering Networks

Equipped with a better understanding of CLIP models as data filtering
networks, we aim to create better data filtering networks. DFNs can be trained
and modified in the same ways as standard machine learning models. We start by
training a CLIP model on a high-quality dataset, and then we can fine-tune the
filtering network on subsequent datasets that we want to do especially well
on. We use weight ensembling to reduce overfitting on the fine-tuned datasets.
Standard machine learning techniques such as augmentation, using a different
initialization, and training for more steps with a larger batch size seem to
improve the filtering model. We demonstrate the effect of these interventions
in Table 2. On the other hand, using a different model size seems to have
limited benefits, while model ensembling increases filtering costs without
bringing gains. Compared to previous datasets such as DataComp-1B (DC-1B)
which involved combining CLIP filtering with clustering-based heuristics, DFNs
simplify the data filtering process into a single pipeline while also reducing
computational costs.

Table 2: Standard interventions used to improve models can be used on DFNs to induce stronger datasets, leading to better models. DFNs are used to filter and train DataComp large (ViT-B/16, 1.28B samples seen).  Intervention |  | IN | IN Shifts | VTAB | Retrieval | Average
---|---|---|---|---|---|---
| ✗ | 0.620 | 0.493 | 0.534 | 0.515 | 0.536
Augmentation | ✓ | 0.626 | 0.501 | 0.534 | 0.516 | 0.542
| 2.56B / 4096 | 0.626 | 0.506 | 0.536 | 0.511 | 0.545
Samples Seen / Batch Size | 5.12B / 8192 | 0.624 | 0.508 | 0.551 | 0.517 | 0.550
| ✗ | 0.624 | 0.508 | 0.551 | 0.517 | 0.550
Fine-tune | ✓ | 0.678 | 0.540 | 0.555 | 0.534 | 0.560
| ✗ | 0.674 | 0.535 | 0.533 | 0.529 | 0.548
OAI-Init | ✓ | 0.678 | 0.540 | 0.555 | 0.534 | 0.560
Table 3: Training on DFN-2B produces state-of-the-art CLIP models. Here we evaluate on the DataComp benchmark, comparing against LAION-2B, DC-1B, MetaCLIP and OpenAI WIT-400M. Additional comparisons can be found on the DataComp leaderboard. | DataComp |  |  |  |  |
---|---|---|---|---|---|---
Dataset | Scale | IN | IN Shifts | VTAB | Retrieval | Average
DC-1B | medium | 0.297 | 0.239 | 0.346 | 0.231 | 0.328
DFN-2B | medium | 0.371 | 0.298 | 0.388 | 0.288 | 0.373
DC-1B | large | 0.631 | 0.508 | 0.546 | 0.498 | 0.537
DFN-2B | large | 0.678 | 0.540 | 0.555 | 0.534 | 0.560
LAION-2B | xlarge | 0.731 | 0.603 | 0.586 | 0.589 | 0.601
OpenAI WIT-400M | xlarge | 0.755 | 0.649 | 0.586 | 0.543 | 0.617
DC-1B | xlarge | 0.792 | 0.679 | 0.652 | 0.608 | 0.663
DFN-2B | xlarge | 0.814 | 0.688 | 0.656 | 0.649 | 0.669
LAION-2B | N/A, ViT-G/14 | 0.801 | - | - | - | 0.667
DC-1B (CLIPA-v2) | N/A, ViT-H/14 | 0.818 | - | - | - | -
MetaCLIP | N/A, ViT-H/14 | 0.805 | - | - | - | -
DFN-5B | N/A, ViT-H/14 | 0.830 | 0.708 | 0.677 | 0.678 | 0.696

To create our best DFN, we train a ViT-B/32 CLIP model on High-Quality Image-
Text Pairs (HQITP-350M), which is a high-quality dataset of 357 million image-
text samples with human-verified captions. This dataset is similar to the
HQITP-135M used in Ranasinghe et al. (2023), but expanded to 357M examples. We
initialize the weights with OpenAI’s checkpoint. We then fine-tune on the
combined MS COCO training set, Flickr30k training set, and ImageNet 1k with
OpenAI templates as the captions. We use additional augmentation at both
training and fine-tuning time. Additional training details can be found in
Appendix B. We create our dataset DFN-2B by applying this DFN on DataComp’s
full 12.8 billion sample CommonPool, with a threshold equivalent to taking the
top 15% of samples.

Our DFN induces datasets that achieve state-of-the-art results on medium,
large, and xlarge scales in DataComp. In particular at xlarge, we train a
ViT-L/14 on DFN-2B for 12.8B samples seen to achieve 81.4% zero-shot accuracy
on ImageNet, and a 0.669 average over 38 DataComp evaluation datasets. As
shown in Table 3, in terms of ImageNet zero-shot improvement, this is a 2.2%
improvement over DC-1B, a 5.9% improvement over OpenAI WIT-400M, and a 8.3%
improvement over LAION-2B. These improvements are beyond ImageNet, as we can
see similar trends across the DataComp evaluation suite in distribution
shifts, retrieval, VTAB, and average performance.

Creating better datasets not only improves model performance, but also
improves model efficiency. Performance that was once only achievable by larger
models can be matched with a smaller model trained on a better dataset. Our
ViT-L/14 trained on DFN-2B surpasses a ViT-G/14 trained on LAION-2B for 34B
samples seen by 1.3% zero-shot accuracy on ImageNet, and by 0.002 average
performance, despite using 16x less computational cost 222calculation does not
take into account patch dropout used to train ViT-G/14 on LAION-2B. Similarly,
we can train a ViT-B/16 on DFN-2B for 12.8B samples seen to achieve
competitive performance with OpenAI’s ViT-L/14, representing a 4x
computational cost reduction. Lastly, we train DFN-5B on a ViT-H/14 for 39B
samples seen, achieving 83.0% zero-shot transfer accuracy on ImageNet, out-
performing the previous state-of-the-art ViT-H/14 trained on DC-1B which
reached 81.8% (Li et al., 2023b).

Table 4: High-quality data is best used to train the filtering model rather than the end model. Training DFNs with HQITP-350M induces a dataset that outperforms the dataset induced by a worse DFN combined with HQITP-350M. Dataset | Model | IN | IN Shifts | VTAB | Retrieval | Average
---|---|---|---|---|---|---
OAI ViT-B/32 Induced Dataset |  |  |  |  |  |
\+ HQITP-350M | ViT-B/16 | 0.706 | 0.572 | 0.582 | 0.575 | 0.596
DFN without FT Induced Dataset | ViT-B/16 | 0.729 | 0.599 | 0.604 | 0.597 | 0.612
DFN-2B | ViT-B/16 | 0.762 | 0.623 | 0.598 | 0.611 | 0.609
OAI ViT-B/32 Induced Dataset |  |  |  |  |  |
\+ HQITP-350M | ViT-L/14 | 0.774 | 0.654 | 0.643 | 0.616 | 0.654
DFN-2B | ViT-L/14 | 0.814 | 0.688 | 0.656 | 0.649 | 0.669
DFN-2B + HQITP-350M | ViT-L/14 | 0.813 | 0.691 | 0.662 | 0.656 | 0.670

The key to training good DFNs is using high-quality data for training the
filtering network. Collecting verified high-quality data is expensive, as it
often requires human annotations, and is thus difficult to scale to large
quantities. But given a sizable high-quality dataset, we can explore if there
are benefits to directly training on it instead of using it to train a DFN. In
Table 4, we compare models trained on datasets induced by our DFNs with a
model trained on HQITP-350M combined with the dataset induced by CLIP
filtering CommonPool with OpenAI’s ViT-B/32. Models trained on DFN induced
datasets outperform the baseline on all major categories within the DataComp
evaluation suite. Furthermore, training on the combination of HQITP-350M and
DFN-2B seems to have little improvement when compared to just training on
DFN-2B. By training a DFN instead of directly training on high-quality data,
we demonstrate a successful recipe for leveraging high-quality data for
creating large-scale high-quality datasets.

Figure 5: Datasets induced by DFNs can be robust to distribution shift. DFNs
can be fine-tuned to maintain robustness of induced datasets, unlike directly
training on ImageNet (IN). DFNs are not performing distillation because
induced datasets lead to higher performing models than the original DFN.
Distribution shifts used are IN-V2, ObjectNet, IN-Sketch, IN-R, and IN-A.

We can also explore the differences between fine-tuning a DFN and directly
training on the fine-tuning dataset. In Figure 5 and Table 8, we compare
models trained on a dataset induced by a baseline DFN, a dataset induced by
the baseline DFN fine-tuned on ImageNet and a dataset induced by the baseline
DFN without fine-tuning on ImageNet combined with ImageNet. While the model
that directly trains on ImageNet has much higher performance on ImageNet and
ImageNet-V2, it does not improve upon the baseline for the ObjectNet,
ImageNet-Sketch, and ImageNet-R. On the other hand, the DFN fine-tuned on
ImageNet induces a dataset that improves over the baseline on ImageNet and all
of its distribution shifts. Fine-tuning on DFNs acts as a regularizer to
induce datasets similar to the fine-tuning dataset, while maintaining strong
robustness properties that come with drawing from a more distributionally
diverse candidate pool.

###  4.1 Better Datasets Beyond Vision Tasks: VQA

Just like how machine learning models would ideally generalize across many
tasks, we would also like our datasets to generalize across diverse tasks. We
show that our datasets not only lead to better models when evaluated on vision
tasks, but also lead to better visual question answering (VQA) models. We
train a BLIP2 model (Li et al., 2023a) which takes as input a CLIP visual
encoder and is trained for zero-shot VQA on COCO and Visual Genome, to measure
zero-shot VQA performance on VQVA2, GQA, and OKVQA (Goyal et al., 2017; Hudson
& Manning, 2019; Marino et al., 2019). We compare the performance on BLIP2
between the standard OpenAI ViT-L visual encoder and the ViT-L trained on
DFN-2B. The DFN-2B model consistently outperforms the OpenAI ViT-L encoder and
is competitive with a much larger EVA ViT-g model trained on LAION-2B333EVA’s
ViT-g has an additional pre-training procedure trained on ImageNet-21k, COCO,
Objects365 and Conceptual Captions 12M

Table 5: Performance of BLIP-2 variants with different visual encoder training datasets. The DFN-2B trained ViT-L provides consistent improvements across multiple zero-shot VQA tasks. Visual Encoder |  |  |  |
---|---|---|---|---
Training Dataset | Architecture | VQAv2 Acc. (%) | GQA Acc. (%) | OKVQ Acc. (%)
OAI-WIT-400M | ViT-L | 45.5 | 30.0 | 19.1
DFN-2B | ViT-L | 48.3 | 31.3 | 21.9
LAION-2B | ViT-g | 48.7 | 31.1 | 24.5

###  4.2 Publicly Reproducible DFNs

Table 6: DFNs are trained with a ViT-B/32, then used to filter DataComp pools. Conceptual 12M, Conceptual Captions 3M, and Shutterstock 15M are publicly available datasets, demonstrating that large-scale high-quality datasets can be constructed with only publicly available resources. | DataComp |  |  |  |  |
---|---|---|---|---|---|---
DFN Training Data | Scale | IN | IN Shifts | VTAB | Retrieval | Average
CC12M + CC3M + SS15M | medium | 0.307 | 0.253 | 0.359 | 0.274 | 0.349
OpenAI WIT-400M | medium | 0.285 | 0.240 | 0.355 | 0.253 | 0.338
CC12M + CC3M + SS15M | large | 0.591 | 0.481 | 0.522 | 0.503 | 0.532
OpenAI WIT-400M | large | 0.578 | 0.466 | 0.525 | 0.475 | 0.527

Scientific research benefits from results that can be reproduced by anyone
from scratch. Though OpenAI’s internal dataset and HQITP-350M are not publicly
accessible, we demonstrate that a competitive DFN can be trained on public
data sources. We train a ViT-B/32 on Conceptual Caption12M, Conceptual
Captions 3M, and Shutterstock 15M (Changpinyo et al., 2021; Sharma et al.,
2018; Nguyen et al., 2023). As shown in Table 6, this DFN matches OpenAI’s
ViT-B/32 in terms of filtering performance at DataComp’s medium and large
scales. Additionally, this DFN can be modified as described in the previous
section to further improve filtering performance.

##  5 Discussion

The simplicity of the data filtering network pipeline makes it a flexible tool
to integrate into existing workflows. As DFNs operates on individual samples,
this approach scales linearly with candidate pool size, enabling the creation
of datasets orders of magnitude larger than those that we introduce in this
work. Additionally, the DFNs we train in this work are relatively small neural
networks which allows for filtering to be directly integrated into training
procedures of much larger networks for minimal marginal cost. DFNs can then
filter batches of raw data that are then trained on, reducing the need for
complex data pre-processing procedures.

As useful as DFNs are in building performant models in this work, there are
still many unanswered questions to address in future work. We still don’t know
exactly how to optimize directly for dataset quality, and thus opt for weak
proxies such as alignment. It is not even clear what that proxy would be for
other domains where DFNs could be applied such as speech, text or video data.
We hope that these open questions and the bridge DFNs build between modeling
work and dataset work can lead to fruitful new avenues of research.

## References

  * Aghajanyan et al. (2023) Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer.  Scaling laws for generative mixed-modal language models.  _arXiv preprint arXiv:2301.03728_ , 2023.
  * Apple (2023) Apple.  axlearn, July 2023.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Byeon et al. (2022) Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim.  Coyo-700m: Image-text pair dataset.  https://github.com/kakaobrain/coyo-dataset, 2022.
  * Changpinyo et al. (2021) Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 3558–3568, 2021.
  * Cherti et al. (2023) Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev.  Reproducible scaling laws for contrastive language-image learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2818–2829, 2023.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2009.  https://ieeexplore.ieee.org/abstract/document/5206848.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Elsen et al. (2023) Erich Elsen, Augustus Odena, Maxwell Nye, Sağnak Taşırlar, Tri Dao, Curtis Hawthorne, Deepak Moparthi, and Arushi Somani.  Releasing Persimmon-8B, 2023.  URL https://www.adept.ai/blog/persimmon-8b.
  * Fang et al. (2022) Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt.  Data determines distributional robustness in contrastive language image pre-training (clip), 2022.
  * Gadre et al. (2023) Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al.  Datacomp: In search of the next generation of multimodal datasets.  _arXiv preprint arXiv:2304.14108_ , 2023.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al.  The pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_ , 2020.
  * Geng et al. (2022) Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurams, Sergey Levine, and Pieter Abbeel.  Multimodal masked autoencoders learn transferable representations.  _arXiv preprint arXiv:2205.14204_ , 2022.
  * Goyal et al. (2022) Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan.  Finetune like you pretrain: Improved finetuning of zero-shot vision models, 2022.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.  https://arxiv.org/abs/1512.03385.
  * Hestness et al. (2017) Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou.  Deep learning scaling is predictable, empirically, 2017.
  * Highleyman & Kamentsky (1959) W. H. Highleyman and L. A. Kamentsky.  A generalized scanner for pattern- and character-recognition studies.  In _Papers Presented at the the March 3-5, 1959, Western Joint Computer Conference_ , IRE-AIEE-ACM ’59 (Western), pp. 291–294, New York, NY, USA, 1959. Association for Computing Machinery.  ISBN 9781450378659.  doi: 10.1145/1457838.1457894.  URL https://doi.org/10.1145/1457838.1457894.
  * Hudson & Manning (2019) Drew A. Hudson and Christopher D. Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019.
  * Ilharco et al. (2021) Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.  Openclip, July 2021.  URL https://doi.org/10.5281/zenodo.5143773.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision, 2021.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images, 2009.  https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2012.  https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf.
  * LeCun et al. (1989) Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel.  Backpropagation applied to handwritten zip code recognition.  _Neural Computation_ , 1(4):541–551, 1989.  doi: 10.1162/neco.1989.1.4.541.
  * LeCun et al. (1998) Yann LeCun, Yann LeCun, and Yann LeCun.  The mnist database of handwritten digits, 1998.  http://yann.lecun.com/exdb/mnist/.
  * Li et al. (2023a) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023a.
  * Li et al. (2023b) Xianhang Li, Zeyu Wang, and Cihang Xie.  Clipa-v2: Scaling clip training with 81.1accuracy within a $10,000 budget; an extra $4,000 unlocks 81.82023b.
  * Marino et al. (2019) Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Mazumder et al. (2022) Mark Mazumder, Colby Banbury, Xiaozhe Yao, Bojan Karlaš, William Gaviria Rojas, Sudnya Diamos, Greg Diamos, Lynn He, Douwe Kiela, David Jurado, David Kanter, Rafael Mosquera, Juan Ciro, Lora Aroyo, Bilge Acun, Sabri Eyuboglu, Amirata Ghorbani, Emmett Goodman, Tariq Kane, Christine R. Kirkpatrick, Tzu-Sheng Kuo, Jonas Mueller, Tristan Thrush, Joaquin Vanschoren, Margaret Warren, Adina Williams, Serena Yeung, Newsha Ardalani, Praveen Paritosh, Ce Zhang, James Zou, Carole-Jean Wu, Cody Coleman, Andrew Ng, Peter Mattson, and Vijay Janapa Reddi.  Dataperf: Benchmarks for data-centric ai development, 2022.  https://arxiv.org/abs/2207.10062.
  * Nguyen et al. (2023) Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt.  Quality not quantity: On the interaction between dataset design and robustness of clip, 2023.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report, 2023.
  * Pham et al. (2023) Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, Mingxing Tan, and Quoc V. Le.  Combined scaling for zero-shot transfer learning, 2023.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning (ICML)_ , 2021.  https://arxiv.org/abs/2103.00020.
  * Radford et al. (2022) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.  Robust speech recognition via large-scale weak supervision.  _arXiv preprint arXiv:2212.04356_ , 2022.
  * Ranasinghe et al. (2023) Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander Toshev, and Jonathon Shlens.  Perceptual grouping in contrastive vision-language models, 2023.
  * Sambasivan et al. (2021) Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo.  “everyone wants to do the model work, not the data work”: Data cascades in high-stakes ai.  In _proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , pp. 1–15, 2021.
  * Schuhmann et al. (2021) Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  LAION-400M: Open dataset of clip-filtered 400 million image-text pairs, 2021.  https://arxiv.org/abs/2111.02114.
  * Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  LAION-5B: An open large-scale dataset for training next generation image-text models.  In _Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track_ , 2022.  https://openreview.net/forum?id=M3Y74vmsMcY.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of ACL_ , 2018.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Wenzek et al. (2019) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave.  Ccnet: Extracting high quality monolingual datasets from web crawl data.  _arXiv preprint arXiv:1911.00359_ , 2019.
  * Wortsman et al. (2022) Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt.  Robust fine-tuning of zero-shot models, 2022.
  * Xu et al. (2023) Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer.  Demystifying clip data, 2023.
  * Zhai et al. (2022a) Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.  Scaling vision transformers, 2022a.
  * Zhai et al. (2022b) Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  Lit: Zero-shot transfer with locked-image text tuning, 2022b.

##  Appendix A Training Hyperparameters

Dataset | Model | Pool size and | Batch | Max LR | Weight | Warmup | Beta2
---|---|---|---|---|---|---|---
|  | # seen samples | Size |  | Decay |  |
DataComp-medium | ViT-B/32 | 128M | 4096 | 5e-4 | 0.2 | 500 | -
DataComp-large | ViT-B/16 | 1.28B | 8192 | 5e-4 | 0.2 | 500 | -
DataComp-xlarge | ViT-L/14 | 12.8B | 90112 | 1e-3 | 0.2 | 10000 | 0.95
DFN-5B-pool | ViT-H/14 | 39B | 79872 | 2e-3 | 0.2 | 10000 | 0.95
Table 7: We follow the hyperparameter settings of the DataComp paper for the
medium, large and xlarge pool

##  Appendix B DFN Hyperparameters

DFNs trained for ablations use DataComp large scale hyperparameters with a
ViT-B/32 instead of a ViT-B/16. Final DFNs that induce DC-2B train for 5.12B
samples, 16,384 batch size, and 2,000 steps of warmup.

##  Appendix C Robustness of Using ImageNet at Filtering vs. Training Time

Table 8: Fine-tuning a DFN on ImageNet induces datasets with nice robustness properties that are lost when directly training on ImageNet. Ran at DataComp large scale (ViT-B/16, 1.28B samples).  Dataset | IN | IN-V2 | ObjectNet | IN-Sketch | IN-R | IN-A
---|---|---|---|---|---|---
Baseline DFN | 0.624 | 0.547 | 0.511 | 0.510 | 0.724 | 0.257
Baseline DFN FT on ImageNet | 0.678 | 0.594 | 0.536 | 0.536 | 0.743 | 0.284
Baseline DFN + IN | 0.757 | 0.652 | 0.509 | 0.512 | 0.703 | 0.272

##  Appendix D Full Experimental Evaluation

We will release a link containing the performance of models in Table 3 on each
of the 38 DataComp evaluation datasets.

##  Appendix E Figures measuring average performance instead of ImageNet

Figure 6: Average accuracy version of Figure 3. Filtering strength is
uncorrelated with image task performance. The models are trained using CLIP,
and the number of samples seen and the training data are displayed on the
right hand side. Filtering performance is measured by filtering on DataComp
medium. Figure 7: Average accuracy version of Figure 4. Data quality
determines the filtering performance of models. We create these filter
training datasets of various quality by having a set pool size of 10 million
samples, and interpolating between CC-12M (high quality) and CommonPool (low
quality). We then train models induced by the DFN filtering DataComp medium.

##  Appendix F Log-Scale plot of Figure 1

Figure 8: Compute scaling behavior of training CLIP models on various datasets
(log scale)

##  Appendix G Additional Tables

Table 9: We can produce high-quality DFNs completely from scratch. Specifically, we do not use any OpenAI CLIP models for results in this table. We also use HQITP-135M for DFN training, a subset of HQITP-350M that we use in the rest of the paper | DataComp |  |  |  |  |
---|---|---|---|---|---|---
Dataset | Scale | IN | IN Shifts | VTAB | Retrieval | Average
Induced by DFN HQITP-135M |  |  |  |  |  |
with FT on IN1k, |  |  |  |  |  |
no OAI-Init, no Aug., |  |  |  |  |  |
samples 1B, BS 4096 | xlarge | 0.805 | 0.665 | 0.641 | 0.639 | 0.663


* [152] Y. G. Sinai, Introduction to Ergodic Theory (Princeton University Press, Princeton, New Jersey, 1977), Vol. 18.
  * [153] I. P. Cornfield, S. V. Fomin, and Y. G. Sinai, Ergodic Theory (Springer-Verlag, New York, 1982).
  * [154] P. R. Halmos, Lectures on Ergodic Theory (Dover, Mineola, New York, 2017).
  * [155] A. S. Pikovsky and U. Feudel, Correlations and spectra of strange nonchaotic attractors J. Phys. A 27, 5209 (1994).
  * [156] J. M. Luck, H. Orland, and U. Smilansky, On the response of a two-level quantum system to a class of time-dependent quasiperiodic perturbations, J. Stat. Phys. 53, 551 (1988).
  * [157] A. S. Pikovsky, M. A. Zaks, and J. Kurths, Complexity of a quasiperiodically driven spin system, J. Phys. A 29, 295 (1996).
  * [158] P. M. Blekher, H. R. Jauslin, and J. L. Lebowitz, Floquet spectrum for two-level systems in quasiperiodic time-dependent fields, J. Stat. Phys. 68, 271 (1992).
  * [159] T. Geisel, Quasiperiodicity versus mixing instability in a kicked quantum system, Phys. Rev. A 41, 2989 (1990).
  * [160] M. Agrawal, A. Prasad, and R. Ramaswamy, Quasiperiodic forcing of coupled chaotic systems, Phys. Rev. E 81, 026202 (2010).
  * [161] W. L. Ditto, M. L. Spano, H. T. Savage, S. N. Rauseo, J. Heagy, and E. Ott, Experimental observation of a strange nonchaotic attractor, Phys. Rev. Lett. 65, 533 (1990).
  * [162] P. T. Dumitrescu, R. Vasseur, and A. C. Potter, Logarithmically Slow Relaxation in Quasiperiodically Driven Random Spin Chains, Phys. Rev. Lett. 120, 070602 (2018).
  * [163] S. Ray, S. Sinha, and D. Sen, Dynamics of quasiperiodically driven spin systems, Phys. Rev. E 100, 052129 (2019).
  * [164] S. Nandy, A. Sen, and D. Sen, Aperiodically Driven Integrable Systems and Their Emergent Steady States, Phys. Rev. X 7, 031034 (2017).
  * [165] S. Nandy, A. Sen, and D. Sen, Steady states of a quasiperiodically driven integrable system, Phys. Rev. B 98, 245144 (2018).
  * [166] S. Maity, U. Bhattacharya, A. Dutta, and D. Sen, Fibonacci steady states in a driven integrable quantum system, Phys. Rev. B 99, 020306(R) (2019).
  * [167] B. V. Chirikov, Research Concerning the Theory of Non-linear Resonance and Stochasticity, Preprint No. 267 (Institute of Nuclear Physics, Novosibirsk, 1969).
  * [168] M. S. Santhanam, S. Paul, and J. B. Kannan, Quantum kicked rotor and its variants: Chaos, localization and beyond, Phys. Rep. 956, 1 (2022).
  * [169] Classical and quantum chaos for a kicked top, F. Haake, M. Kus, and R. Scharf, Z. Phys. B 65, 381 (1987).
  * [170] A. N. Kolmogorov, A new metric invariant of transient dynamical systems and automorphisms in lebesgue spaces, Doklady Akademii Nauk SSSR (Russian Academy of Sciences, 1958), Vol. 119, pp. 861–864.
  * [171] A. N. Kolmogorov, Entropy per unit time as a metric invariant of automorphisms, Doklady Akademii Nauk SSSR (Russian Academy of Sciences, 1959), Vol. 124, pp. 754–755.
  * [172] Y. B. Pesin, Characteristic Lyapunov exponents and smooth ergodic theory, Russ. Math. Surv. 32, 55 (1977).
  * [173] O. Bohigas, M. J. Giannoni, and C. Schmit, Characterization of Chaotic Quantum Spectra and Universality of Level Fluctuation Laws, Phys. Rev. Lett. 52, 1 (1984).
  * [174] M. V. Berry and M. Tabor, Level Clustering in the Regular Spectrum, Proc. R. Soc. A 356, 375 (1977).
  * [175] C. Emary and T. Brandes, Quantum Chaos Triggered by Precursors of a Quantum Phase Transition: The Dicke Model, Phys. Rev. Lett. 90, 044101 (2003).
  * [176] C. Emary and T. Brandes, Chaos and the quantum phase transition in the Dicke model, Phys. Rev. E 67, 066203 (2003).
  * [177] V. Oganesyan and D. A. Huse, Localization of interacting fermions at high temperature, Phys. Rev. B 75, 155111 (2007).
  * [178] Y. Y. Atas, E. Bogomolny, O. Giraud, and G. Roux, Distribution of the Ratio of Consecutive Level Spacings in Random Matrix Ensembles, Phys. Rev. Lett. 110, 084101 (2013).
  * [179] T. Brody, A statistical measure for the repulsion of energy levels, Lett. Nuovo Cimento 7, 482 (1973).
  * [180] G. Lenz and F. Haake, Transitions between Universality Classes of Random Matrices, Phys. Rev. Lett. 65, 2325 (1990).
  * [181] T. Prosen and M. Robnik, Energy Level Statistics in the Transition Region between Integrability and Chaos, J. Phys. A 26, 2371 (1993).
  * [182] M. V. Berry and M. Robnik, Semiclassical Level Spacings When Regular and Chaotic Orbits Coexist, J. Phys. A 17, 2413 (1984).
  * [183] E. B. Bogomolny, U. Gerland, and C. Schmit, Models of Intermediate Spectral Statistics, Phys. Rev. E 59, R1315 (1999).
  * [184] R. Ketzmerick, K. Kruse, and T. Geisel, Avoided Band Crossings: Tuning Metal-Insulator Transitions in Chaotic Systems, Phys. Rev. Lett. 80, 137 (1998).
  * [185] B. B. Mandelbrot, The Fractal Geometry of Nature (Freeman, New York, 1982).
  * [186] A. Rényi, On measures of entropy and information, in Fourth Berkeley Symposium on Mathematical Statistics and Probability (Statistical Laboratory of the University of California, Berkeley, 1961), pp. 547–561.
  * [187] P. Jacquod and I. Varga, Duality Between the Weak and Strong Interaction Limits for Randomly Interacting Fermions, Phys. Rev. Lett. 89, 134101 (2002).
  * [188] S. Goldstein, J. L. Lebowitz, R. Tumulka, and N. Zanghì, Canonical typicality, Phys. Rev. Lett. 96, 050403 (2006).
  * [189] S. Popescu, A. J. Short, and A. Winter, Entanglement and the foundations of statistical mechanics, Nat. Phys. 2, 754 (2006)
  * [190] P. Reimann, Typicality for Generalized Microcanonical Ensembles, Phys. Rev. Lett. 99, 160404 (2007).
  * [191] P. Reimann, Foundation of Statistical Mechanics under Experimentally Realistic Conditions, Phys. Rev. Lett. 101, 190403 (2008).
  * [192] C. Murthy, A. Babakhani, F. Iniguez, M. Srednicki, and N. Y. Halpern, Non-Abelian Eigenstate Thermalization Hypothesis, Phys. Rev. Lett. 130, 140402 (2023).
  * [193] M. Rigol, Breakdown of Thermalization in Finite One-Dimensional Systems, Phys. Rev. Lett. 103, 100403 (2009).
  * [194] G. Biroli, C. Kollath, and A. M. Láuchli, Effect of Rare Fluctuations on the Thermalization of Isolated Quantum Systems, Phys. Rev. Lett. 105, 250401 (2010).
  * [195] R. Steinigeweg, A. Khodja, H. Niemeyer, C. Gogolin, and J.Gemmer, Pushing the Limits of the Eigenstate Thermalization Hypothesis Towards Mesoscopic Quantum Systems, Phys. Rev. Lett. 112, 130403 (2014).
  * [196] S. Sorg, L. Vidmar, L. Pollet, and F. Heidrich-Meisner, Relaxation and thermalization in the one-dimensional BoseHubbard model: A case study for the interaction quantum quench from the atomic limit, Phys. Rev. A 90, 033606 (2014).
  * [197] H. Kim, T. N. Ikeda, and D. A. Huse, Testing whether all eigenstates obey the eigenstate thermalization hypothesis, Phys. Rev. E 90, 052105 (2014).
  * [198] W. Beugeling, R. Moessner, and M. Haque, Finite-size scaling of eigenstate thermalization, Phys. Rev. E 89, 042112 (2014).
  * [199] W. Beugeling, R. Moessner, and M. Haque, Off-diagonal matrix elements of local operators in many-body quantum systems, Phys. Rev. E 91, 012144 (2015).
  * [200] C. Neuenhahn and F. Marquardt, Thermalization of interacting fermions and delocalization in Fock space, Phys. Rev. E 85, 060101(R) (2012).
  * [201] R. Mondaini, K. R. Fratus, M. Srednicki, and M. Rigol, Eigenstate thermalization in the two-dimensional transverse field Ising model, Phys. Rev. E 93, 032104 (2016).
  * [202] D. Villaseñor, S. Pilatowsky-Cameo, M. A. Bastarrachea-Magnani, S. Lerma-Hernández, L. F. Santos, and J. G. Hirsch, Chaos and Thermalization in the Spin-Boson Dicke Model ,Entropy 25, 8 (2022).
  * [203] A. Khodja, R. Steinigeweg, and J. Gemmer, Relevance of the eigenstate thermalization hypothesis for thermal relaxation, Phys. Rev. E 91, 012120 (2015).
  * [204] G. De Palma, A. Serafini, V. Giovannetti, and M. Cramer, Necessity of Eigenstate Thermalization, Phys. Rev. Lett. 115, 220401 (2015).
  * [205] T. Mori and N. Shiraishi, Thermalization without eigenstate thermalization hypothesis after a quantum quench, Phys. Rev. E 96, 022153 (2017).
  * [206] R. H. Dicke, Coherence in Spontaneous Radiation Processes, Phys. Rev. 93, 99 (1954).
  * [207] K. Hepp and E. H. Lieb, On the Superradiant Phase Transition for Molecules in a Quantized Radiation Field: The Dicke Maser Model, Ann. Phys. (N.Y.) 76, 360 (1973).
  * [208] A. Altland and F. Haake, Quantum Chaos and Effective Thermalization, Phys. Rev. Lett. 108, 073601 (2012).
  * [209] A. Altland and F. Haake, Equilibration and macroscopic quantum fluctuations in the Dicke model, New J. Phys. 14, 073011 (2012).
  * [210] T. Brandes, Excited-state quantum phase transitions in Dicke superradiance models, Phys. Rev. E 88, 032133 (2013).
  * [211] M. A. Bastarrachea-Magnani, S. Lerma-Hernández, and J. G. Hirsch, Comparative quantum and semiclassical analysis of atom-field systems. I. density of states and excited-state quantum phase transitions, Phys. Rev. A 89, 032101 (2014).
  * [212] M. A. Bastarrachea-Magnani, S. Lerma-Hernández, and J. G. Hirsch, Comparative quantum and semiclassical analysis of atom-field systems. II. Chaos and regularity, Phys. Rev. A 89, 032102 (2014).
  * [213] J. Chávez-Carlos, M. A. Bastarrachea-Magnani, S. Lerma-Hernández, and J. G. Hirsch, Classical chaos in atom-field systems, Phys. Rev. E 94, 022209 (2016).
  * [214] M. A. Bastarrachea-Magnani, B. López-del-Carpio, J. Chávez-Carlos, S. Lerma-Hernández, and J. G. Hirsch, Delocalization and quantum chaos in atom-field systems, Phys. Rev. E 93, 022215 (2016).
  * [215] J. Radcliffe, Some properties of coherent spin states, J. Phys. A 4, 313 (1971).
  * [216] D. Mondal, S. Sinha, and S. Sinha, Chaos and quantum scars in a coupled top model, Phys. Rev. E 102, 020101(R) (2020).
  * [217] D. Mondal, S. Sinha, and S. Sinha, Quantum transitions, ergodicity, and quantum scars in the coupled top model, Phys. Rev. E 105, 014130 (2022).
  * [218] Quantum signature of chaos and thermalization in the kicked Dicke model, S. Ray, A. Ghosh, and S. Sinha, Phys. Rev. E 94, 032103 (2016).
  * [219] M. Albiez, R. Gati, J. Fölling, S. Hunsmann, M. Cristiani, and M. K. Oberthaler, Direct Observation of Tunneling and Nonlinear Self-Trapping in a Single Bosonic Josephson Junction, Phys. Rev. Lett. 95, 010402 (2005).
  * [220] T. Schumm, S. Hofferberth, L. M. Andersson, S. Wildermuth, S. Groth, I. Bar-Joseph, J. Schmiedmayer, and P. Krüger, Matter-wave interferometry in a double well on an atom chip, Nat. Phys. 1, 57 (2005).
  * [221] S. Levy, E. Lahoud, I. Shomroni, and J. Steinhauer, The a.c. and d.c. Josephson effects in a Bose–Einstein condensate, Nature (London) 449, 579 (2007).
  * [222] T. Zibold, E. Nicklas, C. Gross, and M. K. Oberthaler, Classical Bifurcation at the Transition from Rabi to Josephson Dynamics, Phys. Rev. Lett. 105, 204101 (2010).
  * [223] G. J. Milburn, J. Corney, E. M. Wright, and D. F. Walls, Quantum dynamics of an atomic Bose-Einstein condensate in a double-well potential, Phys. Rev. A 55, 4318 (1997).
  * [224] A. Smerzi, S. Fantoni, S. Giovanazzi, and S. R. Shenoy, Quantum Coherent Atomic Tunneling between Two Trapped Bose-Einstein Condensates, Phys. Rev. Lett. 79, 4950 (1997).
  * [225] S. Raghavan, A. Smerzi, S. Fantoni, and S. R. Shenoy, Coherent oscillations between two weakly coupled Bose-Einstein condensates: Josephson effects, $\pi$ oscillations, and macroscopic quantum self-trapping, Phys. Rev. A 59, 620 (1999).
  * [226] I. Zapata, F. Sols, and A. J. Leggett, Josephson effect between trapped Bose-Einstein condensates, Phys. Rev. A 57, R28(R) (1998).
  * [227] A. P. Tonel, J. Links, and A. Foerster, Quantum dynamics of a model for two Josephson-coupled Bose–Einstein condensates, J. Phys. A: Math. Gen. 38, 1235 (2005).
  * [228] M. Chuchem, K. S.-Mannschott, M. Hiller, T. Kottos, A. Vardi, and D. Cohen, Quantum dynamics in the bosonic Josephson junction, Phys. Rev. A 82, 053617 (2010).
  * [229] M. Trujillo-Martinez, A. Posazhennikova, and J. Kroha, Nonequilibrium Josephson Oscillations in Bose-Einstein Condensates without Dissipation, Phys. Rev. Lett. 103, 105302 (2009).
  * [230] R. Gati and M. K. Oberthaler, A bosonic Josephson junction, J. Phys. B: At., Mol. Opt. Phys. 40, R61 (2007).
  * [231] H. Lipkin, N. Meshkov, and A. Glick, Validity of many-body approximation methods for a solvable model: (I). Exact solutions and perturbation theory, Nucl. Phys. 62, 188 (1965).
  * [232] N. Meshkov, A. J. Glick, and H. J. Lipkin, Validity of many-body approximation methods for a solvable model: (II). Linearization procedures, Nucl. Phys. 62, 199 (1965).
  * [233] A. J. Glick, H. J. Lipkin, and N. Meshkov, Validity of many-body approximation methods for a solvable model: (III). Diagram summations, Nucl. Phys. 62, 211 (1965).
  * [234] S. Sinha and S. Sinha, Dissipative Bose-Josephson junction coupled to bosonic baths, Phys. Rev. E 100, 032115 (2019).
  * [235] S. Sinha and S. Sinha, Chaos and Quantum Scars in Bose-Josephson Junction Coupled to a Bosonic Mode, Phys. Rev. Lett. 125, 134101 (2020).
  * [236] Q. Wang, Quantum Chaos in the Extended Dicke Model, Entropy 24, 1415 (2022).
  * [237] D. Mondal, S. Sinha, S. Ray, J. Kroha, and S. Sinha, Classical route to ergodicity and scarring phenomena in a two-component Bose-Josephson junction, Phys. Rev. A 106, 043321 (2022).
  * [238] K. Baumann, C. Guerlin, F. Brennecke, and T. Esslinger, Dicke quantum phase transition with a superfluid gas in an optical cavity, Nature (London) 464, 1301 (2010).
  * [239] H. Ritsch, P. Domokos, F. Brennecke, and T. Esslinger, Cold atoms in cavity-generated dynamical optical potentials, Rev. Mod. Phys. 85, 553 (2013).
  * [240] M. P. Baden, K. J. Arnold, A. L. Grimsmo, S. Parkins, and M. D. Barrett, Realization of the Dicke Model Using Cavity-Assisted Raman Transitions Phys. Rev. Lett. 113, 020408 (2014).
  * [241] J. Klinder, H. Keßler, M. Wolke, L. Mathey, and A. Hemmerich, Dynamical phase transition in the open Dicke model, Proc. Natl. Acad. Sci. USA 112, 3290 (2015).
  * [242] Z. Zhiqiang, C. H. Lee, R. Kumar, K. J. Arnold, S. J. Masson, A. S. Parkins, and M. D. Barrett, Nonequilibrium phase transition in a spin-1 Dicke model, Optica 4, 424 (2017).
  * [243] P. Kirton, M. M. Roses, J. Keeling, and E. G. Dalla Torre, Introduction to the Dicke model: From equilibrium to nonequilibrium, and vice versa, Adv. Quantum Technol. 2, 1970013 (2019).
  * [244] A. Mezzacapo, U. Las Heras, J. S. Pedernales, L. DiCarlo, E. Solano, and L. Lamata, Digital Quantum Rabi and Dicke Models in Superconducting Circuits, Sci. Rep. 4, 7482 (2014).
  * [245] S. Sachdev and J. Ye, Gapless Spin-Fluid Ground State in a Random Quantum Heisenberg Magnet, Phys. Rev. Lett. 70, 3339 (1993).
  * [246] Kitaev A 2015 A simple model of quantum holography KITP Strings Seminar and Entanglement 2015 Program (February 12, April 7, and May 27 2015) (http://online.kitp.ucsb.edu/online/entangled15/)
  * [247] M. Gärttner, J. G. Bohnet, A. Safavi-Naini, M. L. Wall, J. J. Bollinger, and A. M. Rey, Measuring out-of-time-order correlations and multiple quantum spectra in a trapped-ion quantum magnet, Nat. Phys. 13, 781 (2017).
  * [248] J. Li, R. Fan, H. Wang, B. Ye, B. Zeng, H. Zhai, X. Peng, and J. Du, Measuring Out-of-Time-Order Correlators on a Nuclear Magnetic Resonance Quantum Simulator, Phys. Rev. X 7, 031011 (2017).
  * [249] K. A. Landsman, C. Figgatt, T. Schuster, N. M. Linke, B. Yoshida, N. Y. Yao, and C. Monroe, Verified quantum information scrambling, Nature 567, 61 (2019).
  * [250] X. Mi, P. Roushan, C. Quintana, S. Mandra, J. Marshall, C. Neill, F. Arute, K. Arya, J. Atalaya, R. Babbush et al., Information scrambling in quantum circuits, Science 374, 1479 (2021).
  * [251] J. Braumüller, A. H. Karamlou, Y. Yanay, B. Kannan, D. Kim, M. Kjaergaard, A. Melville, B. M. Niedzielski, Y. Sung, A. Vepsäläinen, R. Winik, J. L. Yoder, T. P. Orlando, S. Gustavsson, C. Tahan, and W. D. Oliver, Probing quantum information propagation with out-of-time-ordered correlators, Nat. Phys. 18, 172 (2022).
  * [252] R. J. Lewis-Swan, A. Safavi-Naini, J. J. Bollinger, and A. M. Rey, Unifying scrambling, thermalization and entanglement through measurement of fidelity out-of-time-order correlators in the Dicke model, Nat. Commun. 10, 1581 (2019).
  * [253] S. P. Cameo, J. C. Carlos, M. A. B. Magnani, P. Stransky, S. L. Hernández, L. F. Santos, and J. G. Hirsch, Positive quantum Lyapunov exponents in experimental systems with a regular classical limit, Phys. Rev. E 101, 010202(R) (2020).
  * [254] R. A. Kidd, A. Safavi-Naini, and J. F. Corney, Thermalization in a Bose-Hubbard dimer with modulated tunneling, Phys. Rev. A 102, 023330 (2020).
  * [255] D. Mondal, S. Sinha, and S. Sinha, Dynamical route to ergodicity and quantum scarring in kicked coupled top, Phys. Rev. E 104, 024217 (2021).
  * [256] V. Khemani, D. A. Huse, and A. Nahum, Velocity-dependent Lyapunov exponents in many-body quantum, semiclassical, and classical chaos, Phys. Rev. B 98, 144304 (2018).
  * [257] A. Das, S. Chakrabarty, A. Dhar, A. Kundu, D. A. Huse, R. Moessner, S. S. Ray, and S. Bhattacharjee, Light-Cone Spreading of Perturbations and the Butterfly Effect in a Classical Spin Chain, Phys. Rev. Lett. 121, 024101 (2018).
  * [258] T. Bilitewski, S. Bhattacharjee, and R. Moessner, Temperature Dependence of the Butterfly Effect in a Classical Many-Body System, Phys. Rev. Lett. 121, 250602 (2018).
  * [259] T. Bilitewski, S. Bhattacharjee, and R. Moessner, Classical many-body chaos with and without quasiparticles, Phys. Rev. B 103, 174302 (2021).
  * [260] J. Lopez-Piqueres, B. Ware, S. Gopalakrishnan, and R. Vasseur, Operator front broadening in chaotic and integrable quantum chains,Phys. Rev. B 104, 104307 (2021).
  * [261] S. Liu, J. Willsher, T. Bilitewski, J. Li, A. Smith, K. Christensen, R. Moessner, and J. Knolle, Butterfly effect and spatial structure of information spreading in a chaotic cellular automaton, Phys. Rev. B 103, 094109 (2021).
  * [262] S. D. Murugan, D. Kumar, S. Bhattacharjee, and S. S. Ray, Many-Body Chaos in Thermalized Fluids, Phys. Rev. Lett. 127, 124501 (2021).
  * [263] A. K. Chatterjee, A. Kundu, and M. Kulkarni, Spatiotemporal spread of perturbations in a driven dissipative Duffing chain: An out-of-time-ordered correlator approach, Phys. Rev. E 102, 052103 (2020).
  * [264] S. Ruidas, and S. Banerjee, Many-body chaos and anomalous diffusion across thermal phase transitions in two dimensions, SciPost Phys. 11, 087 (2021).
  * [265] W. Kirkby, D. H. J. O’Dell, and J. Mumford, False signals of chaos from quantum probes, Phys. Rev. A 104, 043308 (2021).
  * [266] T. Xu, T. Scaffidi, and X. Cao, Does Scrambling Equal Chaos?, Phys. Rev. A 124, 140602 (2020).
  * [267] S. Pappalardi, A.Russomanno, B. Žunkovič, F. Iemini, A. Silva, and R. Fazio, Scrambling and entanglement spreading in long-range spin chains, Phys. Rev. B 98, 134303 (2018).
  * [268] E. B. Rozenbaum, L. A. Bunimovich, and V. Galitski, Early-Time Exponential Instabilities in Nonchaotic Quantum Systems, Phys. Rev. Lett. 125, 014101 (2020).
  * [269] K. Hashimoto, K.-B. Huh, K.-Y. Kim, and R. Watanabe, Exponential growth of out-of-time-order correlator without chaos: Inverted harmonic oscillator, J. High Energy Phys. 11 (2020) 068.
  * [270] S. Ghosh, K. S. Gupta, and S. C. L. Srivastava, Exact relaxation dynamics and quantum information scrambling in multiply quenched harmonic chains, pre 100, 012215 (2019).
  * [271] L. Amico, R. Fazio, A. Osterloh, and V. Vedral, Entanglement in many-body systems, Rev. Mod. Phys. 80, 517 (2008).
  * [272] J. Vidal, S. Dusuel, and T. Barthel, Entanglement entropy in collective models, J. Stat. Mech. (2007) P01015.
  * [273] I. Peschel and V. Eisler, Reduced Density Matrices and Entanglement Entropy in Free Lattice Models, J. Phys. A 42, 504003 (2009).
  * [274] R. Horodecki, P. Horodecki, M. Horodecki, and K. Horodecki, Quantum Entanglement, Rev. Mod. Phys. 81, 865 (2009).
  * [275] M. M. Lennert, F. Dupuis, O. Szehr, S. Fehr, and M. Tomamichel, On quantum Rényi entropies: A new generalization and some properties, J. Math. Phys. 54 122203 (2013).
  * [276] E. Bianchi, L. Hackl, M. Kieburg, M. Rigol, and L. Vidmar, Volume-Law Entanglement Entropy of Typical Pure Quantum States ,PRX Quantum 3, 030201 (2022).
  * [277] J. Eisert, M. Cramer, and M. B. Plenio, Colloquium: Area laws for the entanglement entropy, Rev. Mod. Phys. 82, 277 (2010).
  * [278] J. M. Deutsch, Thermodynamic entropy of a many-body energy eigenstate, New J. Phys. 12, 075021 (2010).
  * [279] J. M. Deutsch, H. Li, and A. Sharma, Microscopic origin of thermodynamic entropy in isolated systems, Phys. Rev. E 87, 042135 (2013).
  * [280] D. N. Page, Average entropy of a subsystem, Phys. Rev. Lett. 71, 1291 (1993).
  * [281] L. Vidmar and M. Rigol, Entanglement Entropy of Eigenstates of Quantum Chaotic Hamiltonians, Phys. Rev. Lett. 119, 220603 (2017).
  * [282] L. Vidmar, L. Hackl, E. Bianchi, and M. Rigol, Entanglement Entropy of Eigenstates of Quadratic Fermionic Hamiltonians, Phys. Rev. Lett. 119, 020601 (2017).
  * [283] C. Murthy and M. Srednicki, Structure of chaotic eigenstates and their entanglement entropy, Phys. Rev. E 100, 022131 (2019).
  * [284] T. C. Lu and T. Grover, Renyi entropy of chaotic eigenstates, Phys. Rev. E 99, 032111 (2019).
  * [285] P. Hyllus, W. Laskowski, R. Krischek, C. Schwemmer, W. Wieczorek, H. Weinfurter, L. Pezzé, and A. Smerzi, Fisher information and multiparticle entanglement, Phys. Rev. A 85, 022321 (2012).
  * [286] P. Hauke, M. Heyl, L. Tagliacozzo, and P. Zoller, Measuring multipartite entanglement through dynamic susceptibilities, Nat. Phys. 12, 778 (2016).
  * [287] K. Gietka, J. Chwedeńczuk, T. Wasak, and F. Piazza, Multipartite entanglement dynamics in a regular-to-ergodic transition: Quantum Fisher information approach, Phys. Rev. B 99, 064303 (2019).
  * [288] S. Chaudhury, A. Smith, B. E. Anderson, S. Ghose, and P. S. Jessen, Quantum signatures of chaos in a kicked top, Nature 461, 768 (2009).
  * [289] C. Neill et al, Ergodic dynamics and thermalization in an isolated quantum system, Nat. Phys. 12, 1037 (2016).
  * [290] A. Piga, M. Lewenstein, and J. Q. Quach, Quantum chaos and entanglement in ergodic and nonergodic systems, Phys. Rev. E 99, 032213 (2019).
  * [291] X. Wang, S. Ghose, B. C. Sanders, and B. Hu, Entanglement as a signature of quantum chaos, Phys. Rev. E 70, 016217 (2004).
  * [292] M. Lombardi and A. Matzkin, Entanglement and chaos in the kicked top, Phys. Rev. E 83, 016207 (2011).
  * [293] J. B. Ruebeck, J. Lin, and A. K. Pattanayak, Entanglement and its relationship to classical dynamics, Phys. Rev. E 95, 062222 (2017).
  * [294] A. Lerose and S. Pappalardi, Bridging entanglement dynamics and chaos in semiclassical systems, Phys. Rev. A 102, 032404 (2020)
  * [295] V. A. Marchenko, and L. A. Pastur, Distribution of eigenvalues for some sets of random matrices, Math USSR Sbornik 1, 457 (1967).
  * [296] J. N. Bandyopadhyay, and A. Lakshminarayan, Testing Statistical Bounds on Entanglement Using Quantum Chaos, Phys. Rev. Lett. 89, 060402 (2002).
  * [297] M. Serbyn, A. A. Michailidis, D. A. Abanin, and Z. Papić, Power-Law Entanglement Spectrum in Many-Body Localized Phases, Phys. Rev. Lett. 117, 160601 (2016).
  * [298] S. D. Geraedts, N. Regnault, and R. M. Nandkishore, Characterizing the many-body localization transition using the entanglement spectrum, New J. Phys. 19, 113021 (2017).
  * [299] E. Bairey, G. Refael, and N. H. Lindner, Driving induced many-body localization, Phys. Rev. B 96, 020201(R) (2017).
  * [300] P. Bordia, H. Lüschen, U. Schneider, M. Knap, and I. Bloch, Periodically driving a many-body localized quantum system, Nat. Phys. 13, 460 (2017).
  * [301] J. Keski-Rahkonen, A. Ruhanen, E. J. Heller, and E. Räsänen, Quantum Lissajous Scars, Phys. Rev. Lett. 123, 214101 (2019).
  * [302] D. Jaksch, J. I. Cirac, P. Zoller, S. L. Rolston, R. Côté, and M. D. Lukin, Fast Quantum Gates for Neutral Atoms, Phys. Rev. Lett. 85, 2208 (2000).
  * [303] M. D. Lukin, M. Fleischhauer, R. Côté, L. M. Duan, D. Jaksch, J. I. Cirac, and P. Zoller, Dipole Blockade and Quantum Information Processing in Mesoscopic Atomic Ensembles, Phys. Rev. Lett. 87, 037901 (2001).
  * [304] A. Barut, A. Bohm, and Y. Ne’eman, Dynamical Groups and Spectrum Generating Algebras (World Scientific Publishing Company, Singapore, 1988).
  * [305] S. Pilatowsky-Cameo, D. Villaseñor, M. A. Bastarrachea-Magnani, S. Lerma-Hernández, L. F. Santos, and J. G. Hirsch, Ubiquitous quantum scarring does not prevent ergodicity, Nat. Commun. 12, 852 (2021).
  * [306] B. Mukherjee, S. Nandy, A. Sen, and K. Sengupta, Collapse and revival of quantum many-body scars via Floquet engineering, Phys. Rev. B 101, 245107 (2020).
  * [307] K. Mizuta, K. Takasan, and N. Kawakami, Exact Floquet quantum many-body scars under Rydberg blockade, Phys. Rev. Research 2, 033284 (2020).
  * [308] S. Sugiura, T. Kuwahara, and K. Saito, Many-body scar state intrinsic to periodically driven system, Phys. Rev. Research 3, L012010 (2021).
  * [309] A. Hudomal, J.-Y. Desaules, B. Mukherjee, G.-X. Su, J. C. Halimeh, and Z. Papić, Driving quantum many-body scars in the PXP model, Phys. Rev. B 106, 104302 (2022).
  * [310] B. Huang, T.-H. Leung, D. M. Stamper-Kurn, and W. V. Liu, Discrete Time Crystals Enforced by Floquet-Bloch Scars, Phys. Rev. Lett. 129, 133001 (2022).
  * [311] A. Haldar, D. Sen, R. Moessner, and A. Das, Dynamical Freezing and Scar Points in Strongly Driven Floquet Matter: Resonance vs Emergent Conservation Laws, Phys. Rev. X 11, 021008 (2021).
  * [312] A. Polkovnikov, K. Sengupta, A. Silva, and M. Vengalattore, Colloquium: Nonequilibrium Dynamics of Closed Interacting Quantum Systems, Rev. Mod. Phys. 83, 863 (2011).
  * [313] T. Langen, R. Geiger, and J. Schmiedmayer, Ultracold Atoms Out of Equilibrium, Annu. Rev. Condens. Matter Phys. 6, 201 (2015).
  * [314] R. Gati, B. Hemmerling, J. Fölling, M. Albiez, and M. K. Oberthaler, Noise Thermometry with Two Weakly Coupled Bose-Einstein Condensates, Phys. Rev. Lett. 96, 130404 (2006).
  * [315] S. Genway, W. Li, C. Ates, B. P. Lanyon, and I. Lesanovsky, Generalized Dicke Nonequilibrium Dynamics in Trapped Ions, Phys. Rev. Lett. 112, 023603 (2014).
  * [316] N. Shammah, S. Ahmed, N. Lambert, S. De Liberato, and F. Nori, Open quantum systems with local and collective incoherent processes: Efficient numerical simulations using permutational invariance, Phys. Rev. A 98, 063815 (2018).
  * [317] K. C. Stitely, A. Giraldo, B. Krauskopf, and S. Parkins, Nonlinear semiclassical dynamics of the unbalanced, open Dicke model, Phys. Rev. Research 2, 033131 (2020).
  * [318] S. Ray, A. Vardi, and D. Cohen, Quantum Signatures in Quench from Chaos to Superradiance, Phys. Rev. Lett. 128, 130604 (2022).
  * [319] L. Sá, P. Ribeiro, and T. Prosen, Complex Spacing Ratios: A Signature of Dissipative Quantum Chaos, Phys. Rev. X 10, 021019 (2020).
  * [320] G. Akemann, M. Kieburg, A. Mielke, and T. Prosen, Universal Signature from Integrability to Chaos in Dissipative Open Quantum Systems, Phys. Rev. Lett. 123, 254101 (2019).
  * [321] K. Klobas, B. Bertini, and L. Piroli, Exact Thermalization Dynamics in the “Rule 54” Quantum Cellular Automaton, Phys. Rev. Lett. 126, 160602 (2021).
  * [322] Z. Gong, L. Piroli, and J. I. Cirac, Topological Lower Bound on Quantum Chaos by Entanglement Growth, Phys. Rev. Lett. 126, 160601 (2021).
  * [323] P. W. Claeys and A. Lamacraft, Ergodic and Nonergodic Dual-Unitary Quantum Circuits with Arbitrary Local Hilbert Space Dimension, Phys. Rev. Lett. 126, 100603 (2021).
  * [324] M. P. A. Fisher, V. Khemani, A. Nahum, and S. Vijay, Random quantum circuits, Ann. Rev. Cond. Matter Phys. 14, 335 (2022).
  * [325] L. K. Joshi, A. Elben, A. Vikram, B. Vermersch, V. Galitski, and P. Zoller, Probing Many-Body Quantum Chaos with Quantum Simulators, Phys. Rev. X 12, 011018 (2022).