# How to Leverage Predictive Uncertainty Estimates for
Reducing Catastrophic Forgetting in Online Continual Learning

Giuseppe Serra<EMAIL_ADDRESS>Goethe University Frankfurt
German Cancer Consortium (DKTK) ,  Ben Werner<EMAIL_ADDRESS>Goethe University Frankfurt and  Florian Buettner  florian.buettner@dkfz-
heidelberg.de Goethe University Frankfurt
German Cancer Consortium (DKTK)
German Cancer Research Center

###### Abstract.

Many real-world applications require machine-learning models to be able to
deal with non-stationary data distributions and thus learn autonomously over
an extended period of time, often in an online setting. One of the main
challenges in this scenario is the so-called catastrophic forgetting (CF) for
which the learning model tends to focus on the most recent tasks while
experiencing predictive degradation on older ones. In the online setting, the
most effective solutions employ a fixed-size memory buffer to store old
samples used for replay when training on new tasks. Many approaches have been
presented to tackle this problem. However, it is not clear how predictive
uncertainty information for memory management can be leveraged in the most
effective manner and conflicting strategies are proposed to populate the
memory. Are the easiest-to-forget or the easiest-to-remember samples more
effective in combating CF? Starting from the intuition that predictive
uncertainty provides an idea of the samples’ location in the decision space,
this work presents an in-depth analysis of different uncertainty estimates and
strategies for populating the memory. The investigation provides a better
understanding of the characteristics data points should have for alleviating
CF. Then, we propose an alternative method for estimating predictive
uncertainty via the generalised variance induced by the negative log-
likelihood. Finally, we demonstrate that the use of predictive uncertainty
measures helps in reducing CF in different settings.

Uncertainty, Online Continual Learning, Catastrophic Forgetting

††conference: 3rd Workshop on Uncertainty Reasoning and Quantification in
Decision Making; August 26, 2024; Barcelona, Spain

##  1\. Introduction

Typical machine learning models assume to work in a single-task static
scenario where multiple epochs are performed over the same data until
convergence. In many real-world situations, however, this setting is too
limiting. As an example, let us consider the problem of classifying new
COVID-19 variants. Given the evolving nature of the virus, new variants (i.e.,
new classes) arise over time. In this context, a typical learning model would
fail since the standard setting does not consider the dynamic increment of new
classes. For this reason, online Continual Learning (online-CL) has been
constantly more explored. In online-CL, a single model is required to learn
continuously from a sequence of tasks that comes as a stream of tiny batches
which can be processed only once (Aljundi et al., 2019, 2017; Mai et al.,
2022). Given the overlap between old and new information, the model tends to
forget about the past knowledge to focus more on the newest tasks, leading to
a performance degradation on previous tasks. This challenge is usually
referred to as catastrophic forgetting (CF) (McCloskey and Cohen, 1989;
Ratcliff, 1990). Many approaches have been developed to prevent catastrophic
forgetting and the most successful ones employ a memory buffer for rehearsal
(Chaudhry et al., 2019; Soutif-Cormerais et al., 2023). Memory-based
strategies tackle CF by training the model on both current samples and some
old samples stored in a limited size memory. In this context, what
differentiates each memory-based approach are the retrieval strategy and the
update strategy (Mai et al., 2022) — i.e., how to populate and update the
memory with meaningful and representative samples, and how to efficiently
sample from the memory respectively. Although many approaches have been
developed in this direction – ranging from using random approaches (Chaudhry
et al., 2019) to exploiting the gradient (Lopez-Paz and Ranzato, 2017;
Chaudhry et al., 2018b; Aljundi et al., 2019) or the loss (Belilovsky et al.,
2019) information – the effect of predictive uncertainty estimates for memory
management is largely unexplored. One of the few methods in this direction is
presented in Bang et al. (2021), where the authors propose a combination of
predictive uncertainty and data augmentation at the memory-level to generate a
memory of diverse samples – from the most uncertain to the most representative
ones – for each class. Despite remarkable results, the contribution of the
uncertainty score on the overall performance is not easily quantifiable.
Furthermore, considering the quickly expanding landscape of recently
rehearsal-based methods proposed, it is difficult to identify a clear strategy
to exploit the memory at its best. In fact, contrasting strategies can be
found in the literature. Some of them suggest to use the most representative
samples (Yoon et al., 2021; Hurtado et al., 2023), while others consider the
samples near the decision boundary as the most useful to reduce CF (Kumari et
al., 2022). Hence, it is unclear which type of data points would reduce CF in
a consistent manner: Are the easiest-to-forget or the easiest-to-remember
samples more effective in combating CF? To answer these and other questions,
this paper investigates the contribution of predictive uncertainty scores on
reducing CF. Intuitively, using predictive uncertainty to populate the memory
represents a solution for identifying the location of the instances in the
decision space and considering either the most representative – i.e., the k
samples with the lowest uncertainty (bottom-k) – or the marginal ones (i.e.,
the top-k with the highest uncertainty). In the first part of this work, we
evaluate and compare different combinations of uncertainty scores and sorting
on CIFAR-10 (Krizhevsky et al., 2009), a dataset commonly used in CL. The
objective is to understand how different uncertainty estimates behave when
used in different ways under same conditions. In order to achieve the intended
goal, following a recent trend on research transparency and comparability
(Mundt et al., 2021), the evaluation framework will be freed from any other
’trick’ that could create ambiguity in the assessment. The investigation will
focus on describing the characteristics of the considered uncertainty scores,
while providing an in-depth analysis of the effect of the metrics under a
memory-based regime. In addition to the comparison of established uncertainty
scores, we propose a new selection strategy based on recent advances in
uncertainty quantification via the logit space. More specifically, we
introduce a memory management strategy based on the Bregman Information (BI)
as a generalised variance measure, which stems directly from a bias-variance
decomposition of the model loss (Gruber and Buettner, 2023).
In the second part of our work, we evaluate our findings on more challenging
and realistic scenarios. In realistic setups, recent tasks have less data
points available than older tasks as the time to collect instances is
considerably shorter than for previous tasks. Considering this behaviour, we
will focus on long-tailed (LT) datasets with this characteristic. We test our
findings on two datasets with controllable degrees of data imbalance, as well
as a real-world imbalanced dataset for classification of biomedical images
(Yang et al., 2023).

The main contributions of this paper are the followings:

  * •

We systematically evaluate the performance of established predictive
uncertainty scores in reducing CF in online-CL for image classification.

  * •

We propose an alternative estimate for predictive uncertainty via the
generalised variance induced by the loss function.

  * •

We demonstrate that choosing the most representative samples (bottom-k) to
populate the memory consistently reduces CF compared to choosing marginal
samples (top-k) – see Figure 1.

  * •

We validate our findings in realistic scenarios with data imbalance, such as
medical image classification.

Figure 1. Relative CF improvement of different uncertainty scores over ER when
using the easiest-to-forget (top-k) and the easiest-to-remember (bottom-k)
samples.

##  2\. Related Work

We can identify three different scenarios of incremental learning (IL) (van de
Ven et al., 2022); task-, domain-, and class-IL. In task-IL, task IDs are
explicitly provided along with the data during training and testing. In
domain-IL, the problem setting remains unchanged over time but the input
distribution (or domain) shifts. Finally, in the class-IL scenario, the model
learns to discriminate across a growing number of classes over time. An
additional categorization of CL problems considers whether the data can be
stored and seen multiple times (offline) or the data are considered as a
stream of data which can only be used once (online) (Mai et al., 2022). In
this work, we focus on the most challenging and realistic scenario, namely,
online class-IL.

In online-CL, where data arrive in single mini-batches and the model is
updated with high frequency, memory-based methods are preferred to more
sophisticated solutions, e.g., generative methods, for their flexibility and
reduced training time (Mai et al., 2022). In this context, the main question
is how to optimally manage the memory. In order to reduce catastrophic
forgetting, samples in the memory should be representative of their own class,
discriminative towards the other classes, and informative enough for the model
to recall the information about the old classes. In the literature, many
conflicting strategies can be found. Some of them suggest to use the most
representative samples (Yoon et al., 2021; Hurtado et al., 2023), while others
consider the samples near the decision boundary as the most useful to reduce
CF (Kumari et al., 2022).

As already anticipated, Rainbow Memory (RM) (Bang et al., 2021) is the only
approach that makes explicit use of uncertainty measures to evaluate the
relative position of perturbed samples in the decision space. Given a list $P$
of perturbations, the authors propose to use an agreement score $u_{RM}(x)$
such that, if all the perturbed versions of $x$ are predicted with the same
label, then $u_{RM}(x)$ will be 0. Otherwise, the higher is $u_{RM}(x)$ the
most uncertain is the model about the considered sample.

Noticeable, the definition of online setting in their paper differs from the
popular one. In their case, online means that the training stream is only
processed once and the memory is updated at the end of each task. In the
standard scenario, instead, online implies that only a mini-batch of data
points is available to be processed (Mai et al., 2022), and the model and the
memory are updated with high frequency (Soutif-Cormerais et al., 2023). In
addition to this, to further enhance diversity, they employ data augmentation
on the memory. Thus, the contribution of $u_{RM}$ on reducing CF is unclear.

In the remainder of this work, we systematically investigate the effect of
established uncertainty scores - including $u_{RM}$ \- as well as the newly
proposed Bregman Information on combating catastrophic forgetting in a
realistic online-CL setting.

##  3\. Preliminaries and Notation

Following the notation proposed in Bang et al. (2021), we assume to have a set
$\bm{\mathcal{C}}=\left\\{c_{1},\dots,c_{n}\right\\}$ of n different classes.
Each class can be randomly assigned to a task $t$. $T_{t}$ represents a subset
of classes determined by an assign function $\psi(c)$ such that
$T_{t}=\\{c|\psi(c)=t\\}$. For each task $t$, we have an associated dataset
$D_{t}=\\{(x_{i},y_{i})\\}_{i=1}^{n_{t}}$ with $x_{i}$ an input sample,
$y_{i}$ the corresponding class label, and $n_{t}$ the number of training
samples. In the proposed online setting, we assume that samples for each task
$t$ come gradually in a stream of mini-batches $b_{t}=\\{{(x_{i},y_{i})\in
D_{t}\\}_{i=1}^{bs}}$ which can be only processed once. It is important to
notice that, for each task $t$, $n_{t}$ can vary depending on whether we are
working on a class-balanced setting or not. Finally, for replay, we introduce
a fixed-size memory buffer $\mathcal{M}$ to store a portion of samples from
past tasks. The memory is updated with high-frequency whenever a mini-batch
from the stream of data is processed. Following the standard procedure in
online-CL, for replay, we assume to extract from the memory a number of
samples equal in size to the batch size.

##  4\. Methodology

To facilitate a fair evaluation of the framework, we focus on the assessment
of the uncertainty metrics under same conditions. In this way, we eliminate
any ambiguous effect which could be inherited from other methodological
choices. As anticipated in Section 1, there are two main objectives when
dealing with memory-based approaches: 1) memory populating, and 2) memory
sampling.

In the following subsections, we describe the strategies for each of the two
objectives, and the uncertainty scores considered in our evaluation. Then, we
outline our uncertainty-aware memory management detailing the properties of
the employed uncertainty estimate and its advantages compared to standard
scores.

###  4.1. Memory Management

Similar to Chrysakis and Moens (2020), the memory population strategy is based
on a class-balanced update, which is crucial to consider in case the stream of
data is highly imbalanced. In fact, if classes are not equally represented in
the memory, sampling from the memory may further deteriorate the predictive
performance of the framework for under-represented classes. Differently from
the populating strategy proposed in Chrysakis and Moens (2020), the criteria
to decide which samples to keep in the memory is not random, but based on
predictive uncertainty estimates in order to intentionally store samples with
desired characteristics for each class. There are different ways to select the
samples. We can decide to store a) the class-representative data points by
selecting the least uncertain ones for each class (bottom-k); b) a diverse set
of data points, spanning from the most representative to the most uncertain
ones (step-size); and c) the easiest-to-forget samples by sampling the ones
with high uncertainty (top-k).

From the second task on, we need a strategy for sampling from the memory a
subset of data points used for replay (replay set). Since the memory
represents a prototypical set of data points for each class, we assume that a
random sampling is sufficient to extract informative data points from the
memory. Following the standard practice, the number of samples in the replay
set is equal to the batch size. It is important to note that the memory may
already contain samples from the current task. In such cases, samples from the
current task are excluded from the sampling process to ensure that the focus
remains solely on those belonging to past tasks.

Figure 2. Illustration of the different aspects of uncertainty captured by
model confidence scores and Bregman Information (BI). Close to the decision
boundary we can notice how, due to the high aleatoric uncertainty, data points
have low-confidence scores. Contrarily, due to high density of observed data,
there is a low uncertainty about the data generating process resulting in a
low BI.

###  4.2. Uncertainty in the Logit Space

Different measure of predictive uncertainty can capture distinct aspects of a
model’s irreducible aleatoric uncertainty (inherent in the data) and its
epistemic uncertainty (that stems from the finite training data and can be
reduced by gathering more data). In online CL, the most commonly used measures
are derived directly from the confidence scores of the model and mostly
capture the irreducible aleatoric uncertainty (Wimmer et al., 2023). Here, we
hypothesize it may be more beneficial for the model to replay instances which
are representative in the sense that there is low uncertainty about the data
generating process (we refer to this as low epistemic uncertainty, while
acknowledging that there exist varying definitions). To not rely on
specialized Bayesian models, we leverage a recently proposed bias-variance
decomposition of the cross-entropy loss and compute a generalized variance
term directly from the loss (Gruber and Buettner, 2023). Such bias-variance
decomposition decomposes the expected prediction error (loss) into the model’s
bias, variance, and an irreducible error (noise term). The latter is related
to aleatoric uncertainty, whereas the variance term can directly be related to
epistemic uncertainty (Gruber et al., 2023).
Gruber and Buettner (2023) have recently shown that a bias-variance
decomposition of cross-entropy loss gives rise to the Bregman Information as
the variance term and measures the variability of the prediction in the logit
space. We illustrate the different aspects of uncertainty captured by
confidence scores and BI respectively in Figure 2. For example, data points
close to the decision boundary have a low confidence score due to the
inherently high aleatoric uncertainty; in contrast, due to the high density of
observed data, there is actually a low uncertainty about the data generating
process (DGP), resulting in a low BI (low epistemic uncertainty). Outliers far
away from the decision boundary can have a high confidence score, but will
have a high BI due to the high uncertainty regarding the DGP. We hypothesize
that it is samples with a low BI that are most useful for replay in online-
FCL.
To populate the memory with representative samples, we therefore propose using
an uncertainty estimator based on Bregman Information (BI) (Gruber and
Buettner, 2023). The authors demonstrate that BI at the sample level can be
estimated through deep ensembles or test-time augmentation (TTA). However, to
reduce the computational overhead, we employ TTA for computing the
estimations. Let us consider the problem of multi-class classification (as in
our case), where the standard loss is represented by the cross-entropy.
Considering a set $P$ of perturbations for a given data point $x$, we can
compute the variance term of the classification loss $u(x)_{BI}$ as follows:

(1)
$u(x)_{BI}=\frac{1}{P}\sum_{i=1}^{P}\text{LSE}(\hat{z}_{i})-\text{LSE}\left(\frac{1}{P}\sum_{i=1}^{p}\hat{z}_{i}\right),$

where $\hat{z}_{i}\in\mathbb{R}^{c}$ represent the logit predictions and
$\text{LSE}(x_{1},\dots,x_{n})=\ln\sum_{i=1}^{n}e^{x_{i}}$ the LogSumExp (LSE)
function respectively. Intuitively, a large value of $u(x)_{BI}$ means that
the logits predicted across the perturbations vary significantly, suggesting a
high uncertainty of the prediction and the DGP at this point of the input
space. The use of this estimator is also motivated by the fact that, in
comparison with other uncertainty scores such as entropy, smallest margin, or
least confidence, there is no information loss in the estimation step. In
fact, if we inspect these alternative metrics, we can notice that they either
need a normalization step to move the logits in the probability space or rest
on the largest activation value only. Furthermore, as reported in Gruber and
Buettner (2023); Ovadia et al. (2019), and Tomani and Buettner (2021), common
confidence scores are reliable only in case of well-calibrated models.
In contrast, the BI-based estimation of the epistemic uncertainty is
meaningful also under distribution shift and able to identify robust and
representative samples (Gruber and Buettner, 2023).

##  5\. Experiments

###  5.1. Datasets and Settings

Datasets. To understand the change in the performance when using different
uncertainty strategies, we employ CIFAR-10 (Krizhevsky et al., 2009). To
configure the online-CIL setup, we randomly assign with different random seeds
a set of classes to 5 tasks. Thus, for each random seed, the class assignment
per task is different. In this way, we can identify the strategy providing
greater flexibility and effectiveness irrespective of the composition of the
tasks. Once the best strategy is identified, we evaluate our findings on
class-imbalanced scenarios to assess the performance under more realistic
conditions. First, we employ two artificially controlled imbalanced datasets,
CIFAR10-LT and CIFAR100-LT (Cao et al., 2019). As explained in Section 1, with
this experiment we want to replicate a realistic scenario in which recent
tasks contain less data than the older ones. For this, the most appropriate
type of imbalance is the long-tailed (LT) one (Cui et al., 2019) which follows
an exponential decay to choose the sample size of each class. Specifically,
for both datasets, we decided to use an imbalance factor $\rho$ equal to
$0.1$. This means that, if the largest class contains, e.g., $500$ data
points, then the smallest one consists of $50$ instances. Finally, we decide
to focus on biomedical image analysis using the microscopic peripheral blood
cell images dataset (BloodCell) which consists of 8 classes annotated by
expert clinical pathologists (Acevedo et al., 2020; Yang et al., 2023). To
reflect the intended realistic conditions, we assign 2 classes for each tasks
by increasing size. In this way, older tasks have more data points than the
most recent ones.

Experimental Settings. The baseline for assessing the goodness of the proposed
strategies will be Experience Replay (ER) (Chaudhry et al., 2019). Our
decision to use ER is based on the fact that, despite its simplicity, it is
surprisingly competitive when compared with more sophisticated and newer
approaches as shown in recently conducted empirical surveys (Soutif-Cormerais
et al., 2023).

In all the experiments, we employ a slim version of Resnet18 (He et al., 2016)
– as done in previous works (Hurtado et al., 2023; Lopez-Paz and Ranzato,
2017; Soutif-Cormerais et al., 2023; Kumari et al., 2022) –, and use the SGD
optimizer with a learning rate of $0.1$. Following standard practice, we set
the batch size equal to $10$. We set the memory size to different values for
each dataset to evaluate a variety of memory configurations considering both
large or small buffers. For evaluation, we employ last accuracy (A) and last
forgetting (F) defined in Chaudhry et al. (2018a). The compared uncertainty
scores are least confidence (LC), smallest margin (SM), ratio of confidence
(RC), entropy (EN), and the rainbow memory agreement score (RM) (Shannon,
1948; Campbell et al., 2000; Lewis and Gale, 1994; Culotta and McCallum, 2005;
Bang et al., 2021). All the experiments are run on three different random
seeds.

Table 1. Comparison of last accuracy (A) and last forgetting (F) on CIFAR10.

Score |  | M=500 | M=1000
---|---|---|---
ER |  | 26.38 $\pm$ 2.27 | 36.50 $\pm$ 2.16
| 74.92 $\pm$ 4.93 | 57.42 $\pm$ 5.93
|  | Top | Step | Bottom | Top | Step | Bottom
LC | A$(\uparrow)$ |  25.27 $\pm$ 0.93 |  30.51 $\pm$ 3.32 |  34.49 $\pm$ 2.91 |  30.70 $\pm$ 2.22 |  38.03 $\pm$ 2.98 |  38.30 $\pm$ 2.98
F$(\downarrow)$ |  72.88 $\pm$ 2.76 |  58.07 $\pm$ 6.23 |  53.11 $\pm$ 5.85 |  64.48 $\pm$ 4.09 |  48.04 $\pm$ 6.54 |  41.71 $\pm$ 5.68
SM | A$(\uparrow)$ |  26.33 $\pm$ 2.07 |  30.16 $\pm$ 0.99 |  32.50 $\pm$ 2.31 |  33.29 $\pm$ 1.41 |  38.85 $\pm$ 4.31 |  39.36 $\pm$ 1.39
F$(\downarrow)$ |  71.08 $\pm$ 5.00 |  61.19 $\pm$ 5.74 |  56.03 $\pm$ 4.50 |  59.75 $\pm$ 1.87 |  48.89 $\pm$ 8.57 |  43.84 $\pm$ 3.42
RC | A$(\uparrow)$ |  27.84 $\pm$ 2.33 |  30.57 $\pm$ 2.43 |  32.49 $\pm$ 3.86 |  36.35 $\pm$ 1.35 |  37.68 $\pm$ 1.17 |  39.63 $\pm$ 0.80
F$(\downarrow)$ |  66.13 $\pm$ 7.78 |  59.70 $\pm$ 6.71 |  54.73 $\pm$ 7.66 |  54.53 $\pm$ 4.54 |  48.08 $\pm$ 3.69 |  41.9 $\pm$ 3.53
EN | A$(\uparrow)$ |  23.72 $\pm$ 1.50 |  28.76 $\pm$ 1.76 |  35.20 $\pm$ 1.52 |  26.57 $\pm$ 0.98 |  36.13 $\pm$ 1.56 |  35.78 $\pm$ 4.99
F$(\downarrow)$ |  77.04 $\pm$ 3.78 |  61.20 $\pm$ 3.33 |  46.12 $\pm$ 3.02 |  71.66 $\pm$ 3.78 |  50.37 $\pm$ 3.44 |  42.65 $\pm$ 8.07
RM | A$(\uparrow)$ |  26.92 $\pm$ 1.75 |  28.57 $\pm$ 0.80 |  35.71 $\pm$ 4.12 |  32.87 $\pm$ 2.56 |  38.23 $\pm$ 0.95 |  38.57 $\pm$ 2.09
F$(\downarrow)$ |  71.45 $\pm$ 2.92 |  61.16 $\pm$ 4.40 |  51.01 $\pm$ 6.60 |  62.01 $\pm$ 4.36 |  47.21 $\pm$ 3.68 |  42.59 $\pm$ 2.79
BI | A$(\uparrow)$ |  26.41 $\pm$ 1.17 |  28.88 $\pm$ 1.96 |  34.10 $\pm$ 6.24 |  28.62 $\pm$ 2.14 |  37.44 $\pm$ 2.87 |  40.44 $\pm$ 3.09
F$(\downarrow)$ |  72.68 $\pm$ 3.81 |  61.21 $\pm$ 6.12 |  43.03 $\pm$ 5.65 |  69.02 $\pm$ 4.78 |  47.68 $\pm$ 5.96 |  37.79 $\pm$ 6.07

###  5.2. Empirical Results

Table 1 shows the results for all the uncertainty metrics and sorting
strategies employed for populating the memory. From the values reported in the
tables, we can observe that selecting the most representative samples for each
class (’Bottom’ column) consistently improves the results in terms of both
accuracy (A) and forgetting (F) across all the considered uncertainty scores
and memory sizes. Thus, our results indicate that the easiest-to-remember
samples are beneficial in improving the predictive performance of the learning
model and in reducing considerably CF.
To provide a graphical intuition of the differences between the sorting
strategies and their plausible effect on the learning capabilities of the
model, Figure 3 depicts a subset of class-specific samples stored in the
memory at the end of the training for the two opposite population strategies,
i.e. bottom-k and top-k. By inspecting the images, we note that the ones
selected via the bottom-k strategy show some consistency and are easily
identifiable as birds. On the opposite side, the top-k samples present images
with a different perspective, zooming, or background making it more difficult,
even for the human eye, to identify a bird. For this reason, we believe that
the easiest-to-remember images are more useful to the model for recalling the
past information while the easiest-to-forget data points may hamper the
generalization of the model – since we could be in the presence of noisy
images or outliers.
The comparison among the different uncertainty strategies suggests that BI
outperforms the other metrics in most cases. While the difference among the
accuracy values is less pronounced, BI stands out in its effectiveness at
mitigating CF. Finally, in Table 2 we evaluate our findings in the more
realistic long-tailed data imbalance scenario. In both cases (artificial and
real data imbalance), the results support our findings from the standard
balanced setting: BI substantially reduces CF while delivering competitive or
improved predictive performance.

Figure 3. Class-specific samples stored in the memory with respect to
different sorting strategies. Table 2. Comparison of last accuracy (A) and
last forgetting (F) on long-tailed imbalanced datasets.

Score |  | CIFAR10-LT | CIFAR100-LT | BloodCell
---|---|---|---|---
|  | M=200 | M=500 | M=500 | M=1000 | M=40 | M=160
ER | A$(\uparrow)$ |  22.06 $\pm$ 1.13 |  31.62 $\pm$ 1.65 |  8.22 $\pm$ 0.36 |  9.3 $\pm$ 0.22 |  58.82 $\pm$ 6.62 |  64.08 $\pm$ 11.98
F$(\downarrow)$ |  76.19 $\pm$ 3.17 |  52.09 $\pm$ 3.21 |  20.16 $\pm$ 0.9 |  15.78 $\pm$ 0.62 |  49.60 $\pm$ 8.23 |  28.50 $\pm$ 5.75
LC | A$(\uparrow)$ |  31.16 $\pm$ 1.24 |  32.5 $\pm$ 1.05 |  7.15 $\pm$ 0.73 |  8.93 $\pm$ 1.05 |  51.19 $\pm$ 5.49 |  66.08 $\pm$ 1.78
F$(\downarrow)$ |  42.31 $\pm$ 2.81 |  37.59 $\pm$ 2.01 |  17.53 $\pm$ 1.71 |  13.98 $\pm$ 2.66 |  27.33 $\pm$ 0.64 |  17.03 $\pm$ 3.05
SM | A$(\uparrow)$ |  29.62 $\pm$ 1.70 |  33.48 $\pm$ 1.23 |  7.20 $\pm$ 0.57 |  9.11 $\pm$ 0.66 |  55.28 $\pm$ 5.04 |  68.34 $\pm$ 3.38
F$(\downarrow)$ |  47.31 $\pm$ 2.64 |  39.46 $\pm$ 2.45 |  19.36 $\pm$ 1.79 |  14.43 $\pm$ 1.85 |  34.41 $\pm$ 5.90 |  14.21 $\pm$ 2.80
RC | A$(\uparrow)$ |  31.09 $\pm$ 0.79 |  32.48 $\pm$ 1.21 |  7.86 $\pm$ 0.45 |  9.47 $\pm$ 0.59 |  51.88 $\pm$ 3.59 |  63.92 $\pm$ 4.31
F$(\downarrow)$ |  44.38 $\pm$ 1.21 |  40.65 $\pm$ 1.75 |  18.51 $\pm$ 1.38 |  15.33 $\pm$ 2.04 |  28.43 $\pm$ 2.91 |  21.20 $\pm$ 7.00
EN | A$(\uparrow)$ |  31.34 $\pm$ 0.49 |  32.53 $\pm$ 1.89 |  7.02 $\pm$ 0.49 |  8.48 $\pm$ 0.71 |  49.32 $\pm$ 3.86 |  69.38 $\pm$ 1.13
F$(\downarrow)$ |  41.98 $\pm$ 2.06 |  32.70 $\pm$ 3.08 |  17.79 $\pm$ 2.18 |  14.34 $\pm$ 2.99 |  32.37 $\pm$ 2.22 |  11.41 $\pm$ 1.36
RM | A$(\uparrow)$ |  28.50 $\pm$ 2.04 |  33.98 $\pm$ 2.75 |  7.89 $\pm$ 0.90 |  9.9 $\pm$ 1.12 |  57.13 $\pm$ 3.18 |  67.97 $\pm$ 4.19
F$(\downarrow)$ |  51.20 $\pm$ 5.82 |  39.17 $\pm$ 3.95 |  18.36 $\pm$ 1.27 |  13.54 $\pm$ 1.64 |  31.93 $\pm$ 4.18 |  14.29 $\pm$ 7.72
BI | A$(\uparrow)$ |  30.35 $\pm$ 1.01 |  34.37 $\pm$ 1.26 |  8.93 $\pm$ 1.02 |  10.35 $\pm$ 0.58 |  58.08 $\pm$ 7.08 |  71.88 $\pm$ 2.33
F$(\downarrow)$ |  40.66 $\pm$ 1.06 |  34.93 $\pm$ 0.50 |  16.31 $\pm$ 1.48 |  13.27 $\pm$ 1.10 |  18.27 $\pm$ 7.25 |  8.71 $\pm$ 5.00

##  6\. Conclusion

In this paper, we propose to leverage estimates of predictive uncertainty for
memory management. The investigation focuses on understanding the
characteristics samples should have to alleviate CF. The results demonstrate
that the most representative samples – i.e., the easiest-to-remember – are the
most effective in mitigating CF while competitively performing in the
classification task. Additionally, the newly proposed uncertainty score for
memory management based on the Bregman Information shows its superiority in
reducing CF compared to well-established uncertainty scores. The results on
different realistic setups further support our findings and demonstrate that
the use of predictive uncertainty estimates for reducing CF in online-CL is
well-motivated.

###### Acknowledgements.

This work was supported by the The Federal Ministry for Economic Affairs and
Climate Action of Germany (BMWK, Project OpenFLAAS 01MD23001E). Co-funded by
the European Union (ERC, TAIPO, 101088594). Views and opinions expressed are
however those of the authors only and do not necessarily reflect those of the
European Union or the European Research Council. Neither the European Union
nor the granting authority can be held responsible for them.

## References

  * (1)
  * Acevedo et al. (2020) Andrea Acevedo, Anna Merino, Santiago Alférez, Ángel Molina, Laura Boldú, and José Rodellar. 2020.  A dataset of microscopic peripheral blood cell images for development of automatic recognition systems.  _Data in brief_ 30 (2020).
  * Aljundi et al. (2017) Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. 2017.  Expert gate: Lifelong learning with a network of experts. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 3366–3375.
  * Aljundi et al. (2019) Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. 2019.  Gradient based sample selection for online continual learning.  _Advances in neural information processing systems_ 32 (2019).
  * Bang et al. (2021) Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Ha, and Jonghyun Choi. 2021.  Rainbow memory: Continual learning with a memory of diverse samples. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 8218–8227.
  * Belilovsky et al. (2019) Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, Tinne Tuytelaars, Rahaf Aljundi, et al. 2019\.  Online continual learning with maximally interfered retrieval.  _Advances in Neural Information Processing Systems,(NeurIPS)_ (2019).
  * Campbell et al. (2000) Colin Campbell, Nello Cristianini, Alex Smola, et al. 2000\.  Query learning with large margin classifiers. In _ICML_ , Vol. 20. 0.
  * Cao et al. (2019) Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. 2019.  Learning imbalanced datasets with label-distribution-aware margin loss.  _Advances in neural information processing systems_ 32 (2019).
  * Chaudhry et al. (2018a) Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. 2018a.  Riemannian walk for incremental learning: Understanding forgetting and intransigence. In _Proceedings of the European conference on computer vision (ECCV)_. 532–547.
  * Chaudhry et al. (2018b) Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2018b.  Efficient Lifelong Learning with A-GEM. In _International Conference on Learning Representations_.
  * Chaudhry et al. (2019) Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, P Dokania, P Torr, and M Ranzato. 2019.  Continual learning with tiny episodic memories. In _Workshop on Multi-Task and Lifelong Reinforcement Learning_.
  * Chrysakis and Moens (2020) Aristotelis Chrysakis and Marie-Francine Moens. 2020.  Online continual learning from imbalanced data. In _International Conference on Machine Learning_. PMLR, 1952–1961.
  * Cui et al. (2019) Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. 2019.  Class-balanced loss based on effective number of samples. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 9268–9277.
  * Culotta and McCallum (2005) Aron Culotta and Andrew McCallum. 2005.  Reducing labeling effort for structured prediction tasks. In _AAAI_ , Vol. 5. 746–751.
  * Gruber et al. (2023) Cornelia Gruber, Patrick Oliver Schenk, Malte Schierholz, Frauke Kreuter, and Göran Kauermann. 2023.  Sources of Uncertainty in Machine Learning–A Statisticians’ View.  _arXiv preprint arXiv:2305.16703_ (2023).
  * Gruber and Buettner (2023) Sebastian Gruber and Florian Buettner. 2023.  Uncertainty Estimates of Predictions via a General Bias-Variance Decomposition. In _International Conference on Artificial Intelligence and Statistics_. PMLR, 11331–11354.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Hurtado et al. (2023) Julio Hurtado, Alain Raymond-Sáez, Vladimir Araujo, Vincenzo Lomonaco, Alvaro Soto, and Davide Bacciu. 2023.  Memory Population in Continual Learning via Outlier Elimination. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 3481–3490.
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al. 2009\.  Learning multiple layers of features from tiny images.  _online_ (2009).
  * Kumari et al. (2022) Lilly Kumari, Shengjie Wang, Tianyi Zhou, and Jeff A Bilmes. 2022.  Retrospective adversarial replay for continual learning.  _Advances in Neural Information Processing Systems_ 35 (2022), 28530–28544.
  * Lewis and Gale (1994) D Gale Lewis and William A Gale. 1994.  A sequential algorithm for training text classifiers. In _Proceedings of SIGIR-94_. 3–12.
  * Lopez-Paz and Ranzato (2017) David Lopez-Paz and Marc’Aurelio Ranzato. 2017.  Gradient episodic memory for continual learning.  _Advances in neural information processing systems_ 30 (2017).
  * Mai et al. (2022) Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. 2022.  Online continual learning in image classification: An empirical survey.  _Neurocomputing_ 469 (2022), 28–51.
  * McCloskey and Cohen (1989) Michael McCloskey and Neal J Cohen. 1989.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_. Vol. 24. Elsevier, 109–165.
  * Mundt et al. (2021) Martin Mundt, Steven Lang, Quentin Delfosse, and Kristian Kersting. 2021.  CLEVA-Compass: A Continual Learning Evaluation Assessment Compass to Promote Research Transparency and Comparability. In _International Conference on Learning Representations_.
  * Ovadia et al. (2019) Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. 2019.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  _Advances in neural information processing systems_ 32 (2019).
  * Ratcliff (1990) Roger Ratcliff. 1990.  Connectionist models of recognition memory: constraints imposed by learning and forgetting functions.  _Psychological review_ 97, 2 (1990).
  * Shannon (1948) Claude Elwood Shannon. 1948.  A mathematical theory of communication.  _The Bell system technical journal_ 27, 3 (1948), 379–423.
  * Soutif-Cormerais et al. (2023) Albin Soutif-Cormerais, Antonio Carta, Andrea Cossu, Julio Hurtado, Vincenzo Lomonaco, Joost Van de Weijer, and Hamed Hemati. 2023.  A comprehensive empirical evaluation on online continual learning. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 3518–3528.
  * Tomani and Buettner (2021) Christian Tomani and Florian Buettner. 2021.  Towards trustworthy predictions from deep neural networks with fast adversarial calibration. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 9886–9896.
  * van de Ven et al. (2022) Gido M van de Ven, Tinne Tuytelaars, and Andreas S Tolias. 2022.  Three types of incremental learning.  _Nature Machine Intelligence_ 4, 12 (2022), 1185–1197.
  * Wimmer et al. (2023) Lisa Wimmer, Yusuf Sale, Paul Hofman, Bernd Bischl, and Eyke Hüllermeier. 2023.  Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures?. In _Uncertainty in Artificial Intelligence_. PMLR, 2282–2292.
  * Yang et al. (2023) Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. 2023.  MedMNIST v2-A large-scale lightweight benchmark for 2D and 3D biomedical image classification.  _Scientific Data_ 10, 1 (2023).
  * Yoon et al. (2021) Jaehong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. 2021.  Online Coreset Selection for Rehearsal-based Continual Learning. In _International Conference on Learning Representations_.


# Region-Aware Pretraining for Open-Vocabulary Object Detection with
Vision Transformers

Dahun Kim Anelia Angelova Weicheng Kuo
Google Research, Brain Team
{mcahny, anelia<EMAIL_ADDRESS>
###### Abstract

We present Region-aware Open-vocabulary Vision Transformers (RO-ViT) – a
contrastive image-text pretraining recipe to bridge the gap between image-
level pretraining and open-vocabulary object detection. At the pretraining
phase, we propose to randomly crop and resize regions of positional embeddings
instead of using the whole image positional embeddings. This better matches
the use of positional embeddings at region-level in the detection finetuning
phase. In addition, we replace the common softmax cross entropy loss in
contrastive learning with focal loss to better learn the informative yet
difficult examples. Finally, we leverage recent advances in novel object
proposals to improve open-vocabulary detection finetuning. We evaluate our
full model on the LVIS and COCO open-vocabulary detection benchmarks and zero-
shot transfer. RO-ViT achieves a state-of-the-art 32.1 $AP_{r}$ on LVIS,
surpassing the best existing approach by +5.8 points in addition to
competitive zero-shot transfer detection. Surprisingly, RO-ViT improves the
image-level representation as well and achieves the state of the art on 9 out
of 12 metrics on COCO and Flickr image-text retrieval benchmarks,
outperforming competitive approaches with larger models.

##  1 Introduction

Figure 1: Existing vision-language models are designed for image-level tasks,
_e.g_., classification and retrieval. In this paper, we aim to enhance the
image-text pretraining for region-level downstream task: open-vocabulary
object detection. At pretraining, we propose to randomly crop and resize
regions of positional embeddings (PE) instead of using the whole image PE.
This better matches the use of PE at region-level in the detection finetuning
and results in better performance in detection and surprisingly also retrieval
task.

The ability to detect objects in the visual world is a hallmark of computer
vision and machine intelligence. It is key to enable many applications, e.g.
autonomous agents adapting to new environments with many novel objects, a
shopping system handling fine-grained user queries such as “fedora”, “bonnet”
outside of the training vocabulary. However, modern object detectors typically
rely on manual annotations of regions and class labels, which limits their
vocabulary size to an order of $10^{3}$ and it is prohibitively expensive to
scale up further. This is orders of magnitude smaller than the objects we
encounter in the visual world.

Recently, the open-vocabulary detection task (OVD) has been proposed to
overcome such limitation by leveraging abundant image-text pairs for training
and ingesting the text queries provided by users at test time [55]. By
treating the categories as text embeddings rather than discrete ids, open-
vocabulary detectors can flexibly predict a wide range of objects unseen
during the training time. Most existing works leverage image-text pre-training
which contains rich semantic knowledge of open-vocabulary concepts. Knowledge
distillation [20, 13], weak supervision [64], self-training [61, 42, 58], and
frozen model [30] have been proposed, and CNN backbones are most commonly
used. With the growing popularity of vision transformers in image
understanding [12, 33, 57], multimodal [2, 1, 48], and self-supervised tasks
[23, 5, 6], it is important to understand how to build capable open-vocabulary
detectors with vision transformers [12, 36].

To our best knowledge, all existing works assume pretrained Vision-Language
Models (VLM) are given, and develop adaptation or finetuning recipes to bridge
the gap between image-level pretraining and object-level finetuning [20, 13,
61, 58, 42]. Since the VLMs are designed for image-level tasks _e.g_.
classification, retrieval, the notion of objects/regions are not adequately
utilized in the pretraining process. We believe it would be beneficial for
open-vocabulary detection if we bake in locality information in the image-text
pretraining.

We present RO-ViT, a simple recipe to pretrain vision transformers in a
region-aware manner for open-vocabulary object detection. Standard pretraining
typically uses full-image positional embeddings, which does not generalize
well to detection tasks. Thus, we propose a novel positional embedding scheme
called “Cropped Positional Embedding” which better matches the use of region
crops in detection finetuning (see Fig. 1). In addition, we found it
beneficial to replace the softmax cross entropy loss with focal loss in
contrastive learning, which affords us more control to learn from harder and
more informative examples. Finally, we leverage recent advances in novel
object proposals [29] to improve open-vocabulary detection finetuning. The
motivation is that existing approaches often miss novel objects in the object
proposal stage because the proposals tend to overfit to the foreground
categories.

We evaluate the approach on the standard LVIS and COCO open-vocabulary
benchmarks. On LVIS, our best model achieves a state-of-the-art 32.1 APr at
the system level, surpassing the best existing approach by +5.8 APr. Compared
to the best existing ViT-based approach, ours outperforms by a healthy margin
of +6.5 APr. Our LVIS-trained model outperforms existing baselines on transfer
detection to Objects365 without re-training. Although not explicitly optimized
for retrieval, RO-ViT surprisingly achieves the state-of-the-art performance
on 9 out of 12 metrics in image-text retrieval benchmark and outperforms
strong baselines with standard positional embeddings, cross entropy loss, and
larger model capacity. We conduct ablations to confirm the benefits of each
proposed component. Visualization of the learnt positional embeddings shows
that our approach (CPE) leads to more symmetrical and diverse patterns than
the baseline. In summary:

  * •

We propose RO-ViT to bridge the positional embeddings in image-text
pretraining to open-vocabulary detection finetuning.

  * •

We show that image-text pretraining with focal loss is more effective than
existing softmax CE loss.

  * •

We improve the open-vocabulary detection finetuning recipe with novel object
proposals.

  * •

RO-ViT achieves the state of the art on the LVIS open-vocabulary detection
benchmark, and 9 out 12 metrics on COCO and Flickr image-text retrieval
benchmarks.

We hope these findings will facilitate the research community to further
explore open-vocabulary detection from the perspective of image-text
pretraining with potential benefits for both region-level and image-level
tasks.

##  2 Related Works

#### Learning open-vocabulary and zero-shot recognition.

Building general representation for open-vocabulary and zero-shot recognition
is a fundamental machine learning task. DeViSE [15] and ConSE [37] are
pioneering works to learn a shared image-text embedding space for zero-shot
recognition by convolutional networks. As image and language often co-occur on
the internet, the community has explored learning such representation from
various data sources including image tags [8, 11, 28], captions/text
descriptions [10, 24, 43, 46, 60], alt-texts [27, 38, 56], and image search
queries [40]. Recently, contrastive learning has become a popular paradigm to
learn from large image-text corpus due to its simplicity and scalability,
where increasing model size and data yield consistent improvement on open-
vocabulary classification benchmarks. The aforementioned works focus on image-
level recognition, whereas we study how to best tailor them for region-level
understanding.

#### Self-supervised representation learning for localization.

Due to the challenge to scale up labeling for localization tasks, many efforts
have been made to learn locality-sensitive representation in a self-supervised
manner. Existing approaches roughly fall into the contrastive or generative
paradigms. Contrastive approaches typically involve region or point-level
contrastive learning using sliding windows [50], object proposals [49, 25], or
point samples [3]. Most existing contrastive methods are CNN-based, while ViT
backbones have been popular recently with generative approaches [23, 5] and
has shown promise with contrastive approaches [6]. Masked image modeling is
commonly used to learn the ViT features for localization tasks. Even though
these self-supervised methods are well-suited for standard localization tasks,
they lack the image-text correspondence necessary for open-vocabulary
recognition. Thus, we study contrastive image-text representation learning for
open-vocabulary detection with ViTs.

#### Open-vocabulary object detection and segmentation.

Zero-shot detection was proposed to scale up detection models beyond their
limited training categories. Popular approaches accomplish this by learning
the alignment between region visual representation and category word
embeddings [4, 41, 9, 59] or hallucinating visual features with a generative
model [22, 65]. Without any visual knowledge of the novel categories, zero-
shot detectors tend to struggle with low performance. This motivates the open-
vocabulary detection benchmark [55] to bridge the gap between zero-shot and
fully-supervised detection.

With the rise of image-text pretraining, many works have explored adapting
these pretrained models to open-vocabulary detection and segmentation [20, 61,
17, 31, 62]. For example, ViLD [20] proposes to distill the image-text
knowledge into the detector. DetPro [13] improves ViLD by incorporating the
idea of prompt optimization [63]. Moreover, region-text self-training has been
shown effective on image caption data [61], classification data [42], or even
unlabeled data [58]. Weak supervision [64] and phrase grounding [32]
approaches have also been proposed. In terms of architecture, CNNs are most
commonly used but vision transformers have recently been adopted as well [36,
62]. While existing works typically assume image-text pretrained models are
given and focus on finetuning or adaptation strategies, our focus is to
improve the upstream image-text pretraining with vision transformers.

##  3 Method

Figure 2: RO-ViT framework overview. Our Region-aware Open-vocabulary Vision
Transformer (RO-ViT) attempts to bridge the gap between the image-level
vision-language model (VLM) pretraining and downstream open-vocabulary
detection. For the pretraining, we propose Cropped Positional Embedding (CPE)
which randomly crops and resizes a region of positional embeddings instead of
using the whole-image PE. In addition, we use focal loss instead of the common
softmax cross entropy loss for contrastive learning. The pretrained ViT
backbone is transferred to the downstream open-vocabulary detection by
replacing the global average pooling with detector heads. RO-ViT detector
takes the whole-image positional embeddings as input, and the detections are
used to crop out region embeddings from the ViT backbone features. The region
embeddings then match with the cached category embeddings to obtain the VLM
score $z$, which is combined with the detection score $p$ into the open-
vocabulary detection score $s^{\text{OVD}}$ (see Equation 1).

In this paper we address the problem of open-vocabulary detection. At training
time, the model has access to the detection labels of $C_{B}$ base categories,
but needs to detect objects from a set of $C_{N}$ novel categories at test
time. We follow existing works [20, 61, 13, 30] to leverage pretrained vision
and language models (VLM) [40]. Unlike existing works, we explore how to best
pretrain our own VLMs with vision transformers [12] for open-vocabulary
detection.

###  3.1 Preliminaries

#### Contrastive image-text pretraining.

Two-tower contrastive image-text learning involves an image encoder and a text
encoder [40, 27]. The text tower is typically transformer encoder, whereas the
image tower can be CNN-based or ViT as in our case. Given a set of image-text
corpus, the model learns to bring each image and its corresponding text
together and push other non-matching texts away in the embedding space. The
most common objective is softmax cross-entropy loss. The image/text embeddings
are typically obtained by taking the pre-pended class token embedding, self-
attention pooling, or average pooling. We use the global average pooling
followed by L2 normalization for both image and text embeddings.

#### Open-vocabulary object detector.

An open-vocabulary object detector is trained with the labels of $C_{B}$ base
categories, but needs to detect the union of base and novel categories
($C_{B}\cup C_{N}$) at test time. Most object detectors use K-way classifiers
because the number of categories are the same between train and test time. To
deal with additional categories at test time, the common practice is to
replace the conventional fixed-size classifier fully-connected layer with the
text embeddings of base categories [55, 20]. It is important that the text
embeddings come from the matching text encoder of the image encoder during
pretraining, so that the open-vocabulary knowledge would be preserved. We
represent the background category by a “background” phrase and the proposals
not matched to any annotations in $C_{B}$ are labeled as background.

During training, we compute the detection scores $p_{i}$ for each region $i$
as the cosine similarity between the RoI-Align feature (_i.e_., region
embedding) and the text embeddings of $C_{B}$, followed by a softmax. At test
time, we expand the text embeddings from $C_{B}$ to $C_{B}\cup C_{N}$ plus the
“background” category for open-vocabulary detection. We extract the VLM
embedding of region $i$ by RoI-Align on the output feature map of the ViT
backbone, and compute the VLM region scores $z_{i}$ as the cosine similarity
with the $C_{B}\cup C_{N}$ text embeddings. Similarly, the detection scores
$p_{i}$ are now computed with the $C_{B}\cup C_{N}$ text embeddings. The
combined open-vocabulary detection score ${s_{i}}^{\text{OVD}}$ is obtained by
geometric means [20, 30]:

${s_{i}}^{\text{OVD}}=\begin{cases}z_{i}^{(1-\alpha)}\cdot
p_{i}^{\alpha}&\text{if }i\in C_{B}\\\ z_{i}^{(1-\beta)}\cdot
p_{i}^{\beta}&\text{if }i\in C_{N}\end{cases}$ (1)

, where $\alpha,\beta\in[0,1]$ control the weights for base and novel
categories. The background score comes directly from the detection score
$p_{i}$, because the VLM score with “background” phrase tends to be not as
reliable.

We adopt Mask R-CNN heads in our detector and use class-agnostic box
regression and mask prediction heads following existing works [13, 20, 55, 61,
30]. Note that we use a contrastively pretrained ViT to initialize our
detector backbone, and adopt the simple feature pyramid and windowed attention
to handle higher-resolution image inputs as proposed by Li  _et al_. [33].

###  3.2 Region-Aware Image-Text Pretraining

Existing vision-language models are trained to match an image as a whole to a
text description [40, 27]. However, the pretraining is unaware of the
alignment between its region-level representations and text tokens, which is
essential to downstream open-vocabulary detection. We propose a novel Cropped
Positional Embeddings (CPE) to bridge this gap, and also find it beneficial to
learn from hard examples with a focal loss. Our improvements introduce no
extra parameters and minimal computation costs.

#### Cropped Positional Embedding (CPE).

The positional embeddings are important to transformers as they provide the
information of where each element in the set comes from. This information is
often useful for downstream recognition and localization tasks.

There is a mismatch between the way the positional embeddings are used in
existing contrastive pretraining approaches and open-vocabulary detection
finetuning. The pretraining approaches typically apply full-image positional
embeddings [40, 27] during training, and use the same positional embeddings
for downstream tasks, _e.g_., zero-shot recognition. However, the recognition
occurs at region-level for open-vocabulary detection finetuning, which
requires the full-image positional embeddings to generalize to regions that
they never see during the pretraining.

To bridge this gap, we propose Cropped Positional Embedding (CPE) (see center
of Figure 2). First, we up-sample the positional embeddings from the image
size typical for pretraining, _e.g_., 224 to that typical for detection tasks,
_e.g_., 1024. Then we randomly crop and resize a region from the up-sampled
positional embeddings and use that as the image-level positional embeddings
during pretraining. The regions are uniformly sampled from the normalized
coordinates as $x_{1}\thicksim\text{Uniform}(0,1)$,
$y_{1}\thicksim\text{Uniform}(0,1)$, $x_{2}\thicksim\text{Uniform}(x_{1},1)$,
$y_{2}\thicksim\text{Uniform}(y_{1},1)$, while keeping the crop scale ratio in
[0.1, 1.0].

Intuitively, this causes the model to view an image not as a full image in
itself, but as a region crop from some larger unknown image. This better
matches the downstream use case of detection where recognition occurs at
region- rather than image-level.

#### Focal Loss.

We desire to have finer control over how hard examples are weighted than what
the softmax cross entropy loss can provide. Focal loss offers a natural option
to tune the weights of hard examples [34] . Let $v_{i}$ and $l_{i}$ be the
normalized image and text embeddings, and the image-to-text (I2T) contrastive
losses be $L_{\text{softmax}}$ and $L_{\text{focal}}$ for the softmax
(baseline) or focal loss (RO-ViT). We define the losses mathematically below:

$L_{\text{softmax}}=-{1\over{B}}\sum_{i=1}^{B}\log({\text{exp}(v_{i}l_{i}/\tau)\over{\sum_{j=1}^{B}\text{exp}(v_{i}l_{j}/\tau)}})$
(2)

$L_{\text{focal}}=-{1\over{B}}\sum_{i=1}^{B}\sum_{j=1}^{B}(1-p_{i})^{\gamma}\text{log}(p_{i})$
(3)

, where $p_{i}$ denotes the true class probability as below:

$p_{i}=\begin{cases}\sigma(v_{i}l_{j}/\tau)&\text{if }i=j\\\
1-\sigma(v_{i}l_{j}/\tau)&\text{if }i\neq j\end{cases}$ (4)

Here $\sigma$ denotes the sigmoid function. We adopt the simpler non alpha-
balanced form of focal loss [34]. The text-to-image (T2I) contrastive losses
are symmetrical with the I2T losses by simply exchanging the inner/outer
summation loops. The total loss is the sum of both I2T and T2I losses.

###  3.3 Open-vocabulary Detector Finetuning

Here we present two simple techniques to improve the downstream open-
vocabulary detector (Sec. 3.1). Despite the backbone features pretrained from
the vast open-vocabulary data, the added detector layers (neck and heads) are
newly trained with the downstream detection dataset (_e.g_., LVIS base
categories). Existing approaches often miss novel/unlabeled objects in the
object proposal stage because the proposals tend to classify them as
background. To remedy this, we leverage recent advances in novel object
proposal method [29] and adopt the localization quality-based objectness
(_i.e_., centerness score) instead of object-or-not binary classification
score. Following [29], we use a single anchor per location and combine the
predicted objectness score $o_{i}$ with the ensemble detection score in
Equation 1 to obtain the final OVD score as:
${S_{i}}^{\text{OVD}}={o_{i}}^{\delta}\cdot{s_{i}}^{\text{OVD}.}$.

method | pretrained | detector | APr | AP
---|---|---|---|---
model | backbone
ConvNet based: |  |  |  |
DetPro-Cascade [13] | ViT-B/32 | R-50 | 20.0 | 27.0
Detic-CN2 [64] | ViT-B/32 | R-50 | 24.6 | 32.4
RegionCLIP [61] | R-50x4 | R-50x4 | 22.0 | 32.3
ViLD-Ens [20] | ViT-B/32 | R-152 | 18.7 | 26.0
ViLD-Ens [20] | ViT-L/14 | EffNet-B7 | 21.7 | 29.6
ViLD-Ens [20] | EffNet-B7 | EffNet-B7 | 26.3 | 29.3
VL-PLM [58] | ViT-B/32 | R-50 | 17.2 | 27.0
OV-DETR [54] | ViT-B/32 | R-50 | 17.4 | 26.6
Rasheed  _et al_. [42] | ViT-B/32 | R-50 | 21.1 | 25.9
PromptDet [14] | ViT-B/32 | R-50 | 21.4 | 25.3
ViT based: |  |  |  |
OWL-ViT [36] | ViT-H/14 | ViT-H/14 | 23.3 | 35.3
OWL-ViT [36] | ViT-L/14 | ViT-L/14 | 25.6 | 34.7
RO-ViT (ours) | ViT-B/16 | ViT-B/16 | 28.0 | 30.2
RO-ViT (ours) | ViT-L/14 |  ViT-L/14$\dagger$ | 31.4 | 34.0
RO-ViT (ours) | ViT-L/16 | ViT-L/16 | 32.1 | 34.0
Table 1: LVIS open-vocabulary object detection (mask APs). RO-ViT outperforms
the best existing approach by +5.8 APr on novel categories. When using ViT-
Base and Large RO-ViT outperforms OWL-ViT based on ViT-Large by +3.0 and +6.5
APr, respectively. All methods use the same instance-level supervision from
LVIS base categories for detection training. $\dagger$: We use image size 860
during detection for a fair comparison with OWL-ViT.

Additionally, we replace the standard classifier and mask output layer with
the normalized layers [47]. This L2-normalizes the weights $w$ and features
$x$ as: $f(x;w,b,\tau)={\tau\over\|w\|_{2}\|x\|_{2}}w^{T}x+b$, where $\tau$ =
20. Although we do not have rare categories at training (_i.e_., open-
vocabulary setting), we empirically found it beneficial (see Sec. 4).

##  4 Experimental Results

method | pretrained | detector | novel AP | AP
---|---|---|---|---
model | backbone
ConvNet based: |  |  |  |
ViLD [20] | ViT-B/32 | R-50 | 27.6 | 51.3
OV-DETR [54] | ViT-B/32 | R-50 | 29.4 | 52.7
w/ pseudo box labels: |  |  |  |
XPM  _et al_. [26] | R-50 | R-50 | 27.0 | 41.2
RegionCLIP [61] $\dagger$ | R-50x4 | R-50x4 | 39.3 | 55.7
PromptDet [14] | ViT-B/32 | R-50 | 26.6 | 50.6
VL-PLM [58] | ViT-B/32 | R-50 | 34.4 | 53.5
Rasheed  _et al_. [42] $\ddagger$ | ViT-B/32 | R-50 | 36.9 | 51.5
w/ weak supervision: |  |  |  |
Detic-CN2 [64] | ViT-B/32 | R-50 | 24.6 | 32.4
ViT based:* |  |  |  |
RO-ViT (ours) | ViT-B/16 | ViT-B/16 | 30.2 | 41.5
RO-ViT (ours) | ViT-L/16 | ViT-L/16 | 33.0 | 47.7
Table 2: COCO open-vocabulary object detection (box AP50). RO-ViT represents
the first ViT-based approach and demonstrates a very competitive novel AP
without using pseudo labeling or weak supervision. $\dagger$: RegionCLIP uses
an off-the-shelf RPN during its pretraining. $\ddagger$: Rasheed  _et al_.
uses an external MViT detector [35] during pretraining. *: The other ViT-based
method [36] report their results on LVIS only.

#### Pretraining details.

Our contrastive image-text pretraining is performed from scratch. We adopt the
widely used ViT-B/16 and ViT-L/16 as the image encoder. The input image size
is 224$\times$224 which results in 14$\times$14 positional embeddings with
patch size 16$\times$16\. To generate the Cropped Positional Embedding (CPE),
we first interpolate the positional embeddings to size 64$\times$64\. We
randomly crop a region with the scale ratio in [0.1, 1.0], and the aspect
ratio in [0.5, 2.0]. The region crop is resized back to the size 14$\times$14
(_i.e_., CPE), and is added to the patch embeddings. We use the global average
pooling at the last ViT layer to obtain the image embedding. The text encoder
is a 12-layer Transformer following [40, 52], and the maximum length of the
input text as 64 tokens. Both image and text embeddings are L2 normalized and
trained with focal loss. We train on the same image-text dataset as Chao  _et
al_. [27]. Unless otherwise specified, we use a batch size of 4096 for
ablation and 16384 to compare with other methods. We use the AdamW optimizer
with learning rate of 5e-4 and a linear warmup of 10k steps, and train for
500k iterations.

#### Downstream detection details.

We train RO-ViT with base categories $C_{B}$ for 46.1k/11.3k iterations with
image size 1024, large scale jittering [16], batch size 256/128, the SGD
optimizer with weight decay 1e-4/1e-2, momentum 0.9 and an initial learning
rate of 0.36/0.02 for LVIS/COCO datasets. The pretrained positional embeddings
are bilinearly interpolated to adjust to the size of patch embeddings of
higher resolution [12]. We set the backbone learning rate lower (_e.g_.,
0.1$\times$) than the rest of the model to retain the pretraining knowledge
during detection finetuning. We use the score combination of
($\alpha,\beta,\delta$) = (0.65, 0.3, 3) in Sec. 3.3. We use CLIP [40] prompt
templates and take the average text embeddings of each category. We use OLN-
RPN [29] at the RPN stage which uses centerness as objectness, a single anchor
per location and IoU loss. The RPN NMS threshold is set to 0.7 at training and
1.0 at testing.

| image | MS COCO (5K test set) | Flickr30K (1K test set)
---|---|---|---
| backbone | ——-image-to-text——- | ——-text-to-image——- | ——-image-to-text——- | ——-text-to-image——-
method | size | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R10 | R@1 | R@5 | R@10
CLIP [40] | 302M | 58.4 | 81.5 | 88.1 | 37.8 | 62.4 | 72.2 | 88.0 | 98.7 | 99.4 | 68.7 | 90.6 | 95.2
ALIGN [27] | 408M | 58.6 | 83.0 | 89.7 | 45.6 | 69.8 | 78.6 | 88.6 | 98.7 | 99.7 | 75.7 | 93.8 | 96.8
FLAVA [45] | 86M | 42.7 | 76.8 | - | 38.4 | 67.5 | - | 67.7 | 94.0 | - | 65.2 | 89.4 | -
FILIP [51] | 302M | 61.3 | 84.3 | 90.4 | 45.9 | 70.6 | 79.3 | 89.8 | 99.2 | 99.8 | 75.0 | 93.4 | 96.3
Florence [53] | 637M | 64.7 | 85.9 | - | 47.2 | 71.4 | - | 90.9 | 99.1 | - | 76.7 | 93.6 | -
CoCa-Large [52] | 303M | 65.4 | 85.6 | 91.4 | 50.1 | 73.8 | 81.8 | 91.4 | 99.2 | 99.9 | 79.0 | 95.1 | 97.4
CoCa [52] | 1B | 66.3 | 86.2 | 91.8 | 51.2 | 74.2 | 82.0 | 92.5 | 99.5 | 99.9 | 80.4 | 95.7 | 97.7
RO-ViT (ours) | 303M | 68.9 | 87.8 | 92.2 | 51.8 | 75.0 | 83.0 | 92.1 | 99.4 | 99.7 | 80.7 | 96.1 | 97.7
Table 3: Zero-shot image-text retrieval results on COCO and Flickr30K
benchmarks. We compare with dual-encoder methods. We achieve state-of-the-art
results on COCO benchmark. We outperform CoCa-Large with the same backbone by
+3.5 / +1.7 R@1, and even surpass CoCa with 3$\times$ larger backbone (ViT-
Giant) by +2.6 / +0.6 R@1 on image-to-text / text-to-image retrieval. We also
match or outperform the state-of-the-art methods on Flickr benchmark

###  4.1 Open-vocabulary Object Detection

#### LVIS benchmark.

We conduct evaluation on the LVIS dataset [21] which contains a large and
diverse set of 1203 object categories suitable for open-vocabulary detection.
Following the existing works [20, 60, 13], we use the frequent and common
categories as base categories $C_{B}$ for training, and hold out the rare
categories as novel categories $C_{N}$ for testing. Mask APr is the main
benchmark metric. We report the mean over three runs following [20].

As shown in Table 1, our best model achieves 32.1 APr, which significantly
outperforms best existing ViT-based approach OWL-ViT [36] by +6.5 points.
Notably, our method with smaller ViT-B/16 backbone surpasses OWL-ViT with
ViT-L/14 by +3.0 APr. Compared to the best existing approach (ViLD-Ens with
EffNet-B7 backbone), we outperform by +5.8 APr. We note that RO-ViT is simply
trained end-to-end with the cross-entropy loss without the employment of LVIS-
tailored losses such as federated loss [36, 61, 64], weak supervision [64], or
self-training [61, 58, 42].

#### COCO benchmark.

We also present the comparison on the COCO benchmark where the setup uses 48
base categories for training and 17 novel categories for testing [20]. The
main metric is AP50 of novel categories. Due to the smaller number of training
categories, we observe a tendency to overfit to these categories. Unlike most
competing methods, RO-ViT does not use any auxiliary objectives to mitigate
overfitting such as pseudo box/mask labels [26, 14, 61, 58, 42], knowledge
distillation [20, 13], weak supervision [64]. Still, Table 2 shows that RO-ViT
is very competitive among the other methods trained with various sources. In
addition, RO-ViT represents the first ViT-based method to be reported on this
benchmark, as the other method OWL-ViT [36] only benchmarks on LVIS.

###  4.2 Image-Text Retrieval

Apart from evaluating region-level representation through open-vocabulary
detection, we evaluate the image-level representation of RO-ViT in image-text
retrieval through the MS-COCO [7] and Flickr30K [39] benchmarks. We further
train RO-ViT (ViT-L/16), the same model as the last row of Table 1 and 4, for
40K iterations more at a higher resolution _e.g_. 448, following the standard
of existing works [27, 52].

Table 3 shows the comparison with other dual-encoder methods on zero-shot
image-to-text (I2T) and text-to-image (T2I) retrieval. Surprisingly, RO-ViT
outperform all published works on the MS COCO benchmark, and is on par with
the state of the art [52] on Flickr. Compared to CoCa [52] with the same
backbone capacity (ViT-L), RO-ViT outperforms on 11 out of 12 image-text
retrieval metrics. Our model has 303M parameters and achieves the best
performance overall. This shows that our pretraining method not only improves
the region-level representation for open-vocabulary detection but also the
global image-level representation for retrieval.

method | backbone | AP |  AP50 |  AP75
---|---|---|---|---
supervised [20] | R-50 | 25.6 | 38.6 | 28.0
ViLD [20] | R-50 | 11.8 | 18.2 | 12.6
DetPro [13] | R-50 | 12.1 | 18.8 | 12.9
RO-ViT (ours) | ViT-B/16 | 14.0 | 22.3 | 14.9
RO-ViT (ours) | ViT-L/16 | 17.1 | 26.9 | 18.5
Table 4: Transfer detection on Objects365 (Box APs). All models are trained on
the LVIS base categories and tested on Objects365 dataset, without finetuning.

pretraining method | CPE | focal |  APr | AP
---|---|---|---|---
base |  |  |  21.4 (+0.0) | 26.6
no PE |  |  |  18.7 (+0.0) | 25.2
SinCos PE |  |  |  21.7 (+0.0) | 26.9
feat Crop-Resize |  |  |  20.6 (+0.0) | 26.6
ours | ✓ |  |  23.8 (+2.4) | 27.4
ours |  | ✓ |  21.9 (+0.5) | 27.4
ours | ✓ | ✓ |  24.3 (+2.9) | 27.6
(a)

frozen backbone |  APr | AP
---|---|---
base |  9.7 (+0.0) | 12.2
CPE |  16.2 (+6.5) | 17.1
CPE + focal |  16.5 (+6.8) | 17.1
(b)

bblr |  APr | AP
---|---|---
0.0 | 16.5 | 17.1
0.001 | 19.7 | 22.9
0.01 | 20.4 | 23.0
0.1 | 24.3 | 27.6
1.0 | 17.9 | 25.1
(c)

backbone | loc.obj. | norm.lyr |  APr | AP
---|---|---|---|---
ViT-B/16 |  |  |  21.4 (+0.0) | 26.6
ViT-B/16 | ✓ |  |  23.4 (+2.0) | 27.2
ViT-B/16 |  | ✓ |  22.0 (+0.6) | 26.8
ViT-B/16 | ✓ | ✓ |  23.9 (+2.5) | 27.5
ViT-L/16 |  |  |  25.1 (+0.0) | 30.1
ViT-L/16 | ✓ | ✓ |  27.2 (+1.9) | 30.6
(d)

backbone | batch | imp.(d) | CPE + focal |  APr | AP
---|---|---|---|---|---
ViT-B/16 | 4096 |  |  |  21.4 (+0.0) | 26.6
ViT-B/16 | 4096 | ✓ |  |  23.9 (+0.0) | 27.5
ViT-B/16 | 4096 | ✓ | ✓ |  26.2 (+2.3) | 29.2
ViT-B/16 | 16384 | ✓ |  |  26.4 (+0.0) | 30.3
ViT-B/16 | 16384 | ✓ | ✓ |  28.0 (+1.6) | 30.2
ViT-L/16 | 4096 | ✓ |  |  27.2 (+0.0) | 30.6
ViT-L/16 | 4096 | ✓ | ✓ |  29.9 (+2.7) | 31.1
ViT-L/16 | 16384 | ✓ | ✓ |  32.1 (+0.0) | 34.0
(e)

Table 5: Ablation studies. We follow LVIS open-vocabulary detection benchmark.
We train on base (‘frequent’ + ‘common’) categories, test on novel (‘rare’)
categories, and report APr. We use ViT-B/16 backbone and contrastive batch
size 4096 unless otherwise noted.

###  4.3 Transfer Object Detection

We evaluate the generalization ability of RO-ViT in zero-shot transfer
detection. We use the same detector trained on the LVIS base categories (Sec.
4.1) and test on Objects365-v1 validation split [44] following the setup of
ViLD [20, 13]. We replace the LVIS with Objects365 vocabulary embeddings to
perform the transfer detection without finetuning.

Table 4 summarizes the box AP scores in comparison with prior works. Our best
model achieves 17.1 AP, which outperforms existing works ViLD by +5.3 AP and
DetPro by +5.0 AP. Due to the different backbone capacity (R50 vs ViT), we
note that this comparison is to showcase that RO-ViT can achieve strong cross-
dataset detection generalization. For transfer detection, we assume all
categories are novel and set $\alpha,\beta$=(0.0, 0.65) in Equation 1.

###  4.4 Ablation Study

We provide ablation studies on the LVIS open-vocabulary detection and image-
text retrieval benchmarks.

#### Pretraining strategy.

Table LABEL:tab:ablation:pretraining ablates our proposed methods in the
contrastive image-text pretraining. We start with “base” that uses the common
full-image positional embeddings and cross entropy loss. We explore different
PE and observe that removing the positional embeddings hurts the performance
(no PE), while sinusoidal PE (SinCos PE) yields a similar performance to the
baseline. We also try the random crop and resize on the last backbone feature
map (“feat Crop-Resize”) to use region features during pretraining but see no
improvement. In contrast, our proposed Cropped Positional Embedding (CPE)
offers a clear benefit of +2.4 APr and focal contrastive loss (focal) yields
additional gains. Our method achieves a gain of +2.9 APr in total. This shows
that CPE can learn strong region-level representations, and our focal
contrastive learning helps preserving the knowledge about the pretrained
image-text concepts, both of which are essential for open-vocabulary
detection.

backbone | focal | CPE | MS COCO | Flickr30K
---|---|---|---|---
I2T | T2I | I2T | T2I
ViT-B/16 |  |  | 59.1 | 42.5 | 84.8 | 70.9
ViT-B/16 | ✓ |  | 60.3 | 44.0 | 85.4 | 71.6
ViT-B/16 | ✓ | ✓ | 62.2 | 44.2 | 86.5 | 71.8
ViT-L/16 | ✓ | ✓ | 67.0 | 49.7 | 89.5 | 77.2
Table 6: Pretraining evaluation on zero-shot image-text retrieval (Recall@1).
We evaluate the image-level representation of our pretrained model on COCO and
Flickr30k retrieval tasks. We ablate the focal contrastive loss, Cropped
Positional Embedding (CPE) and backbone size.

#### Image-text retrieval.

In Table 6, we demonstrate that our proposed focal contrastive loss and
Cropped Positional Embedding (CPE) both improve the image-text retrieval. We
use a batch size of 16384 , image size 224, and ViT-B/16 backbone as our
baseline, and report Recall@1 metrics on MS COCO and Flickr datasets. We first
observe that focal loss brings clear improvements on all metrics, and adding
CPE brings further improvements with a total of +3.1 / +1.7 image-to-text
(I2T) and +1.7 / +0.9 text-to-image (T2I) improvements on the MS COCO / Flickr
datasets. By replacing ViT-B/16 with ViT-L/16, we observe marked improvements
of +4.8 / +5.5 / +3.0 / +5.4 on these metrics, showing that our recipe is
highly scalable with backbone capacity.

#### Frozen backbone study.

We investigate the effectiveness of the frozen backbone features in open-
vocabulary detection in Table LABEL:tab:ablation:frozen. This evaluation
setting is more rigorous for representation learning because finetuning an
entire network evaluates not only the quality of the representations but also
the initialization and optimization method, as discussed in Goyal  _et al_.
[18]. During the detection finetuning, all ViT backbone layers are frozen, and
only the added detector layers (neck and heads) are trained. We observe frozen
RO-ViT backbone improves the baseline by +6.8 APr, a much larger margin
compared to the full-finetuning setting. This study shows the region-aware
representation of RO-ViT is critical for downstream detection tasks.

#### Backbone learning rate ratio.

As discussed in Sec. 3.1, our open-vocabulary object detector depends on the
pretrained knowledge in the backbone to recognize novel categories beyond the
detector annotations. Therefore, it is important to set the backbone learning
rates lower than the rest of the detector layers to prevent forgetting in the
finetuning process. We present ablations on the learning rate ratio between
the backbone and the detector layers in Table LABEL:tab:ablation:bblr. We
found 0.1$\times$ to be the sweet spot. Higher ratios lead to forgetting and
hurts the APr, and lower ratios hurt the ability to adapt to detection tasks
and hurts AP overall.

#### Downstream detector improvements.

In addition, Table LABEL:tab:ablation:detector demonstrates our improvements
in the downstream detector. Learning the objectness by localization quality,
_i.e_., centerness (“loc.obj.”) [29] and leveraging it into detection score
improves APr by a clear gain of +1.9 points compared to using the conventional
binary-classification in the proposal. This indicates the novel, unlabeled
objects are indeed often missed in the proposal stage, which may limit the
final open-vocabulary detection performance. Localization-quality based
objectness helps remedy this. The use of normalized activation in the last
layers of the classifier and mask heads additionally improves the performance.

#### Scalability with respect to model size and batch size.

While increasing the model capacity and batch size have been beneficial for
contrastive learning in general [38, 40, 52, 27], we study their benefits in
the downstream open-vocabulary detection in Table LABEL:tab:ablation:scale.
Note the detector improvements presented in Sec. 3.3 are applied. We first
observe increasing the batch size from the default 4096 to 16384 shows gains
of +2.1$\sim$2.5 APr for both ViT-B/L. Then, we notice upgrading ViT-B to
ViT-L brings +3.3$\sim$3.7 APr gain across different batch sizes. Last but not
least, the gain of our proposed “CPE + focal” pretraining is consistent among
all settings by improving +2.2$\sim$2.7 APr. Putting everything together, RO-
ViT best model achieves 32.1 APr.

Figure 3: Visualization of learned positional embeddings. Tiles show the
cosine similarity between the positional embedding of the patch (at the
indicated row-column position) and the positional embeddings of all other
patches. ViT-B/16 backbone is used.

###  4.5 Visualization of Positional Embeddings

In Fig. 3, we visualize and compare the learned positional embeddings of RO-
ViT with the baseline, based on ViT-B/16 backbone. Each tile is the cosine
similarity between positional embeddings of one patch and all other patches.
The brightness patterns show in both models, closer patches have more similar
positional embeddings, indicating the encoded distance and locality within the
image.

We observe a few differences between RO-ViT positional embeddings and the
baselines. First, RO-ViT forms more distinct clusters at different patch
locations, _i.e_., ours shows symmetrical global pattern around the center
patch, while the baseline has similar patterns on opposite ends of the image
(_e.g_., the pattern in top-left patch is similar to the bottom-right patch).
Also, the brightness patterns of RO-ViT tend to be more concentrated and
strong (_i.e_., near 1 or -1). To summarize, the visualization indicates RO-
ViT positional embeddings acquire more structure and symmetry than the
baseline in the pretraining process.

##  5 Conclusion

We present RO-ViT– a contrastive image-text pretraining framework to bridge
the gap between image-level pretraining and open-vocabulary detection
finetuning. Our methods are simple, scalable, and easy to apply to any
contrastive backbones with minimal computation overhead and no increase in
parameters. Experiments show that our RO-ViT achieves the state-of-the-art on
LVIS open-vocabulary detection benchmark. Moreover, RO-ViT achieves state of
the art on 9 out of 12 metrics of image-text retrieval benchmark, showing that
the learnt representation is not only beneficial at region-level but also
highly effective on image-level. We hope this study can help the research on
open-vocabulary detection from the perspective of image-text pretraining which
can benefit both region-level and image-level tasks.

## References

  * [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan.  Flamingo: a visual language model for few-shot learning, 2022.
  * [2] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  Vivit: A video vision transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6836–6846, 2021.
  * [3] Yutong Bai, Xinlei Chen, Alexander Kirillov, Alan Yuille, and Alexander C. Berg.  Point-level region contrast for object detection pre-training.  In CVPR, pages 16061–16070, June 2022.
  * [4] Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran.  Zero-shot object detection.  In ECCV, 2018.
  * [5] Hangbo Bao, Li Dong, and Furu Wei.  Beit: Bert pre-training of image transformers.  arXiv preprint arXiv:2106.08254, 2021.
  * [6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In ICCV, pages 9650–9660, October 2021.
  * [7] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  In https://arxiv.org/abs/1504.00325, 2015.
  * [8] Xinlei Chen and Abhinav Gupta.  Webly supervised learning of convolutional networks.  In ICCV, 2015.
  * [9] Berkan Demirel, Ramazan Gokberk Cinbis, and Nazli Ikizler-Cinbis.  Zero-shot object detection by hybrid region embedding.  In BMVC, 2018.
  * [10] Karan Desai and Justin Johnson.  Virtex: Learning visual representations from textual annotations.  In CVPR, 2021.
  * [11] Santosh K Divvala, Ali Farhadi, and Carlos Guestrin.  Learning everything about anything: Webly-supervised visual concept learning.  In CVPR, 2014.
  * [12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [13] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li.  Learning to prompt for open-vocabulary object detection with vision-language model.  In CVPR, 2022.
  * [14] Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma.  Promptdet: Towards open-vocabulary detection using uncurated images.  In European Conference on Computer Vision, pages 701–717. Springer, 2022.
  * [15] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc’Aurelio Ranzato, and Tomas Mikolov.  Devise: A deep visual-semantic embedding model.  In NeurIPS, 2013.
  * [16] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph.  Simple copy-paste is a strong data augmentation method for instance segmentation.  In CVPR, 2021.
  * [17] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin.  Scaling open-vocabulary image segmentation with image-level labels.  In European Conference on Computer Vision, pages 540–557. Springer, 2022.
  * [18] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.  Accurate, large minibatch sgd: Training imagenet in 1 hour.  arXiv preprint arXiv:1706.02677, 2017.
  * [19] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al.  Ego4d: Around the world in 3,000 hours of egocentric video.  In CVPR, pages 18995–19012, 2022.
  * [20] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Open-vocabulary object detection via vision and language knowledge distillation.  In ICLR, 2022.
  * [21] Agrim Gupta, Piotr Dollar, and Ross Girshick.  Lvis: A dataset for large vocabulary instance segmentation.  In CVPR, 2019.
  * [22] Nasir Hayat, Munawar Hayat, Shafin Rahman, Salman Khan, Syed Waqas Zamir, and Fahad Shahbaz Khan.  Synthesizing the unseen for zero-shot object detection.  In ACCV, 2020.
  * [23] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In CVPR, pages 16000–16009, June 2022.
  * [24] Xiangteng He and Yuxin Peng.  Fine-grained image classification via combining vision and language.  In CVPR, 2017.
  * [25] Olivier J. Hénaff, Skanda Koppula, Jean-Baptiste Alayrac, Aaron van den Oord, Oriol Vinyals, and João Carreira.  Efficient visual pretraining with contrastive detection.  In ICCV, pages 10086–10096, October 2021.
  * [26] Dat Huynh, Jason Kuen, Zhe Lin, Jiuxiang Gu, and Ehsan Elhamifar.  Open-vocabulary instance segmentation via robust cross-modal pseudo-labeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7020–7031, 2022.
  * [27] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In ICML, 2021.
  * [28] Armand Joulin, Laurens van der Maaten, Allan Jabri, and Nicolas Vasilache.  Learning visual features from large weakly supervised data.  In ECCV, 2016.
  * [29] Dahun Kim, Tsung-Yi Lin, Anelia Angelova, In So Kweon, and Weicheng Kuo.  Learning open-world object proposals without learning to classify.  IEEE Robotics and Automation Letters, 7(2):5453–5460, 2022.
  * [30] Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova.  F-vlm: Open-vocabulary object detection upon frozen vision and language models.  arXiv preprint arXiv:2209.15639, 2022.
  * [31] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl.  Language-driven semantic segmentation.  arXiv preprint arXiv:2201.03546, 2022.
  * [32] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao.  Grounded language-image pre-training.  In CVPR, 2022.
  * [33] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He.  Exploring plain vision transformer backbones for object detection.  In ECCV, 2022.
  * [34] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In ICCV, pages 2980–2988, 2017.
  * [35] Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, and Ming-Hsuan Yang.  Class-agnostic object detection with multi-modal transformer.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part X, pages 512–531. Springer, 2022.
  * [36] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby.  Simple open-vocabulary object detection with vision transformers.  In ECCV, 2022.
  * [37] Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S Corrado, and Jeffrey Dean.  Zero-shot learning by convex combination of semantic embeddings.  2014\.
  * [38] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingxing Tan, and Quoc V. Le.  Combined scaling for zero-shot transfer learning.  CoRR, abs/2111.10050, 2021.
  * [39] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In ICCV, pages 2641–2649, 2015.
  * [40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [41] Shafin Rahman, Salman Khan, and Nick Barnes.  Improved visual-semantic alignment for zero-shot object detection.  In AAAI, 2020.
  * [42] Hanoona Rasheed, Muhammad Maaz, Muhammad Uzair Khattak, Salman Khan, and Fahad Shahbaz Khan.  Bridging the gap between object and image-level representations for open-vocabulary detection.  arXiv preprint arXiv:2207.03482, 2022.
  * [43] Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus.  Learning visual representations with caption annotations.  In ECCV, 2020.
  * [44] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun.  Objects365: A large-scale, high-quality dataset for object detection.  In ICCV, 2019.
  * [45] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela.  Flava: A foundational language and vision alignment model.  In CVPR, pages 15638–15650, 2022.
  * [46] Josiah Wang, Katja Markert, Mark Everingham, et al.  Learning models for object recognition from natural language descriptions.  In BMVC, 2009.
  * [47] Jiaqi Wang, Wenwei Zhang, Yuhang Zang, Yuhang Cao, Jiangmiao Pang, Tao Gong, Kai Chen, Ziwei Liu, Chen Change Loy, and Dahua Lin.  Seesaw loss for long-tailed instance segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9695–9704, 2021.
  * [48] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao.  Simvlm: Simple visual language model pretraining with weak supervision.  arXiv preprint arXiv:2108.10904, 2021.
  * [49] Fangyun Wei, Yue Gao, Zhirong Wu, Han Hu, and Stephen Lin.  Aligning pretraining for detection via object-level contrastive learning.  In NeurIPS, 2021.
  * [50] Tete Xiao, Colorado J Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell.  Region similarity representation learning.  In ICCV, pages 10539–10548, October 2021.
  * [51] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  Filip: Fine-grained interactive language-image pre-training.  In ICLR, 2021.
  * [52] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  TMLR, 2022.
  * [53] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang.  Florence: A new foundation model for computer vision.  arXiv preprint arXiv:2111.11432, November 2021.
  * [54] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy.  Open-vocabulary detr with conditional matching.  arXiv preprint arXiv:2203.11876, 2022.
  * [55] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang.  Open-vocabulary object detection using captions.  In CVPR, 2021.
  * [56] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  Lit: Zero-shot transfer with locked-image text tuning.  In CVPR, 2022.
  * [57] Bowen Zhang, Zhi Tian, Quan Tang, Xiangxiang Chu, Xiaolin Wei, Chunhua Shen, and Yifan Liu.  Segvit: Semantic segmentation with plain vision transformers.  arXiv preprint arXiv:2210.05844, 2022.
  * [58] Shiyu Zhao, Zhixing Zhang, Samuel Schulter, Long Zhao, Anastasis Stathopoulos, Manmohan Chandraker, Dimitris Metaxas, et al.  Exploiting unlabeled data with vision and language models for object detection.  In ECCV, 2022.
  * [59] Ye Zheng, Ruoran Huang, Chuanqi Han, Xi Huang, and Li Cui.  Background learnable cascade for zero-shot object detection.  In ACCV, 2020.
  * [60] Yiwu Zhong, Jing Shi, Jianwei Yang, Chenliang Xu, and Yin Li.  Learning to generate scene graph from natural language supervision.  In ICCV, 2021.
  * [61] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao.  Regionclip: Region-based language-image pretraining.  In CVPR, 2022.
  * [62] Chong Zhou, Chen Change Loy, and Bo Dai.  Extract free dense labels from clip.  In ECCV, 2022.
  * [63] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In CVPR, 2022.
  * [64] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra.  Detecting twenty-thousand classes using image-level supervision.  In ECCV, 2022.
  * [65] Pengkai Zhu, Hanxiao Wang, and Venkatesh Saligrama.  Don’t even look once: Synthesizing features for zero-shot detection.  In CVPR, 2020.

## Appendix

In the supplementary materials, we provide our detection visualizations along
with our application on ego-centric data. We also provide more implementation
details with used hyper-parameters and discuss the current limitations in the
proposed RO-ViT in the hope to inspire more future research.

##  Appendix A Implementation Details

Table 7 summarizes the hyper-parameters used in the image-text pretraining and
open-vocabulary detection finetuning.

##  Appendix B Detection Visualization

We visualize our RO-ViT outputs on LVIS novel categories (Sec. 4.1) and
transfer detection (Sec. 4.3) onto Objects365 in Fig. 5 and 5, respectively.

We use the ViT-B/16 backbone for visualization. The model was trained on the
LVIS base categories following Sec. 4.1 of the paper. On LVIS, we only show
the novel categories for clarity. RO-ViT is able to detect many novel objects
(_e.g_., fishbowl, sombrero, shepherd dog, gargoyle, persimmon, chinaware,
gourd, satchel, and washbasin). We also visualize the transfer detection on
Objects365 by replacing the vocabulary without finetuning. RO-ViT can detect a
wide range of objects in complex scenes (_e.g_., power outlet, binocular,
glasses, traffic sign, and shrimp).

Figure 4: LVIS novel category visualization (prediction). We only show the
novel categories for clarity. RO-ViT detects many novel categories (pointed by
the red arrows) that it has never seen during detection training (_e.g_.,
fishbowl, sombrero, shepherd dog, gargoyle, persimmon, chinaware, gourd,
satchel, and washbasin).

Figure 5: Objects365 transfer detection visualization (prediction). Our
trained RO-ViT is able to perform on a new dataset without any finetuning, and
captures many challenging categories including novel categories (pointed by
red arrows, _e.g_., power outlet and shrimp).

##  Appendix C Application on Ego-centric Data

A main advantage of open-vocabulary detection is to deal with out-of-
distribution data with categories given by the users on the fly. We test RO-
ViT’s transfer detection to a real-world ego-centric data, Ego4D [19]. We use
the same RO-ViT trained on LVIS base categories with ViiT-B/16 backbone, as in
Sec. B, _i.e_., the model has been never trained on Ego4D.

The categories are provided by the user’s visual inspection of the video, and
are as follows.

  * •

For the indoor scene: plate, cabinet, stove, towel, cleaning rag, ventilator,
knob, sauce and seasoning, steel lid, window, window blinds, plant, light
switch, light, door, carpet, exit sign, doormat, hair, door lock, tree, poster
on the wall, sticker on the wall, faucet, recycle bin, rack, hand, can,
carton, trash, Christmas tree, plastic container, fridge.

  * •

For the grocery store scene: exit sign, poster, chocolate bar, bag of candy,
bag of cookies, snack, oreo, soy sauce, apple, pear, orange, grapes, price
tag, cereal, instant noodle/ramen, cracker, ATM machine, instant noodle,
wooden basket, red ramen bowls, magazine, drugs and medicine, Mayo, Ketchup,
Cup noodle, burrito, Lays/Sun chips, seasoning sauce, black carton, salad
dressing, canned food.

Fig. 6 shows our RO-ViT prediction. Despite the large domain shift and heavy
camera motions, RO-ViT is able to capture many objects in the ego-centric
videos. Specifically, it is able to detect many novel categories never seen
during training (_e.g_., light switch, exit sign, recycle bin, seasoning
sauce, salad dressing, bag of cookies, canned food, and red ramen bowls).

configuration | contrastive | open-vocab. detection
---|---|---
image-text pretraining | finetuning (LVIS)
optimizer | AdamW | SGD
momentum |  $\beta$=0.9 |  $\beta$=0.9
weight decay | 1e-2 | 1e-4
learning rate | 5e-4 (linear decay) | 0.36 (B) / 0.18 (L)
step decay factor | - |  0.1$\times$
step decay schedule | - | [0.8, 0.9, 0.95]
backbone lr ratio | N/A | 0.1 (B) / 0.5 (L)
warmup steps | 1e4 | 1k
total steps | 5e5 | 46.1k (B) / 36.8k (L)
batch size | 4096 or 16384 | 256 (B) / 128 (L)
image size | 224 | 1024
Table 7: RO-ViT hyper-parameters for image-text pretraining and open-
vocabulary detection finetuning. B and L denote ViT-B/16 and ViT-L/16
backbones respectively.

(a) Indoor scene.

(b) Grocery store scene.

Figure 6: Ego4D transfer detection visualization (prediction). Ego4D [19] is a
real-world and out-of-distribution data. Despite large domain shift and heavy
camera movement, RO-ViT is able to detect novel, unseen objects (_e.g_., light
switch, exit sign, recycle bin, seasoning sauce, salad dressing, bag of
cookies, canned food, and red ramen bowls).

##  Appendix D Limitations

RO-ViT leverages the knowledge in pretrained Vision Language Models (VLM).
Therefore, the biases of trained VLMs can proparate into the downstream
detector. In this paper, we use RO-ViT to demonstrate its capabilities and
compare with existing works in open-vocabulary detection. We recommend careful
analysis of ethical risks before using it for other purposes.

##  Appendix E Dataset License

  * •

LVIS [21]: CC BY 4.0 + COCO license

  * •

COCO Captions (retrieval) [7]: CC BY

  * •

Flickr30k (retrieval) [39]: Custom (research-only, non-commercial)

  * •

Objects365 [44]: Custom (research-only, non-commercial)

  * •

Ego4D [19]: https://ego4d-data.org/pdfs/Ego4D-Licenses-Draft.pdf