# Exploring Discrete Diffusion Models for Image Captioning

Zixin Zhu1 Yixuan Wei2* Jianfeng Wang3 Zhe Gan3 Zheng Zhang3 Le Wang1,† Gang
Hua1
Lijuan Wang3 Zicheng Liu3 Han Hu3,†
1Xi’an Jiaotong University 2Tsinghua University
3Microsoft
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Equal.
†Corresponding authors. Zixin and Yixuan are interns at MSRA.

###### Abstract

The image captioning task is typically realized by an auto-regressive method
that decodes the text tokens one by one. We present a diffusion-based
captioning model, dubbed the name DDCap, to allow more decoding flexibility.
Unlike image generation, where the output is continuous and redundant with a
fixed length, texts in image captions are categorical and short with varied
lengths. Therefore, naively applying the discrete diffusion model to text
decoding does not work well, as shown in our experiments. To address the
performance gap, we propose several key techniques including best-first
inference, concentrated attention mask, text length prediction, and image-free
training. On COCO without additional caption pre-training, it achieves a CIDEr
score of 117.8, which is +5.0 higher than the auto-regressive baseline with
the same architecture in the controlled setting. It also performs +26.8 higher
CIDEr score than the auto-regressive baseline (230.3 v.s. 203.5) on a caption
infilling task. With 4M vision-language pre-training images and the base-sized
model, we reach a CIDEr score of 125.1 on COCO, which is competitive to the
best well-developed auto-regressive frameworks. The code is available at
https://github.com/buxiangzhiren/DDCap.

##  1 Introduction

Diffusion models [25, 61] have been successfully exploited for image
generation [8, 48, 53, 57, 56], producing results with high fidelity. However,
there is limited work, if at all, exploring diffusion models for text
generation such as that in image captioning [66, 31], where natural language
description is generated to describe an image. In view of this under-
exploration, in this work, we aim to conduct a systematic study on how we may
adopt diffusion models for accurate image captioning.

Our first attempt of naively applying either continuous or discrete diffusion
models for image captioning did not work well, producing results that are much
more inferior to state-of-the-art results produced from mainstream auto-
regressive models pre-trained on millions of data and even billions of data,
i.e., OSCAR [40], VinVL [81], UFO [67], ViTCap [13], SimVLM [71], and GIT
[69], to name a few. This motivated us to conduct a careful gap analysis.

Figure 1: (a) Image generation: the noise is continuous and redundant with a
fixed length (b) Text generation: the noise is categorical and short with
varied lengths.

As shown in Figure 1, the diffusion model exhibits different natures between
the image generation and the text generation. On one hand, in image
generation, the output images are continuous and redundant with a fixed
length, and the information residing in the output is redundant. On the other
hand, in image captioning, the output texts are discrete and concise with a
variable length. The diffused image resolution is typically $64\times
64=4096$, while the length of the image caption tokens can be around $10$.
Besides, in the inference procedure of a diffusion model, some correct words
generated at the current step may flip to a wrong words because of furtherly
added noises.

Based on the above analysis and in observation that text tokens are discrete
in nature, we focused our research exploration on making discrete diffusion
model produce accurate image captions. Our proposed model, namely DDCap, is
specifically designed to address the above gaps. First of all, we add a
network branch to predict the total token length to flexibly accommodate
variable lengths of the texts. Secondly, since texts are discrete and concise
in nature, we design a concentrated attention mask module so that we may
adaptively concentrate on more informative tokens.

Thirdly, we propose a first best inference strategy, where the top-$K_{t}$
recovered tokens from each diffusion step unchanged in subsequent diffusion
steps. In other words, in each diffusion step, we only add noises to those
tokens that are not marked as fixed in previous diffusion steps. The
top-$K_{t}$ in the current diffusion step are selected only from these
remaining tokens which had been added noises, and they will be fixed in
subsequent steps. The number $K_{t}$ is roughly set as the total number of
tokens we need to recover divided by the number of diffusion steps.

Last but not least, we further design an image-free training technique,
motivated by the classifier-free diffusion model [64]. In some training
examples, we remove the image conditions and force the network to learn the
text prior knowledge. This enables a trade-off to balance the dataset prior
and the image condition. In inference, the text prior knowledge and image
condition are appropriately combined to obtain better captions than just
depend on images. With these orchestrations, we pre-train the DDCap model with
a CLIP image encoder on datasets with 4M images (roughly 10M image-text pairs
in total). We are able to achieve a CIDEr score of 125.1 on the COCO dataset,
which is competitive when compared with state-of-the-art auto-regressive
model. In further exploration of the capacity of our proposed DDCap model, we
hypothesize that our DDCap model considers more holistic contexts to generate
the texts than the auto-regressive baseline, when generating the text
captions.

We further verified our hypothesis by conducting experiments in an infilling
tasks, which is illustrated in Figure 6. In this task, we remove all
adjectives in the ground truth, and require the model to predict the removed
adjectives. As for evaluation metrics, we add the clip scores [23] for
semantic matching. Our proposed DDCap model outperforms the auto-regressive
baseline by a significant margin.

The contributions of this paper are hence summarized as follows.

  * •

We are the first to apply discrete diffusion models for image captioning, and
provide the first evidence that diffusion models can achieve competitive
performance to the best well-developed auto-regressive models.

  * •

We propose four key designs: ($i$) length prediction is used to deal with the
varied length issue in text generation; ($ii$) concentrated attention mask is
used to extract compact text information without the interference of undesired
noise; ($iii$) best-first inference is proposed to reduce the chance of
contaminating correctly generated tokens; and ($iiii$) image-free training to
balance the information in the text prior and the image condition.

  * •

We further show the advantages of discrete diffusion models in a caption
infilling task.

The remainder of the paper is organized as follows. We summarize related work
in Section 2. The details of the proposed DDCap model is proposed in Section
3. Extensive experimental results are reported in Section 4, with both
quantitative and qualitative discussions. We finally conclude in Section 5.

##  2 Related Work

Figure 2: (a) A caption is first tokenized. Then, these tokens are added noise
to become other text tokens or [MASK] tokens. The strength of noise depends on
the sampled step $t$. Lastly, with the help of image tokens, we take these
text tokens carried noise as the input of transformer to predict the pure text
tokens. Meanwhile, we use a simple MLP to predict the length of the caption
based on the [CLS] token. (b) According to the predicted length which denotes
the number of [MASK] tokens, we take all [MASK] tokens as the input. Moreover,
each step $t$ adjusted by the ratio of the predicted length and total noise
step $T$ is sent to the Adaptive LayerNorm layer of the transformer. Image
tokens are used to conduct cross-attention with text tokens. Combining three
inputs, we can obtain the results of each step. The tokens with high
confidence score will be retained instead of re-masking.

#### Image Captioning.

Image captioning has been a longstanding task and recent years have witness
great progress, especially with large-scale vision-language pretraining. Early
models [66, 31, 55, 2] use a visual encoder to extract visual features and
apply recurrent neural network as decoder for caption generation. Later on,
attention-based methods have been proposed to capture multimodal alignment
[74, 45, 29] and perform object relational reasoning [77, 76]. Besides,
researchers have explored to leverage semantic attributes [73, 79, 18, 78] and
scene graphs [75] for captioning. Enhanced attention-based models are further
proposed to improve the performance, such as ORT [22], AoANet [29], M2
Transformer [7], X-LAN [50], and RSTNet [83]. More recent efforts on image
captioning are reflected from the perspective of novel model architecture
design [13, 50, 47, 15] and the use of prior knowledge [46, 24, 72, 42, 35].
For example, Fang et al. [13] present a detector-free image captioning model
with fully transformer architecture, and introduce a novel module to predict
semantic concepts for high performance. Note that most previous works adopt a
standard auto-regressive method to decode tokens one by one.

For non-regressive approaches, masked language modeling is leveraged in [19]
and reinforcement learning is applied in [21]. Instead, we focus on exploring
diffusion models for image captioning, and achieve better performance.

#### Vision-Language Pre-training.

Recent advances in image captioning can be largely attributed to vision-
language pre-training (VLP). Early VLP models [84, 38, 62, 17] adopt a two-
stage approach, where a pre-trained object detector is first used to extract
regional features offline, and then a transformer is stacked on top for
multimodal fusion. Prominent examples along this line include ViLBERT [44],
LXMERT [63], UNITER [6], OSCAR [40], VinVL [81], and LEMON [27]. More
recently, end-to-end VLP models that use convolutional networks and vision
transformers as the image encoder are becoming the mainstream [33, 37, 11, 70,
36, 10, 5]. For example, SimVLM [71] is pre-trained with a single prefix
language modeling objective on large-scale image-text data, GIT [69]
simplifies the architecture as one image encoder and one text decoder under a
single language modeling task, and CoCa [80] is pre-trained with a mix of
contrastive and generative losses. In this work, we also perform VLP for image
captioning, but distinct from all the above works, we are the first to pre-
train a discrete diffusion model for image captioning (and more generally, for
text generation).

#### Diffusion Models.

Diffusion models can be roughly divided into two categories: ($i$) continuous
diffusion, and ($ii$) discrete diffusion. Continuous diffusion models [25, 61,
8, 82] add noise on the continuous-valued input or latent features. Recently,
this family of models have achieved great successes in text-to-image
generation, such as GLIDE [48], DALLE-2 [53], Imagen [57], and Stable
Diffusion [56]. On the other hand, discrete diffusion [60], especially its
application to text generation, has been rarely studied, which is the focus of
this work. Specifically, ImageBART [12] and VQ-Diffusion [20] propose to model
the latent discrete code space of a VQ-VAE [54] by learning a parametric model
using a conditional variant of DDPM [25], for the task of text-to-image
generation. For text generation, discrete diffusion has only shown some
initial success on relatively toy problems [26, 3]. Recent work [41] has also
tried to use continuous diffusion for text generation, but more for fine-
grained control tasks. We are the first to apply discrete diffusion models for
image captioning, and provide the first evidence that diffusion models can
achieve competitive performance to the best well-developed auto-regressive
models.

##  3 Methods

We present DDCap, which is a discrete diffusion model dedicated for image
captioning task. This allows more decoding flexibility, compared to the
current dominant auto-regressive methods. For example, multiple tokens can be
predicted simultaneously rather than one by one. The tokens from the left side
can also depend on those on the right side, instead of a fixed left-to-right
generation process. In Section 3.1, we first introduce the key idea of
applying discrete diffusion models for the captioning task, and then in
Section 3.2, we detail multiple techniques that play a crucial role to boost
the performance.

###  3.1 Discrete diffusion model for image captioning

Unlike continuous diffusion models which add token-level continuous noises
individually to the RGB values or features of each token, discrete diffusion
models perform a noising process at the sentence level, where mask tokens
randomly replace regular tokens with noise levels represented by masking
ratio.

#### Noising process

We denote each text token in a caption as a discrete state, $x$. In the
experiments, we use the BPE tokenizer [58] in GPT-2 to generate the token
identities of the sentence.

The noising process can be thought of as the Markov process, which gradually
adds noises step-by-step. We denote the corresponding noisy token at Step $t$
as $x_{t}$. At each step, each token has a probability of $\gamma_{t}$
transiting to a special absorbing state of [MASK]. If $x_{t-1}$ is not [MASK],
the transition probability vector from Step $t-1$ to $t$ is defined as

$\displaystyle p(x_{t}|x_{t-1})=\begin{cases}\alpha_{t},\,\,x_{t}=x_{t-1};\\\
\gamma_{t},\,\,x_{t}=\texttt{[MASK]};\\\
1-\alpha_{t}-\gamma_{t},\,\,\text{otherwise}.\end{cases}$ (1)

In other words, the each token of $\boldsymbol{x}_{t-1}$ has a probability of
$\alpha_{t}$ to be unchanged and has a probability of $\gamma_{t}$ to be
replaced by the $[\text{MASK}]$ token, leaving the probability of
$\beta_{t}=1-\alpha_{t}-\gamma_{t}$ to be replaced by other tokens except the
$[\text{MASK}]$ token in the vocabulary. If $x_{t-1}$ is a [MASK] token, the
transition probability vector from Step $t-1$ to $t$ is defined

$\displaystyle
p(x_{t}|x_{t-1}=\texttt{[MASK]})=\begin{cases}1,\,\,x_{t}=\texttt{[MASK]};\\\
0,\,\,\text{otherwise}.\end{cases}$ (2)

By these noising process, with a sufficient number of steps $T$, all tokens
will become [MASK].

#### Denoising process

The denoising diffusion model starts with an all [MASK] sequence and gradually
recovers the original signal step-by-step. A Transformer network is used to
predict the reserve projection, i.e., $p_{\theta}(x_{t-1}|x_{t},y)$, where $y$
is the image feature extracted from an image encoder, i.e., initialized from
CLIP [52] in our experiments. Cross-attention layers are employed to
incorporate image features as conditions for the caption generation process.
The time step $t$ is encoded as a sinusoidal positional embedding to guide the
adaptive layer norm [20] in the Transformer layers:

$\displaystyle p$ $\displaystyle=t*\text{step}_{\text{scale}}/T,$ (3)
$\displaystyle\text{PE}_{i}$
$\displaystyle=\begin{cases}sin(p/10000^{2i/d_{\text{model}}}),i<d_{\text{model}}/2\\\
cos(p/10000^{2i/d_{\text{model}}}),i\geq d_{\text{model}}/2,\end{cases}$ (4)

where $\text{step}_{\text{scale}}$ is the wavelength, i.e., 8000 in our
experiments, and $d_{\text{model}}$ is the hidden dimension.

#### Training

Unlike continuous diffusion models, where denoising networks typically predict
noises, discrete diffusion models perform the noising process at the sentence-
level through a masking strategy, and are difficult to directly predict
noises. Inspired by [3], we directly predict the original text tokens $x_{0}$
instead:

$\mathcal{L}_{x_{0}}=-\log p_{\theta}(x_{0}|x_{t},y).$ (5)

The training process is illustrated as the left part of Figure 2.

#### Inference

Starting from $x_{T}$, the model first predicts $x_{T-1}$, then predicts
$x_{T-2}$, and so on. After $T$ steps, the discrete diffusion model produces
the final result of $x_{0}$.

The process of predicting $x_{t-1}$ from $x_{t}$ is computed as follows.
First, $\hat{x}_{0}$ is estimated from $p_{\theta}(x_{0}|x_{t},y)$ via the
trained denoising network. Then, $\\{t-1\\}$-step noise is added on the
predicted $\hat{x}_{0}$ through a Markov chain to get $x_{t-1}$.

###  3.2 Key Designs for Performance Improvement

Empirically, with naive implementations, we observe significant worse
performance with the diffusion model than the auto-regressive counterpart. We
hypothesize that this is due to that text generation is very different from
image generation. For example, the generated text lengths are different, while
the generated image resolutions are normally fixed. The information in the
text is compact, while the image contains redundant information. The mask
token provides almost no information, while the image is continuously noised
into different levels. To address the above issues, we propose the following
four techniques: ($i$) length prediction, ($ii$) concentrated attention mask,
($iii$) best-first inference and ($iiii$) image-free training.

#### Length Prediction.

In auto-regressive methods, the text tokens are predicted from left to right
one by one until a special [EOS] token. However, diffusion model has no
concept of left-to-right or right-to-left directions. The performance is also
much worse if we respect the [EOS] token and treat only the left-side tokens
as valid. Instead of relying on the special token, we propose to add a network
branch to specifically predict the total token length. Specifically, we regard
the [CLS] token from the CLIP image encoder as the feature for length
prediction, implemented via a simple Multilayer Perceptron (MLP). Cross-
entropy loss is used for training. Empirically, we also find it is beneficial
to cut the gradient back to the image encoder. Without the length as a prior,
we may have to use a maximum token length to generate the captions. Thus,
another benefit is the speed-up with an appropriately predicted token length.

#### Concentrated Attention Mask (CAM).

In our diffusion model, the absorbing [MASK] token carries almost no
information, and it can take a majority in the sequence. For instance, the
denoising process starts from a full [MASK] sequence. With the bidirectional
attention, the [MASK] tokens may overwhelm the attention layers. To make the
tokens only concentrate on informative tokens, we propose the following
changes on the attention mask: ($i$) normal text tokens do not depend on the
[MASK] tokens; ($ii$) the [MASK] token does not depend on the other [MASK]
tokens. Empirically, we find this novel attention mask design significantly
boosts the performance.

#Row | Best-first inference | CAM | Length Prediction | Image-free training | C | B@4 | M | R | S
---|---|---|---|---|---|---|---|---|---
a |  |  |  |  | 20.6 | 7.4 | 18.8 | 34.6 | 12.3
b |  | $\checkmark$ |  |  | 43.6 | 11.4 | 20.5 | 39.1 | 14.4
c | $\checkmark$ |  |  |  | 45.2 | 20.3 | 26.9 | 47.3 | 21.3
d | $\checkmark$ |  | $\checkmark$ |  | 92.6 | 27.3 | 25.4 | 51.8 | 18.7
e | $\checkmark$ | $\checkmark$ |  |  | 97.5 | 28.2 | 28.1 | 54.0 | 21.7
f | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | 116.7 | 34.6 | 28.1 | 57.4 | 21.5
g | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 117.8 | 35.0 | 28.2 | 57.4 | 21.7
Table 1: Ablation study on the effectiveness of each component described in
Sec. 3.2. CAM: concentrated attention mask.

#### Best-first Inference.

During inference, the text token can be recovered earlier than the last step.
To reduce the chance of contaminating these correct tokens, we propose to keep
the top-$K^{t}$ recovered tokens unchanged at each step. Let $N_{L}$ be the
predicted token length. If $N_{L}\leq T$ ($T$ is the total diffusion steps
during training), we also reduce the inference diffusion steps to $N_{L}$,
such that at each step, one extra token is recovered. Meanwhile, the step
index $t$, which is used in the adaptive layer norm, is also shrunk
proportionally. If the predicted token length is larger ($N_{L}>T$), at each
step $t$ (from $T$ to $1$), we keep $K^{t}=\lfloor
N_{L}(T-t+1)/T\rfloor-\lfloor N_{L}(T-t)/T\rfloor$ tokens unchanged.

#### Image-free Training.

Generating the text is the key of the image captioning task. To focus more on
the text modeling part, we propose an image-free strategy to enhance the
weight of text modeling, which is motivated from the classifier-free diffusion
[64]. Specifically, with a probability of $r$ (0.2 in experiments), the image
features are replaced with a trainable embedding $f$, and the diffusion model
calculates $p_{\theta}(x_{0}|x_{t},f)$ without image conditions. The loss then
becomes

$\displaystyle\mathcal{L}_{x_{0}}^{\prime}=-\log
p_{\theta}({x}_{0}|{x}_{t},{f}).$ (6)

With the loss, the network can focus more on learning how to generate text,
rather than how to learn the image features. During inference, the probability
likelihood is correspondingly computed as

$\displaystyle\log p_{\theta}({x}_{0}|{x}_{t},{y})^{\prime}=\log
p_{\theta}({x}_{0}|{x}_{t},{f})+$ (7) $\displaystyle s(\log
p_{\theta}({x}_{0}|{x}_{t},{y})-\log p_{\theta}({x}_{0}|{x}_{t},{f})),$ (8)

where $s$ denotes the guidance scale ($s=1.17$ in experiments). If $s=1$, the
image-free process is effectively removed. If $s>1$, the network will respect
more of the image signals.

##  4 Experiments

###  4.1 Setup

Dataset. We conduct our experiments on the popular COCO dataset [43].
Specifically, the COCO dataset contains 123,287 images labeled with 5 captions
for each, including 82,783 training images, 40,504 validation images and
40,775 images as test set for online evaluation as well. Following the widely-
used “Karpathy” split [32], we use 113,287 images for training, 5000 images
for validation, and 5000 for testing. For our diffusion model, our vocabulary
size is 50,257, and the max length of each sentence is 20. During training,
the weight decay is set to 0.01, and the learning rate first linearly warms up
to 2e-4, and then drops to 0 following cosine decay. The number of training
epochs is 30, and the warm-up epoch is 5. The batch size is 512. Following the
standard evaluation setup, we report the performances of our model and compare
to other methods over five metrics: BLEU@4 [51], METEOR [4], ROUGE-L [30],
CIDEr-D [65], and SPICE [1].

Network backbone and pre-training. ViT-B/16 [9] from the pre-trained CLIP
model [52] is chosen as our image backbone, and the corresponding image patch
size is 16. More specifically, ViT-B/16 has 12 transformer layers with 768 as
the hidden size. For pre-training, following common practice [6, 11], we use a
combined dataset including COCO [43], Conceptual Captions (CC) [59], SBU [49],
and Visual Genome (VG) [34]. This results in roughly 4 million images with 10
million associated captions. We also pre-train the model based on ViT-L/16
with 15 epochs when comparing with state-of-the-art methods. For our diffusion
model, the peak learning rate is 1e-4 with batch size 1024 during pre-training
and 1e-5 with batch size 512 during fine-tuning. The diffusion model is
randomly initialized and the learning rate of the well-initialized image
encoder is reduced to 0.07 of the diffusion model.

###  4.2 Analysis

Due to the consideration of computation cost, we conduct ablation study based
on the fixed image encoder and disable the image-free training unless
explicitly specified. No pretraining is conducted, and all results are
reported on the “Karpathy” validation set.

#### Ablation on the key designs.

As discussed in Section 3.2, we have four key components in our model. Table 1
reports the ablation study with different combinations. Comparing row (c) and
(a), we can see the Best-first Inference strategy can significantly boost the
performance from 20.6 to 45.2 in CIDEr. Thus, it is crucial to keep the best
predictions unchanged during the denoising process. By appropriately masking
the attentions with CAM, the performance can be improved from 45.2 (row c) to
97.5 (row e). This suggests that it is necessary to prevent tokens from
depending on [MASK] tokens. Otherwise, it may reduce the weights of other text
tokens and lead to the reduction of the signal-to-noise ratio. The length
prediction branch facilitates the model to find the end of the caption, and we
can see the performance improved from 45.2 (row c) to 92.6 (row d). Lastly,
image-free training also helps improve the performance (116.7 to 117.8 from
row f to row g).

#### Concentrated attention mask.

In our proposed concentrated attention mask (CAM), we have two modifications:
($i$) the text token does not depend on [MASK] tokens, denoted as T2M; ($ii$)
the [MASK] token does not depend on other [MASK] tokens, denoted as M2M. Table
2 suggests that both T2M and M2M improve the performance and M2M provides much
larger gain (from 92.6 to 115.6). The reason may be that [MASK] token’s
representation is the key to recover new tokens and it is beneficial not to
focus on the other [MASK] tokens, which carry less information. We also
experiment to enable CAM only at the first half or the second half denoising
steps, as in the last two rows of Table 2. In the first half ($t\geq 10$),
more tokens are [MASK] and thus CAM impacts more to the performance. The
results also demonstrates CAM is an effective way to improve the performance.

Method | M2M | T2M | C | B@4 | M | R | S
---|---|---|---|---|---|---|---
DDCap |  |  | 92.6 | 27.3 | 25.4 | 51.8 | 18.7
DDCap |  | $\checkmark$ | 94.1 | 27.2 | 25.7 | 52.1 | 18.9
DDCap | $\checkmark$ |  | 115.6 | 34.2 | 27.9 | 57.2 | 21.4
DDCap | $\checkmark$ | $\checkmark$ | 116.7 | 34.6 | 28.1 | 57.4 | 21.5
DDCap ($t<10$) | $\checkmark$ | $\checkmark$ | 98.8 | 29.8 | 25.2 | 52.8 | 19.4
DDCap ($t\geq 10$) | $\checkmark$ | $\checkmark$ | 112.9 | 33.2 | 27.9 | 56.5 | 21.2

Table 2:  Ablation study on our proposed concentrated attention mask (CAM).
“M2M”: the mask token does not depend on the other mask tokens. “T2M”: the
text token does not depend on mask tokens. “($t<10$)”: CAM is enabled at the
second half of denoising steps. “($t\geq 10$)”: CAM is enabled at the first
half of denoising steps where most of the tokens are mask tokens.

#### Error rate of the length prediction.

Figure 3 shows the error distribution of the length prediction branch. The
error is calculated as $(N_{L}-GT)/GT$, where $N_{L}$ denotes the predicted
length and $GT$ is the ground-truth length. As we can see, most of the
predictions are exactly correct, the error rate of 88.54% images are less than
0.05. The max error rate is less than 0.25. This suggests the predicted length
is reliable to determine the token length in decoding.

#### Embedding for time step $t$.

We use the sinusoidal positional embedding as in Eqn. 4 to map the time step
$t$ to an embedding representation. Another way is to use a learnable
embedding layer. Table 3 compares these two different methods. The results
indicate the sinusoidal positional embedding with appropriate rescaling factor
can achieve better performance.

Figure 3: The accuracy of length prediction.

Method | C | B@4 | M | R | S
---|---|---|---|---|---
DDCap w/o Adaptive T | 115.5 | 34.2 | 28.0 | 57.3 | 21.3
DDCap w/ Adaln T | 115.1 | 34.1 | 28.0 | 57.1 | 21.3
DDCap w/ sin/cos T (4000) | 115.9 | 34.5 | 28.0 | 57.4 | 21.4
DDCap w/ sin/cos T (6000) | 114.8 | 33.7 | 27.9 | 57.0 | 21.5
DDCap w/ sin/cos T (8000) | 116.7 | 34.6 | 28.1 | 57.6 | 21.5
DDCap w/ sin/cos T (10000) | 115.1 | 34.0 | 27.9 | 56.9 | 21.3

Table 3: The influences of different ways of Adaptive LayerNorm layers. “Adaln
T” denotes the normal embedding layer, and “ sin/cos T” denotes the
“Sinusoidal PosEmb” layer.

#### Image-free training.

The impact of hyper-parameters (i.e., $r$ and $s$) is shown in Figure 4. When
the training ratio $r$ is 0.2 and the inference scale is 1.17, our model
achieve the best performance. It is worth noting that the best guidance scale
range is different to the one which is normally set to 5 for image generation
[64]

Figure 4: The guidance scale $s$ and ratio $r$ of image-free training. Top:
The interval of the scale is 0.1. Bottom: The interval of the scale is 0.01.

Method | C | B@4 | M | R | S
---|---|---|---|---|---
AR Basline | 112.8 | 33.9 | 28.0 | 56.4 | 21.3
Continuous Diffusion | 91.9 | 25.0 | 25.0 | 51.1 | 19.1
DDCap | 117.8 | 35.0 | 28.2 | 57.4 | 21.7

Table 4: Comparison to an auto-regressive (AR) baseline and our implementation
of a continuous diffusion model for captioning.

#### vs AR approach and continuous diffusion model.

We compare our proposed method with an auto-regressive (AR) baseline and our
implementation of a continuous diffusion model in a controlled way. For AR
approach, we use the same image encoder to extract the image representation,
and re-use our diffusion transformer network as the decoder, such that the
number of model parameters are similar. For the continuous diffusion model, we
map the discrete text tokens to a continuous reprentation through the
pretrained embedding layer of GPT-2, and use the diffusion transformer to
recover these embeddings. The number of total step $T$ is set to 10,000, and
the noise schedule is linear. We find it helps with the following
modifications: 1) normalize the embedding layers with the mean and variance of
all embedding vectors; 2) during inference, each embedding vector $x_{t}$ is
decoded to tokens and then re-embedded to vectors before estimating $x_{t-1}$.
The training epochs are set to 30, 100 and 30, for AR model, continuous
diffusion model, and our discrete diffusion model, respectively.

The results are shown in Table 4, which indicates that our discrete diffusion
model demonstrates better performance than auto-regressive model and the
carefully designed continuous diffusion model. Compared with auto-regressive
model, the discrete diffusion model can utilize bidirectional context in text.
Compared with continuous diffusion model, discrete diffusion model aligns
better as the text is naturally discrete. Moreover, the differences between
continuous and discrete diffusion models are shown in Figure 5.

Figure 5: An overview of using _discrete_ and _continuous_ diffusion models
for image captioning. Empirically, discrete diffusion achieves better
performance, which is also the focus of this paper.

###  4.3 Comparison with Prior Arts

Table 5 presents the comparison results on the COCO dataset. Our DDCap shows
competitive performance. Specifically, compared with non-autoregressive
methods, our model achieves the best performance, which suggests the
effectiveness of our key designs. Compared with auto-regressive methods, our
performance is better than many methods and is comparable to ViTCap [13].
Hopefully, our study on the diffusion model can motivate more efforts on this
direction.

Method | #Param. | #Images | C | B@4 | M | S
---|---|---|---|---|---|---
Auto-regressive models
$\rm{UVLP}$ [84] | 111.7M | 4M | 116.9 | 36.5 | 28.4 | 21.2
$\rm{MiniVLM}$ [68] | 34.5M | 14M | 119.8 | 35.6 | 28.6 | 21.6
$\rm{DistillVLM}$ [14] | 34.5M | 7M | 120.8 | 35.6 | 28.7 | 22.1
$\rm{UFO_{B}}$ [67] | 0.1B | 4M | 122.8 | 36.0 | 28.9 | 22.2
$\rm{OSCAR_{B}}$ [40] | 0.1B+64M† | 7M | 123.7 | 36.5 | 30.3 | 23.1
$\rm{UNIMO_{B}}$ [39] | - | 9M | 124.4 | 38.8 | - | -
ViTCap [13] | 0.2B | 4M | 125.2 | 36.3 | 29.3 | 22.6
$\rm{VinVL_{B}}$ [81] | 0.1B+0.2B† | 6M | 129.3 | 38.2 | 30.3 | 23.6
$\rm{GIT_{B}}$ [69] | 129M | 4M | 131.4 | 40.4 | 30.0 | 23.0
$\rm{LEMON_{B}}$ [28] | 111.7M | 0.2B | 133.3 | 40.3 | 30.2 | 23.3
$\rm{SimVLM_{B}}$ [71] | - | 1.8B | 134.8 | 39.0 | 32.9 | 24.0
Non-autoregressive models
$\rm{MNIC}$ [19] | - | - | 108.5 | 31.5 | 27.5 | 21.1
$\rm{NAIC_{B,KD}}$ [21] | - | - | 115.5 | 35.3 | 27.3 | 20.8
$\rm{FNIC}$ [16] | - | - | 115.7 | 36.2 | 27.1 | 20.2
$\rm{DDCap}$ (Ours) | 280.1M | 4M | 125.1 | 37.1 | 29.1 | 22.7

Table 5: Performance comparison on COCO captioning Karpathy [32] split with
pretraining, where B@4, M, R, C denote BLEU@4, METEOR, ROUGE-L, CIDEr and
SPICE scores. CIDEr optimization is not used for all models. ($\dagger$)
VinVL/OSCAR: the extra parameters are for object detector. All of the results
do not contain CIDEr optimization. Figure 6: Visualization of the caption
infilling task. Figure 7: Visualization of our model and auto-regressive
baseline on validation images of COCO dataset. The wrong parts of a caption
are highlighted.

Method | C | B@4 | M | R | S | CLIP-Score
---|---|---|---|---|---|---
AR baseline | 203.5 | 76.3 | 49.3 | 89.1 | 36.5 | 75.7
DDCap | 230.3 | 85.1 | 56.3 | 93.1 | 39.9 | 76.4

Table 6: Comparison between our model and an auto-regressive baseline for the
caption infilling task. Figure 8: Visualization of the intermediate steps of
the caption generation process with our model.

###  4.4 Caption Infilling Tasks

As the text tokens are not decoded from a fixed left-to-right order, our
approach is naturally suitable for the fill-in task task, which targets to
fill in the empty words in a sentence as shown in Figure 6. To evaluate the
performance, we remove all the adjective words on the “Karpathy” validation
dataset. The adjectives are detected by nltk111https://www.nltk.org/. The
captions with no detected words are removed. Then, the to-be-filled positions
are filled with the mask token to start a diffusion process to recover the
full sentence. As for evaluation metrics, we add the CLIP scores [23] for
semantic matching. The comparison are shown in Table 6, suggesting the bi-
directional model can well solve the task.

###  4.5 Qualitative Analysis

Figure 8 shows the text tokens in the intermediate steps. Empirically, simple
objects and articles are predicted first, followed by preposition and
adjectives. Finally, DDCap will add nouns to make the sentence grammatically
correct. In other words, the way of describing the image roughly follows the
order of first to describe the objects and then the relationship between the
objects.

Furthermore, Figure 7 shows prediction examples on the MSCOCO validation set,
which shows reasonable prediction results. Compared with auto-regressive
baseline, our DDCap has obvious advantages in recognizing objects. This
advantages maybe come from our concentrated attention mask which provide more
text context.

However, our DDCap sometimes generates some repeated words. This issue may be
handled by two ways: ($i$) we first get the caption, and then find the these
repeated words. these repeated words will be re-masked and re-generated.
($ii$) the repeated words in a caption will be deleted. Then we get a new
length of this caption without repeated words. According to the new length, we
re-generate the whole caption.

##  5 Conclusions

This paper introduces a novel discrete diffusion model, termed DDCap, for
image captioning. DDCap proposes several novel designs: length prediction,
concentrated attention mask, and best-first inference. Ablation experiments
under controlled settings demonstrate the effectiveness of each component.
Results on COCO datast show that DDCap is comparable with state-of-the-art
auto-regressive methods. In addition, we have introduced a new caption
infilling task to highlight our advantages. For future work, we plan to
perform larger-scale pre-training, and we hope our work can inspire more
efforts on using diffusion models for text generation.

## References

  * [1] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould.  Spice: Semantic propositional image caption evaluation.  In ECCV, 2016.
  * [2] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang.  Bottom-up and top-down attention for image captioning and visual question answering.  In CVPR, pages 6077–6086, 2018.
  * [3] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg.  Structured denoising diffusion models in discrete state-spaces.  Advances in Neural Information Processing Systems, 34:17981–17993, 2021.
  * [4] Satanjeev Banerjee and Alon Lavie.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In IEEvaluation@ACL, 2005.
  * [5] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al.  Pali: A jointly-scaled multilingual language-image model.  arXiv preprint arXiv:2209.06794, 2022.
  * [6] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  UNITER: Universal image-text representation learning.  In ECCV, 2020.
  * [7] Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara.  Meshed-memory transformer for image captioning.  In CVPR, pages 10578–10587, 2020.
  * [8] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  In NeurIPS, 2021.
  * [9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  ArXiv, abs/2010.11929, 2021.
  * [10] Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, et al.  Coarse-to-fine vision-language pre-training with fusion in the backbone.  In NeurIPS, 2022.
  * [11] Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Zicheng Liu, Michael Zeng, et al.  An empirical study of training end-to-end vision-and-language transformers.  In CVPR, 2022.
  * [12] Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjorn Ommer.  Imagebart: Bidirectional context with multinomial diffusion for autoregressive image synthesis.  In NeurIPS, 2021.
  * [13] Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lin Liang, Zhe Gan, Lijuan Wang, Yezhou Yang, and Zicheng Liu.  Injecting semantic concepts into end-to-end image captioning.  In CVPR, 2022.
  * [14] Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lijuan Wang, Yezhou Yang, and Zicheng Liu.  Compressing visual-linguistic model via knowledge distillation.  In ICCV, pages 1428–1438, 2021.
  * [15] Zhengcong Fei, Xu Yan, Shuhui Wang, and Qi Tian.  Deecap: Dynamic early exiting for efficient image captioning.  In CVPR, pages 12216–12226, 2022.
  * [16] Zheng-cong Fei.  Fast image caption generation with position alignment.  arXiv preprint arXiv:1912.06365, 2019.
  * [17] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu.  Large-scale adversarial training for vision-and-language representation learning.  In NeurIPS, 2020.
  * [18] Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Li Deng.  Semantic compositional networks for visual captioning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5630–5639, 2017.
  * [19] Junlong Gao, Xi Meng, Shiqi Wang, Xia Li, Shanshe Wang, Siwei Ma, and Wen Gao.  Masked non-autoregressive image captioning.  arXiv preprint arXiv:1906.00717, 2019.
  * [20] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo.  Vector quantized diffusion model for text-to-image synthesis.  In CVPR, pages 10696–10706, 2022.
  * [21] Longteng Guo, Jing Liu, Xinxin Zhu, Xingjian He, Jie Jiang, and Hanqing Lu.  Non-autoregressive image captioning with counterfactuals-critical multi-agent learning.  arXiv preprint arXiv:2005.04690, 2020.
  * [22] Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares.  Image captioning: Transforming objects into words.  arXiv preprint arXiv:1906.05963, 2019.
  * [23] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning.  arXiv preprint arXiv:2104.08718, 2021.
  * [24] Yusuke Hirota, Yuta Nakashima, and Noa Garcia.  Quantifying societal bias amplification in image captioning.  In CVPR, pages 13450–13459, 2022.
  * [25] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In NeurIPS, 2020.
  * [26] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling.  Argmax flows and multinomial diffusion: Towards non-autoregressive language models.  2021\.
  * [27] Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang.  Scaling up vision-language pre-training for image captioning.  In CVPR, 2022.
  * [28] Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang.  Scaling up vision-language pre-training for image captioning.  In CVPR, pages 17980–17989, 2022.
  * [29] Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei.  Attention on attention for image captioning.  In ICCV, 2019.
  * [30] Unnat Jain, Svetlana Lazebnik, and Alexander G. Schwing.  Two can play this game: Visual dialog with discriminative question generation and answering.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5754–5763, 2018.
  * [31] Andrej Karpathy and Li Fei-Fei.  Deep visual-semantic alignments for generating image descriptions.  In CVPR, 2015.
  * [32] Andrej Karpathy and Li Fei-Fei.  Deep visual-semantic alignments for generating image descriptions.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39:664–676, 2017.
  * [33] Wonjae Kim, Bokyung Son, and Ildoo Kim.  ViLT: Vision-and-language transformer without convolution or region supervision.  In ICML, 2021.
  * [34] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International Journal of Computer Vision, 123:32–73, 2016.
  * [35] Chia-Wen Kuo and Zsolt Kira.  Beyond a pre-trained object detector: Cross-modal textual and visual context for image captioning.  In CVPR, pages 17969–17979, 2022.
  * [36] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In ICML, 2022.
  * [37] Junnan Li, Ramprasaath R Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  In NeurIPS, 2021.
  * [38] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  VisualBERT: A simple and performant baseline for vision and language.  arXiv preprint arXiv:1908.03557, 2019.
  * [39] Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang.  Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning.  arXiv preprint arXiv:2012.15409, 2020.
  * [40] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In ECCV, 2020.
  * [41] Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B Hashimoto.  Diffusion-lm improves controllable text generation.  arXiv preprint arXiv:2205.14217, 2022.
  * [42] Yehao Li, Yingwei Pan, Ting Yao, and Tao Mei.  Comprehending and ordering semantics for image captioning.  In CVPR, pages 17990–17999, 2022.
  * [43] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft coco: Common objects in context.  ArXiv, abs/1405.0312, 2014.
  * [44] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In NeurIPS, 2019.
  * [45] Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher.  Knowing when to look: Adaptive attention via a visual sentinel for image captioning.  In CVPR, 2017.
  * [46] Zihang Meng, David Yang, Xuefei Cao, Ashish Shah, and Ser-Nam Lim.  Object-centric unsupervised image captioning.  In ECCV, pages 219–235. Springer, 2022.
  * [47] Van-Quang Nguyen, Masanori Suganuma, and Takayuki Okatani.  Grit: Faster and better image captioning transformer using dual visual features.  In ECCV, pages 167–184. Springer, 2022.
  * [48] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [49] Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg.  Im2text: Describing images using 1 million captioned photographs.  In NIPS, 2011.
  * [50] Yingwei Pan, Ting Yao, Yehao Li, and Tao Mei.  X-linear attention networks for image captioning.  In CVPR, pages 10971–10980, 2020.
  * [51] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In ACL, 2002.
  * [52] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, pages 8748–8763. PMLR, 2021.
  * [53] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [54] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with vq-vae-2.  In NeurIPS, 2019.
  * [55] Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel.  Self-critical sequence training for image captioning.  In CVPR, pages 7008–7024, 2017.
  * [56] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * [57] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [58] Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural machine translation of rare words with subword units.  arXiv preprint arXiv:1508.07909, 2015.
  * [59] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In ACL, 2018.
  * [60] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In ICML, pages 2256–2265. PMLR, 2015.
  * [61] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In ICLR, 2021.
  * [62] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  VL-BERT: Pre-training of generic visual-linguistic representations.  In ICLR, 2019.
  * [63] Hao Tan and Mohit Bansal.  LXMERT: Learning cross-modality encoder representations from transformers.  In EMNLP, 2019.
  * [64] Zhicong Tang, Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen.  Improved vector quantized diffusion models.  arXiv preprint arXiv:2205.16007, 2022.
  * [65] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566–4575, 2015.
  * [66] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.  Show and tell: A neural image caption generator.  In CVPR, pages 3156–3164, 2015.
  * [67] Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, and Lijuan Wang.  Ufo: A unified transformer for vision-language representation learning.  arXiv preprint arXiv:2111.10023, 2021.
  * [68] Jianfeng Wang, Xiaowei Hu, Pengchuan Zhang, Xiujun Li, Lijuan Wang, Lei Zhang, Jianfeng Gao, and Zicheng Liu.  Minivlm: A smaller and faster vision-language model.  arXiv preprint arXiv:2012.06946, 2020.
  * [69] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang.  Git: A generative image-to-text transformer for vision and language.  arXiv preprint arXiv:2205.14100, 2022.
  * [70] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang.  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  In ICML, 2022.
  * [71] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao.  Simvlm: Simple visual language model pretraining with weak supervision.  arXiv preprint arXiv:2108.10904, 2021.
  * [72] Mingrui Wu, Xuying Zhang, Xiaoshuai Sun, Yiyi Zhou, Chao Chen, Jiaxin Gu, Xing Sun, and Rongrong Ji.  Difnet: Boosting visual information flow for image captioning.  In CVPR, pages 18020–18029, 2022.
  * [73] Qi Wu, Chunhua Shen, Lingqiao Liu, Anthony Dick, and Anton Van Den Hengel.  What value do explicit high level concepts have in vision to language problems?  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 203–212, 2016.
  * [74] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In ICML, 2015.
  * [75] Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai.  Auto-encoding scene graphs for image captioning.  In CVPR, 2019.
  * [76] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei.  Exploring visual relationship for image captioning.  In ECCV, 2018.
  * [77] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei.  Hierarchy parsing for image captioning.  In ICCV, 2019.
  * [78] Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, and Tao Mei.  Boosting image captioning with attributes.  In ICCV, 2017.
  * [79] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo.  Image captioning with semantic attention.  In CVPR, pages 4651–4659, 2016.
  * [80] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  TMLR, 2022.
  * [81] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao.  VinVL: Revisiting visual representations in vision-language models.  In CVPR, 2021.
  * [82] Qinsheng Zhang, Molei Tao, and Yongxin Chen.  gddim: Generalized denoising diffusion implicit models.  arXiv preprint arXiv:2206.05564, 2022.
  * [83] Xuying Zhang, Xiaoshuai Sun, Yunpeng Luo, Jiayi Ji, Yiyi Zhou, Yongjian Wu, Feiyue Huang, and Rongrong Ji.  Rstnet: Captioning with adaptive attention on visual and non-visual words.  In CVPR, pages 15465–15474, 2021.
  * [84] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao.  Unified vision-language pre-training for image captioning and vqa.  In AAAI, 2020.

## Appendices

Hyperparameters | Image encoder | Diffusion model
---|---|---
Layers | 12 | 12
Hidden size | 768 | 768
Attention heads | 12 | 12
Patch size | 16 * 16 | -
Adaptive layer norm | - | sinusoidal
Noise schedule | - | linear schedule
The total step $T$ | - | 20
Training steps | 30k | 30k
Batch size | 512 | 512
AdamW $\epsilon$ | 1e-8 | 1e-8
AdamW $\beta$ | (0.9, 0.999) | (0.9, 0.999)
Peak learning rate | 2e-5 | 2e-4
Learning rate schedule | Cosine | Cosine
Warmup steps | 6k | 6k
Drop Path | 0.1 | -
Weight decay | 0.01 | 0.01
Image resolution | 224 | -
Text length | - | 20

Table 7:  Hyperparameters for training DDCap on COCO captioning without
pretraining.

Hyperparameters | Image encoder | Diffusion model
---|---|---
Layers | 12 | 12
Hidden size | 768 | 768
Attention heads | 12 | 12
Patch size | 16 * 16 | -
Adaptive layer norm | - | sinusoidal
Noise schedule | - | linear schedule
The total step $T$ | - | 20
Training steps | 142k | 142k
Batch size | 1024 | 1024
AdamW $\epsilon$ | 1e-8 | 1e-8
AdamW $\beta$ | (0.9, 0.999) | (0.9, 0.999)
Peak learning rate | 1e-5 | 1e-4
Learning rate schedule | Cosine | Cosine
Warmup steps | 47k | 47k
Drop Path | 0.1 | -
Weight decay | 0.01 | 0.01
Image resolution | 224 | -
Text length | - | 20

Table 8:  Hyperparameters for pretraining DDCap.

Hyperparameters | Image encoder | Diffusion model
---|---|---
Training steps | 30k | 30k
Batch size | 512 | 512
Peak learning rate | 7e-6 | 1e-5
Warmup steps | 6k | 6k

Table 9:  Hyperparameters for fine-tuning DDCap on COCO captioning.

##  Appendix A Ablation on the “Karpathy” test dataset

In the main paper, we report the ablation study on “Karpathy” [32] validation
dataset. Here, we report the results on the test set in Table 10, in which we
have consistent conclusions.

#Row | Best-first inference | CAM | Length Prediction | Image-free training | C | B@4 | M | R | S
---|---|---|---|---|---|---|---|---|---
a |  |  |  |  | 20.4 | 7.1 | 18.8 | 34.3 | 12.2
b |  | $\checkmark$ |  |  | 39.0 | 10.5 | 20.3 | 38.9 | 14.1
c | $\checkmark$ |  |  |  | 45.4 | 20.4 | 26.9 | 47.2 | 21.4
d | $\checkmark$ |  | $\checkmark$ |  | 95.3 | 27.8 | 25.8 | 52.1 | 19.3
e | $\checkmark$ | $\checkmark$ |  |  | 97.2 | 27.7 | 28.0 | 53.8 | 21.8
f | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | 116.6 | 34.4 | 28.0 | 57.2 | 21.3
g | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 117.9 | 34.4 | 28.1 | 57.1 | 21.6
Table 10: Ablation study on the effectiveness of each component on the
“Karpathy” test dataset. CAM: concentrated attention mask.

##  Appendix B Hyperparameters

The hyperparameters for training DDCap on COCO captioning are shown in Table 7
with no pretraining. The hyperparameters for pretraining DDCap are shown in
Table 8.

The hyperparameters for fine-tuning DDCap on COCO captioning are shown in
Table 9.


# A Baseline on Continual Learning Methods for Video Action Recognition

##  1 Abstract

Continual learning has recently attracted attention from the research
community, as it aims to solve long-standing limitations of classic
supervisedly-trained models. However, most research on this subject has
tackled continual learning in simple image classification scenarios. In this
paper, we present a benchmark of state-of-the-art continual learning methods
on video action recognition. Besides the increased complexity due to the
temporal dimension, the video setting imposes stronger requirements on
computing resources for top-performing rehearsal methods. To counteract the
increased memory requirements, we present two method-agnostic variants for
rehearsal methods, exploiting measures of either model confidence or data
information to select memorable samples. Our experiments show that, as
expected from the literature, rehearsal methods outperform other approaches;
moreover, the proposed memory-efficient variants are shown to be effective at
retaining a certain level of performance with a smaller buffer size.

Index Terms—  Continual Learning, Video Action Recognition, Benchmark, Memory
Efficiency

##  2 Introduction

Video action recognition is a well-known problem in computer vision, given to
its wide range of applications, including surveillance [1], behavior
understanding [2], content-based retrieval [3]. It is also a complex problem,
due to the difficulty to learn spatio-temporal action patterns in a high-
dimensional space. Recent advances in deep learning, including attention-based
vision transformers [4, 5, 6, 7], have achieved unprecedented results on video
action recognition benchmarks, pushing performance of automatic methods closer
to humans’.

However, current approaches for supervised video action recognition assume a
_stationary_ data distribution, where all dataset classes and samples are
simultaneously available at training time. Recently, this hypothesis has been
put under question, giving rise to a line of research on _continual learning_
methods, designed to deal with changes in the class distribution (new classes
may become available, while others are replaced) and/or in the intra-class
data distribution. The relaxation of the stationarity assumption introduces
several novel challenges; notably, models trained on sequences of tasks tend
to focus on currently-available classes, degrading performance on previously-
seen classes — a phenomenon known as “catastrophic forgetting”; similarly, in
presence of gradual shifts in the data distribution of a certain class, models
become unable to correctly process “older” samples.

While several solutions to continual learning have been proposed, based on
architectural priors [8], knowledge distillation [9], regularization [10] or
experience replay [11], they have been mostly validated on simple image
datasets, such as MNIST, CIFAR10/100 or simplified versions of ImageNet.
Hence, it is hard to assess their suitability to more complex use cases, such
as video action recognition.

This work intends to assess whether state-of-the-art methods for continual
learning generalize to high-dimensional spatio-temporal data. To achieve this
goal, we define an experimental protocol for continual learning in video
action recognition, by selecting multiple class subsets from UCF-101 [12] and
evaluating the performance of state-of-the-art approaches at the task. We also
propose two simple method-agnostic modifications, namely, _confidence-driven
rehearsal_ and _information-driven downsampling_ , which define criteria for
the selection of samples for experience replay, in the attempt to build a more
effective buffer for retaining past knowledge with fewer samples.

Results show that the best performance is achieved by combining experience
replay with knowledge distillation from past samples, e.g., DER [13] and DER++
[13]. Additionally, we show that the proposed memory-efficient variants are
able to significantly improve the performance of even the best methods in our
benchmark, providing an efficient way to improve performance in such complex
scenario.

##  3 Related work

A continual learning problem is typically divided into a sequence of _tasks_ ,
presented one at a time. In classification, each task may require training a
model on a new set of classes or on different data distributions of
previously-seen classes. Continual learning aims to cope with catastrophic
forgetting in this scenario, using different techniques.

_Rehearsal_ methods keep a _buffer_ of samples from previous tasks, to prevent
the model from forgetting past knowledge. Methods in this category include
DER/DER++ [13], ER [11], GSS [14], FDR [15], HAL [16], GEM [17], AGEM [18] and
iCarl [19]. _Knowledge distillation_ approaches employ an earlier version of
the model (trained on previous tasks) to transfer features or to encourage the
current model to emulate past predictions. The pioneering work in this
category is LwF [9]; recent approaches have also attempted to integrate
auxiliary knowledge from unrelated tasks [20]. _Regularization_ methods, e.g.,
EWC [10], introduce loss terms to counteract modification of backbone features
in favor of new tasks. Regularization and knowledge distillation are often
employed alongside other techniques: DER/DER++ [13] regularize a cross-entropy
loss by enforcing logit similarity of buffered past samples. _Architectural_
methods, such as PackNet [21] and HAT [22], progressively extend a model’s
backbone to cope with new tasks, but require that model capacity must increase
with the number of tasks. Pruning methods [23, 24] may help mitigate this
issue.

To date, rehearsal methods show the most promising results. Unfortunately, the
presence of a buffer and the corresponding memory overhead may become
significant, if not prohibitive, when dealing with video sequences, because of
the increase introduced by the temporal dimensions.

Finally, it should be mentioned that a related problem, i.e., _class-
incremental learning_ , has already been studied in video action recognition
[25]. However, class-incremental learning — i.e., progressively showing new
classes to a classification model — only analyzes a portion of the problem
(for instance, it does not address _task-incremental_ performance) and
generally applies a pre-training on a large part (often, half [25, 26, 27]) of
the original dataset, which is not a common procedure in the literature for
continual learning.

##  4 Method

###  4.1 Problem formulation

In this work, as in most of continual learning literature, we focus on
classification tasks. Hence, let _task_
$\mathcal{T}\left(\mathcal{C}\right)\sim p(\mathcal{T})$ be a classification
problem defined on a set of classes
$\mathcal{C}=\left\\{y_{1},\dots,y_{c}\right\\}$. Given two tasks
$\mathcal{T}_{i}$ and $\mathcal{T}_{j}$, we assume that the corresponding sets
of classes are different, i.e.,
$\mathcal{C}_{i}\cap\mathcal{C}_{j}=\varnothing$: in this context, we define
the problem as of either _task-incremental learning_ (T-IL) or _class-
incremental learning_ (C-IL), based on whether knowledge of the task to which
a sample belongs is provided to the model at inference time or not,
respectively.

A _continual learning problem_ consists in a sequence of $T$ tasks
$\left(\mathcal{T}_{1},\dots,\mathcal{T}_{T}\right)$ sampled from
$p(\mathcal{T})$. A model $\mathcal{M}$ is allowed to train on each task
$\mathcal{T}_{i}$, before moving to task $\mathcal{T}_{i+1}$; at a generic
task $\mathcal{T}_{i}$, data from any other tasks cannot be accessed (unless
previously store by the model, as in the case of rehearsal methods). Given the
task sequence $\left(\mathcal{T}_{1},\dots,\mathcal{T}_{T}\right)$, model
$\mathcal{M}$, parameterized by $\bm{\theta}$, is trained to optimize a
classification objective $\mathcal{L}$ (commonly, a cross-entropy loss) on
each task at a time, while attempting to prevent performance decreases on
previous tasks.

###  4.2 Evaluation procedure

Given a _source dataset_ $\mathcal{D}_{s}$, including videos for a set of
class labels $\mathcal{C}_{s}$, we emulate a sampling of the $p(\mathcal{T})$
distribution by splitting the set of classes into random groups of $c$ classes
each. As a result, we obtain a set of tasks
$\mathbb{T}=\left\\{\mathcal{T}_{1},\mathcal{T}_{2},\dots,\mathcal{T}_{N}\right\\}$,
with $\mathcal{T}_{i}$ representing a portion of $\mathcal{D}_{s}$.

We can sample a _continual learning problem_
$\left\\{\mathcal{T}_{1},\dots,\right.$ $\left.\mathcal{T}_{T}\right\\}$ by
selecting a random subset of $T$ tasks from $\mathbb{T}$, and train each of
the methods under analysis on that problem, to guarantee a fair comparison.
This procedure is then repeated for $E$ experiments. For each experiment, a
fraction $p_{\text{test}}$ of samples from each class is left out as a test
set.

Evaluation metrics, averaged over the set of $E$ experiments, include the
following:

  * •

Accuracy in task-incremental learning: for each task, we compute test
performance using only predictions for the classes included in that task, and
average the computed accuracy scores over the set of tasks.

  * •

Accuracy in class-incremental learning: for each task, we compute test
performance using predictions for all classes, and average the computed
accuracy scores over the set of tasks.

  * •

Buffer size: for rehearsal methods, number of elements stored in the buffer
and required memory space.

###  4.3 Memory-efficient variants

Our results (see Sect. 5) show that, as expected from the literature,
rehearsal methods significantly outperform other paradigms. However, buffer
memory requirements increase significantly for video sequences. We hereby
propose two model-agnostic variants for buffer management, aimed at reducing
the number of elements to store while preserving classification accuracy.
Confidence-driven rehearsal. High-dimensionality of videos reflects on a lower
generalization power by a set of random buffered samples, due to the curse of
dimensionality. Hence, rather than attempting to model the entire
distributions of a task’s classes, an alternative lies in selecting samples on
which the model is _most confidently correct_ : while this may hinder the
recognition of under-represented class modes, it reinforces knowledge on the
most discriminant portion of the distribution, which is also expected to cover
the most density mass. In practice, a sample $(\mathbf{x},y)$ is eligible for
buffering if the model’s prediction for class $y$ is above a threshold
$\delta$.
Information-driven downsampling. Common video classification models require a
fixed input size, making it is necessary to downsample longer sequences.
Usually, such downsampling is agnostic to frame content; hence, memory
efficiency may be achieved through frame selection, so that more informative
content is stored in a smaller buffer. We propose to employ the norm of
optical flow [28] as a measure of importance, as it quantifies motion and
redundancy between consecutive frames. Formally, given a video sequence
$\mathbf{X}=\left\\{\mathbf{x}_{1},\dots,\mathbf{x}_{p}\right\\}$, we compute
motion vectors $\left\\{\mathbf{m}_{1},\dots,\mathbf{m}_{p-1}\right\\}$, such
that $\mathbf{m}_{i}=\text{OF}(\mathbf{m}_{i},\mathbf{m}_{i+1})$, where OF is
the optical flow function; then, we select the subset of frames with the
largest $L_{2}$ norm $\left\|\mathbf{m}_{i}\right\|_{2}$, ordered by their
original position in the video.

##  5 Experimental results

###  5.1 Methods

Our benchmark includes state-of-the-art continual learning methods, covering
the range of paradigms from the literature: DER [13], DER++ [13], ER [11], FDR
[15], HAL [16], GSS [14], GEM [17], AGEM [18], EWC [10] and LwF [9]111We
excluded iCarl [19] due to excessive memory requirements, as the reference
implementation required a concatenation of all samples of a task with the
current buffer.. For a fair comparison, all methods employ the same backbone
network, i.e., R(2+1)D [29]: we excluded architectural methods [21, 22] from
our analysis, as model capacity extension would lead to an unfair comparison.
All methods included in our experiments are agnostic to data modality, and can
be applied to videos without any modifications. Implementations are taken from
the Mammoth continual learning library222https://github.com/aimagelab/mammoth.

###  5.2 Dataset and task definition

As source dataset $\mathcal{D}_{s}$, we employ UCF101 [12], a video action
recognition dataset featuring 101 categories, with approximately 100-150
videos each; video duration is mostly between 2 and 10 seconds, although
longer videos may be present for certain categories. In our experiments, we
employ a subset $\mathcal{C}_{s}$ of 30 classes, by selecting distinguishable
actions, removing too similar classes and classes with less than 130 videos.
Classes are grouped into pairs (consistently with continual learning
procedures on other datasets, e.g., CIFAR10) to create the task set
$\mathbb{T}$, including $N=15$ different tasks. Then, we define a fixed set of
50 continual learning problems, each being a task sequence
$\left\\{\mathcal{T}_{1},\dots,\mathcal{T}_{T}\right\\}$ of $T=5$ tasks
randomly sampled from $\mathbb{T}$. To ensure class balance, for each class we
select a subset of 130 videos, with 100 videos used as a training set and the
rest as a test set ($p_{\text{test}}=3/10$).

###  5.3 Training details

We normalize each color channel to zero mean and unitary standard deviation,
and resize spatially to 160$\times$160 pixels. Data augmentation includes
random cropping at 128$\times$128, random horizontal flipping, and temporal
cropping by selecting 16 consecutive frames from a random point. At test time,
we apply center-cropping and process all non-overlapping 16-frame windows from
an entire video. The R(2+1)D backbone is trained from scratch for 80 epochs
per task, using the RMSProp optimizer (learning rate: $10^{-5}$, batch size:
16); in our preliminary experiments, this configuration empirically yielded
the most stable results, compared to standard SGD (which showed slow
convergence) or Adam (which, for some methods, led to exploding losses).
Method-specific hyperparameters were set to default values from the original
papers. Training was carried out on a machine with 8-core Intel Xeon Skylake
CPU, 64 GB RAM, NVIDIA V100 GPU.

Table 1: Classification accuracy in class-incremental learning (C-IL) and task-incremental learning (T-IL). Rehearsal methods are marked with a ✓. Model | Rehars. | C-IL | T-IL
---|---|---|---
DER++ [13] | ✓ | $47.87\pm 4.31$ | $90.60\pm 2.21$
DER [13] | ✓ | $39.80\pm 3.25$ | $88.80\pm 3.64$
ER [11] | ✓ | $31.80\pm 6.02$ | $90.21\pm 1.84$
FDR [15] | ✓ | $29.33\pm 3.46$ | $80.87\pm 3.12$
HAL [16] | ✓ | $25.61\pm 6.35$ | $69.40\pm 8.26$
GSS [14] | ✓ | $20.60\pm 2.97$ | $71.86\pm 3.76$
AGEM [18] | ✓ | $18.53\pm 2.70$ | $83.40\pm 1.46$
LwF [9] |  | $17.13\pm 0.99$ | $57.00\pm 4.14$
GEM [17] | ✓ | $16.13\pm 4.36$ | $72.33\pm 9.00$
EWC [10] |  | $11.87\pm 1.71$ | $50.33\pm 5.20$
Table 2: Impact of the proposed _confidence-driven rehearsal_ (CDR) variant on classification accuracy, for different buffer sizes and values of the $\delta$ threshold. | Buffer | $\bm{\delta}$ | C-IL | T-IL
---|---|---|---|---
DER++ [13] | 500 (2.2 GB) | - | $48.20\pm 2.41$ | $91.90\pm 1.74$
$0.6$ | $47.22\pm 1.39$ | $91.25\pm 2.21$
$0.7$ | $47.00\pm 3.21$ | $92.00\pm 3.09$
$0.8$ | $49.22\pm 4.40$ | $90.89\pm 2.11$
200 (0.9 GB) | - | $47.87\pm 4.31$ | $90.60\pm 2.21$
$0.6$ | $46.47\pm 4.46$ | $89.07\pm 4.39$
$0.7$ | $47.80\pm 5.07$ | $90.20\pm 3.54$
$0.8$ | $49.40\pm 4.92$ | $88.47\pm 2.19$
100 (0.5 GB) | - | $42.00\pm 2.96$ | $88.33\pm 3.90$
| $0.6$ | $41.00\pm 4.20$ | $86.25\pm 2.87$
| $0.7$ | $40.11\pm 2.50$ | $87.00\pm 0.58$
| $0.8$ | $40.00\pm 4.15$ | $88.22\pm 4.70$
DER [13] | 500 (2.2 GB) | - | $44.93\pm 4.88$ | $90.00\pm 1.11$
$0.6$ | $43.40\pm 5.15$ | $90.86\pm 3.11$
$0.7$ | $43.60\pm 2.60$ | $91.33\pm 1.61$
$0.8$ | $43.65\pm 5.21$ | $87.33\pm 3.79$
200 (0.9 GB) | - | $39.80\pm 3.25$ | $88.80\pm 3.64$
$0.6$ | $40.80\pm 5.31$ | $85.87\pm 3.34$
$0.7$ | $44.20\pm 4.65$ | $87.53\pm 3.05$
$0.8$ | $44.33\pm 5.10$ | $88.73\pm 2.30$
100 (0.5 GB) | - | $33.27\pm 3.80$ | $84.95\pm 6.71$
| $0.6$ | $35.33\pm 5.25$ | $84.67\pm 5.60$
| $0.7$ | $36.73\pm 3.47$ | $88.26\pm 6.52$
| $0.8$ | $37.67\pm 5.17$ | $86.53\pm 3.62$

###  5.4 Results

We first report the results obtained by state-of-the-art methods under
comparison. For rehearsal methods, we employ a buffer size of 200 (a standard
value from the literature), corresponding — based on our pre-processing — to
an increase in memory requirements by 940 MB. Table 1 shows each method’s
performance in terms of class-incremental and task-incremental classification
accuracy; we report mean and standard deviations over the set of continual
learning problems defined by our experimental protocol. Methods employing
experience replay perform significantly better, as expected from previous
literature results on images. It is interesting to note that the top three
methods, i.e., DER++, DER and ER achieve very similar task-incremental
accuracy, showing effectiveness in learning each individual task, but differ
significantly in the capability to retain previous knowledge. In absolute
terms, DER++ yields promising accuracy (47.87% in class-incremental learning),
considering that its performance on images (CIFAR-10) is about 65% [13], with
the same buffer size.

We then assess the impact of the proposed memory-efficient variants on the
best-performing methods only, i.e., DER and DER++. Table 2 shows the effect of
the proposed _confidence-driven rehearsal_ (CDR) technique, for different
buffer sizes and values of the $\delta$ confidence threshold. In the case of
DER, enabling CDR with a 100-element buffer pushes class-incremental
performance closer to those with buffer size 200; similarly, CDR with a
200-element buffer reaches similar performance as achieved with a 500-element
buffer. On the other hand, CDR with DER does not have a significant impact
when applied to a 500-element buffer, which may be large enough to compensate
for confidence improvements. CDR does not seem to improve task-incremental
accuracy, which is reasonable, since it acts on the buffer and mainly
addresses forgetting. Applying CDR on DER++ is less effective, though some
improvements can be seen, especially with buffer size of 200. However, it is
reasonable to assume that the stronger recovery capabilities of DER++ make the
contribution of CDR less important. For both DER and DER++, a general trend
shows that larger $\delta$ values lead to better performance; however, we
found that exceeding the $\delta=0.8$ threshold harms performance: we
hypothesize that the distribution of selected samples excessively narrows the
data distribution, worsening generalization. We then evaluate how our
_information-driven downsampling_ (IDD), based on optical flow, affects model
accuracy. Table 3 shows that enabling IDD has a positive impact on class-
incremental accuracy on DER with small buffers, while it is less effective
with larger buffer. On DER++, IDD positively affects only the usage of a
100-element buffer, possibly for similar reasons as discussed in the case of
CDR. However, in no case IDD is able to compensate for a smaller buffer size,
showing a superiority by CDR in this respect.

Table 3: Impact of the proposed _information-driven downsampling_ (IDD) variant on the performance of DER and DER++, for different buffer sizes. | Buffer | IDD | C-IL | T-IL
---|---|---|---|---
DER++ [13] | 500 (2.2 GB) |  | $48.20\pm 2.41$ | $90.90\pm 1.74$
✓ | $48.10\pm 1.80$ | $89.66\pm 2.01$
200 (0.9 GB) |  | $47.87\pm 4.31$ | $89.60\pm 2.21$
✓ | $46.00\pm 1.11$ | $89.93\pm 0.76$
100 (0.5 GB) |  | $42.00\pm 2.96$ | $88.33\pm 3.90$
✓ | $44.64\pm 3.64$ | $90.13\pm 2.50$
DER [13] | 500 (2.2 GB) |  | $44.93\pm 4.88$ | $89.00\pm 1.11$
✓ | $42.50\pm 3.82$ | $88.75\pm 3.11$
200 (0.9 GB) |  | $39.80\pm 3.25$ | $88.80\pm 3.64$
✓ | $42.20\pm 3.58$ | $87.33\pm 3.81$
100 (0.5 GB) |  | $33.27\pm 3.80$ | $84.95\pm 6.71$
✓ | $34.93\pm 4.12$ | $84.33\pm 1.66$

##  6 Conclusion

We presented a benchmark of state-of-the-art continual learning methods for
video action recognition, showing that methods designed for images are able,
to a certain extent, to generalize to videos, achieving promising performance
in class-incremental and task-incremental settings. We also propose two
memory-efficient variants for buffer sample selection, demonstrating that the
CDR variant helps to retain (or even improve) performance even when reducing
the buffer size, while the IDD variant is less effective in this regard.

Future improvements of this work will address two main research directions to
improve the proposed memory-efficient variants. First, we mean to explore
advances to the proposed _confidence-driven rehearsal_ , by integrating
mechanisms for automatic and adaptive threshold setting. Second, rather than
explicitly defining a measure for _information-driven downsampling_ (e.g.,
optical flow norm), we intend to investigate the employment of attention
mechanisms to find measures of correlation between input and model
representations, thus using the latter as a reference for importance
estimation.

## References

  * [1] I. Dave, Z. Scheffer, A. Kumar, S. Shiraz, Y. S. Rawat, and M. Shah,  “Gabriellav2: Towards better generalization in surveillance videos for action detection,”  in WACV, 2022.
  * [2] F. C. Heilbron, V. Escorcia, B. Ghanem, and J. C. Niebles,  “Activitynet: A large-scale video benchmark for human activity understanding,”  in CVPR, 2015.
  * [3] Y. Liu, K. Wang, L. Liu, H. Lan, and L. Lin,  “Tcgl: Temporal contrastive graph for self-supervised video representation learning,”  IEEE TIP, 2022.
  * [4] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, MindererM., G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  in ICML, 2021.
  * [5] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lucic, and C. Schmid,  “Vivit: A video vision transformer,”  in ICCV, 2021.
  * [6] G. Bertasius, H. Wang, and L. Torresani,  “Is space-time attention all you need for video understanding?,”  in ICML, 2021.
  * [7] J. Yang, X. Dong, L. Liu, C. Zhang, J. Shen, and D. Yu,  “Recurring the transformer for video action recognition,”  in CVPR, 2022.
  * [8] A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell,  “Progressive neural networks,”  arXiv preprint, 2016.
  * [9] Z. Li and D. Hoiem,  “Learning without forgetting,”  IEEE TPAMI, 2018.
  * [10] J. Schwarz, W. Czarnecki, J. Luketina, A. Grabska-Barwinska, Y. W. Teh, R. Pascanu, and R. Hadsell,  “Progress & compress: A scalable framework for continual learning,”  in ICML, 2018.
  * [11] M. Riemer, I. Cases, R. Ajemian, L. Liu, I. Rish, Y. Tu, and G. Tesauro,  “Learning to learn without forgetting by maximizing transfer and minimizing interference,”  in ICLR, 2019.
  * [12] K. Soomro, A. R. Zamir, and M. Shah,  “Ucf101: A dataset of 101 human actions classes from videos in the wild,”  arXiv preprint, 2012.
  * [13] P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara,  “Dark experience for general continual learning: a strong, simple baseline,”  in NIPS, 2020.
  * [14] R. Aljundi, M. Lin, B. Goujaud, and Y. Bengio,  “Gradient based sample selection for online continual learning,”  in NIPS, 2019.
  * [15] A. S. Benjamin, D. Rolnick, and K. Kording,  “Measuring and regularizing networks in function space,”  in ICLR, 2019.
  * [16] A. Chaudhry, A. Gordo, P. Dokania, P. Torr, and D. Lopez-Paz,  “Using hindsight to anchor past knowledge in continual learning,”  in AAAI, 2021.
  * [17] D. Lopez-Paz and M. A. Ranzato,  “Gradient episodic memory for continual learning,”  in NIPS, 2017.
  * [18] C. Arslan, M.A. Ranzato, M. Rohrbach, and M. Elhoseiny,  “Efficient lifelong learning with a-GEM,”  in ICLR, 2019.
  * [19] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert,  “icarl: Incremental classifier and representation learning,”  in CVPR, 2017.
  * [20] M. Boschini, L. Bonicelli, A. Porrello, G. Bellitto, M. Pennisi, S. Palazzo, C. Spampinato, and S. Calderara,  “Transfer without forgetting,”  in ECCV, 2022.
  * [21] A. Mallya and S. Lazebnik,  “Packnet: Adding multiple tasks to a single network by iterative pruning,”  in CVPR, 2018.
  * [22] J. Serrà, D. Surís, M. Miron, and A. Karatzoglou,  “Overcoming catastrophic forgetting with hard attention to the task,”  in ICML, 2018.
  * [23] S. Golkar, M. Kagan, and K. Cho,  “Continual learning via neural pruning,”  in NIPS Workshop, 2019.
  * [24] A. Mallya, D. Davis, and S. Lazebnik,  “Piggyback: Adapting a single network to multiple tasks by learning to mask weights,”  in ECCV, 2018.
  * [25] J. Park, M. Kang, and B. Han,  “Class-incremental learning for action recognition in videos,”  in ICCV, 2021.
  * [26] A. Douillard, M. Cord, C. Ollion, T. Robert, and E. Valle,  “Podnet: Pooled outputs distillation for small-tasks incremental learning,”  in ECCV, 2020.
  * [27] S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin,  “Learning a unified classifier incrementally via rebalancing,”  in CVPR, 2019.
  * [28] Gunnar Farnebäck,  “Two-frame motion estimation based on polynomial expansion,”  in SCIA, 2003.
  * [29] D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri,  “A closer look at spatiotemporal convolutions for action recognition,”  in CVPR, 2018.