# Video-Teller: Enhancing Cross-Modal Generation with Fusion and Decoupling

Haogeng Liu1,2, Qihang Fan1,2, Tingkai Liu3, Linjie Yang3, Yunzhe Tao3, Huaibo
Huang1 Ran He1, Hongxia Yang3
1MAIS & CRIPAC, Institute of Automation, Chinese Academy of Sciences, China.
2School of Artificial Intelligence, University of Chinese Academy of Sciences,
Beijing, China.
3ByteDance, Inc.
<EMAIL_ADDRESS>
fanqihang.159<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

This paper proposes Video-Teller, a video-language foundation model that
leverages multi-modal fusion and fine-grained modality alignment to
significantly enhance the video-to-text generation task. Video-Teller boosts
the training efficiency by utilizing frozen pretrained vision and language
modules. It capitalizes on the robust linguistic capabilities of large
language models, enabling the generation of both concise and elaborate video
descriptions. To effectively integrate visual and auditory information, Video-
Teller builds upon the image-based BLIP-2 model and introduces a cascaded
Q-Former which fuses information across frames and ASR texts. To better guide
video summarization, we introduce a fine-grained modality alignment objective,
where the cascaded Q-Former’s output embedding is trained to align with the
caption/summary embedding created by a pretrained text auto-encoder.
Experimental results demonstrate the efficacy of our proposed video-language
foundation model in accurately comprehending videos and generating coherent
and precise language descriptions. It is worth noting that the fine-grained
alignment enhances the model’s capabilities ($4\%$ improvement of CIDEr score
on MSR-VTT) with only 13% extra parameters in training and zero additional
cost in inference.

##  1 Introduction

Large language models (LLMs) have made significant advancements (OpenAI, 2023;
Chowdhery et al., 2022; Bai et al., 2022), and have subsequently been
extensively utilized in multimodal tasks such as image-to-text generation and
video-to-text generation (Zhang et al., 2023a; Xu et al., 2023; Huang et al.,
2023; Alayrac et al., 2022; Wang et al., 2022b), giving rise to a new class of
models called multimodal large language model (MLLM). Prior to LLMs, video
understanding models have been limited by the complexity of generated textual
descriptions (Yan et al., 2023), and downstream video-to-text tasks are
constrained to short-form generations such as single-sentence video
captioning. With the incorporation of large language models, models such as
Video-LLaMA, VideoChat and Video-ChatGPT (Zhang et al., 2023b; Li et al.,
2023c; Maaz et al., 2023) are now capable of not only generating longer and
nuanced video digests but also engaging in conversations grounded in video
content.

To leverage the power of pretrained LLMs such as LLaMA 2 (Touvron et al.,
2023) and Vicuna (Chiang et al., 2023) without incurring the forbidding cost
of retraining LLMs, MLLMs such as BLIP-2 (Li et al., 2023a) have been proposed
to integrate a trainable light-weight visual backbone with a frozen LLM via
adaptor-like mechansims (such as the Q-Former proposed in BLIP-2). The
expansion of BLIP-2 into the realm of video has quickly given rise to models
such as Video-LLaMA (Zhang et al., 2023b), which incorporates both visual and
auditory information by prompting frozen LLM with embeddings computing by two
corresponding encoders. By relying on LLMs for modality integration, however,
increases the computational cost during inference. Additionally, prior
knowledge embedded in very large language models may negatively bias the
generated video descriptions, leading to hallucination. Consequently, the
issue of enhancing the accuracy of text generation in visual-language models
while reducing the computational expense incurred has now become imminent.

Figure 1: Overview of the proposed method. Here we show the detailed
description generation (long-form text).

In order to maintain the efficiency of model training while reducing
computational overhead during inference, we propose cascaded Q-Former which
fuses multi-frame visual information with auditory information prior to
prompting LLMs, effectively reducing the computational overhead of LLM by
half. Additionally, in contrast to Video-LLaMA’s direct usage of raw audio, we
leverage ASR information from videos as the representation for the audio
modality to further enhance the model’s comprehension capabilities. Due to the
incorporation of additional modal information, to decouple crucial comments
directly using methods similar to BLIP-2 becomes more difficult. Therefore, we
propose the utilization of fine-grained modality alignment, thus enhancing the
precision of content generated by the model and alleviating the issue of
hallucination, we propose the utilization of fine-grained modality alignment
as an auxiliary training approach. Figure 1 shows a high-level overview of the
key concepts of Video-Teller.

We evaluated Video-Teller in both video-to-text generation and text-to-video
retrieval tasks. Specifically, in video captioning, our approach achieves
better results than the existing methods, such as HiTeA (Ye et al., 2022) with
a smaller amount of data, thus confirming the effectiveness of fine-grained
alignment and the integration of ASR. Additionally, in the task of long-text
generation (video summarization), we obtained better performance (measured by
BLEURT (Sellam et al., 2020)) than the baseline model with larger frozen LLMs
such as Video-LLaMA and VideoChat (Zhang et al., 2023b; Li et al., 2023c).

Overall, the main contributions of this paper are as follow.

  * •

We propose Video-Teller, a video-language foundation model that integrates
both the visual and speech information. Video-Teller reduces the computational
cost of modality fusion by incorporating visual and ASR information through a
cascaded Q-Former before the LLM.

  * •

We enhance video language learning by employing a text auto-encoder with LLM
as the decoder to decouple textual features, enabling fine-grained alignment
of video-text representations in an unsupervised manner. This approach
improves the fusion of cross-modal information thus boosts the model’s
generation capability.

  * •

In addition to providing demonstrations of Video-Teller output, we
quantitatively compare our proposed method with two representative MLLMs,
Video-LLaMA (Zhang et al., 2023b) and VideoChat (Li et al., 2023b).

##  2 Related work

The pursuit foundation models that integrate and understand multiple
modalities like vision and text have received enormous impetus from the
research community in recently years. Previously, foundation models were
largely end-to-end trainable models with architectures like dual-encoders (Jia
et al., 2021; Li* et al., 2022; Zeng et al., 2022; Bao et al., 2022), fusion-
encoders (Li et al., 2019; Yang et al., 2022; Chen et al., 2020; Su et al.,
2020; Li et al., 2020; Lu et al., 2019; Tan & Bansal, 2019), and encoder-
decoders (Yu et al., 2022; Yan et al., 2023). However, these foundation models
typically require fine-tuning of the entire model during adaptation, resulting
in significant computational expenses. The advent of BLIP-2 (Li et al., 2023a)
changed the multimodal landscape by introducing a lightweight adaptor module,
the Q-Former, which utilizes learnable queries to facilitate alignment of
multiple modalities, reducing the need for fine-tuning the pre-trained
language/visual models. Adaptor like modules have recently been remarkably
successful in allowing multimodal researchers to tap into the tremendous
natural language powers of large language models, with new models emerging
every week such as InstructBLIP, VideoChat, Video-LLaMA and Mini-GPT4 (Dai et
al., 2023; Li et al., 2023c; Zhang et al., 2023b; Zhu et al., 2023).

While adaptor modules like Q-Former can aid in merging input modalities,
explicit alignments between modalities have traditionally been done via
contrastive loss(He et al., 2020) between a single textual [CLS] token and the
other modalities. Specifically, previous models either align [CLS] tokens of
text features and visual features for sample-level modal alignment (Jia et
al., 2021; Radford et al., 2021; Yu et al., 2022), or align [CLS] tokens with
the rest of the tokens of the same modality using momentum (Yang et al.,
2022). Such approaches enable modal alignment from a global perspective (via a
single text token), but overlooks local information.

In contrast, few foundation models have explored fine-grained alignment
between tokens across modalities during pre-training, which we argue is
crucial for detailed understanding of complex input data such as videos. (Li
et al., 2022) propose LOUPE, which learns fine-grained semantic alignment from
the novel perspective of game-theoretic interactions. While (Shukor et al.,
2022) leverage hierarchical cross-modal alignment loss for fine-grained
modality alignment. These methods are highly effective, but they also tend to
be more intricate. So in this paper, we propose to use a text auto-encoder to
decouple the target text and use the decoupled feature to align with the
video’s hidden states.

##  3 Method

###  3.1 Preliminaries: BLIP-2

BLIP-2 (Li et al., 2023a) is an image-text model designed to optimize training
efficiency. It achieves this by utilizing pre-trained image encoders and
frozen large language models. To address the modality gap, BLIP-2 introduces a
lightweight Querying Transformer called Q-Former. Q-Former consists of learned
queries and a transformer module with cross-attention. The input image
initially undergoes the frozen Vision Transformer (ViT) to obtain an image
patch token sequence. Subsequently, the learned queries interact with the
image tokens through cross-attention. This process allows the input image to
be encoded into a fixed-length sequence (as demonstrated in the paper, 32
tokens). These tokens are then projected and fed into the large language model
to generate the corresponding text description of the image. Despite having
$54\times$ fewer trainable parameters, BLIP-2 outperforms Flamingo80B (Alayrac
et al., 2022). However, it should be noted that BLIP-2 is specifically
designed to handle single-image inputs and cannot be directly applied to
video-based applications.

###  3.2 Model Architecture Of Video-Teller

As illustrated in Figure 2, Video-Teller is composed of two primary
components. The first component is a video foundation model, which takes
frames and ASR texts as input and incorporates a LLM as the language decoder.
The second component is a text auto-encoder, which shares a similar structure
with the video foundation model and also employs the same LLM as the language
decoder. It is important to highlight that the text auto-encoder is
exclusively utilized during the training phase and do not incur any additional
computational cost during inference.

Figure 2: Overall architecture of the proposed model. The model consists of
two primary branches. On the right-hand side, we have the text auto-encoder
responsible for encoding the target text into a fixed-length representation
denoted as $\mathbf{R}_{text}$. Conversely, on the left-hand side, we have the
video module, which encodes the input video (comprising frames and ASR) into a
video representation that shares the same shape as the text representation.
Both of these representations are trained to reconstruct the target text by
utilizing the LLM, while they are directly aligned through the Mean Squared
Error (MSE) loss.

####  3.2.1 Video-Teller

The extension of BLIP-2 to process video input can be tackled via multiple
approaches. One approach is to encode each frame individually via image-based
BLIP-2 model, and prompt LLM directly using the concatenated frame-level
embeddings. This approach, although straightforward and powerful, incurs
considerable computational overhead as the input sequence length of the frozen
LLM is now multiple by the number of frames sampled. Additionally, this
approach relies entirely on LLM to perform modality integration and alignment.

To address the aforementioned issue, we propose a cascaded Q-Former approach
for integrating information from different video frames and texts generated by
ASR. Let $\mathbf{V}\in\mathbb{R}^{F\times C\times H\times W}$ denote the
input video frames, where $F$, $C$, $H$, and $W$ represent the number of
frames, image channels, image height, and image weight, respectively. We
utilize the vision encoder and Q-Former from BLIP-2 to individually extract
the representation $\mathbf{R}_{f}\in\mathbb{R}^{Q_{i}\times E_{i}}$ for each
frame, where $Q_{i}$ and $E_{i}$ indicate the number and size of query tokens
in the image Q-Former. The aggregated visual features are obtained by
concatenating all the image tokens from the Q-Former and are denoted as
$\mathbf{R}{vision}\in\mathbb{R}^{FQ_{i}\times E_{i}}$.

For ASR text, we first use encoder-only BERT (Devlin et al., 2019) to process
it and obtain the encoded text features. We use the last hidden states of the
text features $\mathbf{R}_{ASR}\in\mathbb{R}^{E}_{i}$ as the ASR tokens to be
combined with the visual features. In order for the combined ASR and visual
features to be consumed by the LLM, we need to further downscale the dimension
of the combined features since the total number of tokens is too large to be
directly handled by the LLM. Towards this end, we employed another transformer
to reduce the number of tokens and to fuse the information from the ASR and
visual features. We name this added transformer cascaded Q-Former and it
adopts the same BERT structure as the original Q-Former with fixed number of
query tokens to produce a fixed length of result tokens. We concat ASR tokens
$\mathbf{R}_{ASR}$ with visual tokens $\mathbf{R}_{vision}$ as input to the
cascaded Q-Former. Finally, we gain the representation of the whole video
$\mathbf{R}_{video}\in\mathbb{R}^{Q_{v}\times E_{v}}$, where $Q_{v},E_{v}$
denotes the query number and embedded dimension of cascaded Q-Former. Here we
manually split $\mathbf{R}_{video}$ into two components, where the first
component includes the first token of $\mathbf{R}_{video}$ that is used for
video-text contrastive learning, and the second component contains the
remaining 32 tokens that is used for fine-grained modality alignment and
video-grounded text reconstruction.

####  3.2.2 Text Auto-encoder

We present a novel text auto-encoder that deviates from conventional models.
Our approach leverages a frozen large language model as the decoder, while
employing BERT and text Q-Former as the encoder to encode input text into a
fixed-length representation (32 $\times$ 768). Our objective is to find the
mapping between the input text and the soft prompt (fixed-length
representation) that enables the LLM to recover the same input text. Our
experiments show that, for one-sentence captions, it is sufficient to freeze
the pretrained BERT module and only update weights of the text Q-Former.
However, multi-sentence summaries cannot be reconstructed with frozen BERT
encoder. As a result, in our subsequent experiments, the entire text encoder
(including both BERT and text Q-Former) is trainable.

###  3.3 Fine-grained modality alignment

As illustrated before, the text auto-encoder turns the input text into a fixed
length intermediate representation, which covers the crucial information for
text reconstruction. For video to text tasks, we aim at generating text with
intermediate video representation. So we take the decoupled fixed length
intermediate representation from text auto-encoder as intermediate target for
video foundation model. This means we align video with corresponding text not
only through the video-grounded text reconstruction but also the hidden
feature’s consistency, namely fine-grained modality alignment. Our proposed
method is different from previous, instead of using game-theoretic (Li et al.,
2022) or hierarchical cross-modal alignment loss (Shukor et al., 2022) as we
directly utilize the encoded tokens in our text auto-encoder as a learning
target.

##  4 Experiment

Figure 3: An example from Video-CSR. Here frames represent the video’s vision
information.

We evaluate our proposed method on several downstream tasks, including video
captioning, video summarization and video retrieval.

###  4.1 Setup

##### Datasets

We test our method’s video understanding capability on MSR-VTT (Xu et al.,
2016) and Video-CSR (Liu et al., 2023). MSR-VTT is a large-scale video
benchmark for video understanding, especially generating video captions. It
covers 10K videos and each was annotated with about 20 natural sentences.
There are 6513 videos for training and 2990 videos for testing. It should be
noted that as MSR-VTT doesn’t provide ASR, we use “none.” as the input to our
ASR branch. Video-CSR is a newly released large-scale video benchmark for
video understanding, covering roughly 5000 videos ranging from 15 seconds to 1
minute with each video annotated by human. There are 5 captions and 5
summaries for each video. Summaries are long captions that includes more
details about the subject and activities in the video. The average length of
captions is 12.71 and the average length of summaries is 62.93 in Video-CSR.
We adopt Video-CSR since its videos contain rich ASR information and is
suitable to evaluate our framework with both visual and ASR input. Videos in
this dataset can be divided into two parts, one part with rich ASR information
while the other part with little ASR infromation. The ratio of videos with
ample and limited ASR information is approximately 1 to 2. In cases that the
video’s ASR conveys only little information, we use the text “none.” as the
ASR input.

For experiments on MSR-VTT, we use the WebVid-2M (Bain et al., 2022) and CC3M
(Sharma et al., 2018) (used as static video) for pre-training. While in
experiments on Video-CSR, we collect a pre-training dataset consists of 100K
YouTube videos, where each video has 5 captions and 5 summaries generated by
GPT-3.5 with the videos’ metadata from YouTube, which covers description, ASR,
comments and so on. The generated captions contains the key information of the
video, but may miss some essential visual information if it is not described
by the video’s metadata. An example of a video from Video-CSR is shown in
Figure 3.

##### Model Configurations

We construct our model directly from the pre-trained BLIP-2, leveraging its
extensive prior knowledge of images. For text auto-encoder, we use
$\mathbf{BERT_{base}}$ to process the raw input. And then we use the first
five layers of $\mathbf{BERT_{base}}$ as the text Q-Former. 33 learnable
queries are used to project the text into the fix-length representation with
cross-attention where the first one is for video-text contrastive learning.
For the cascaded Q-Former, we construct it with the first 5 layers of the pre-
trained $\mathbf{BERT_{base}}$ with 33 learnable query tokens. Empirically, we
find that using more layers on the cascaded Q-Former and text Q-Former will
deteriorate the performance. For ASR, the weights of $\mathbf{BERT}$ in its
encoder is shared with the text auto-encoder. Totally, there are about 307M
trainable parameters containing the image Q-Former, video Q-Former, text
encoder, text Q-Former and a few linear projection layers. We apply pretrained
opt-6.7b (Zhang et al., 2022) as our frozen LLM. For each video input, we
sampled 8 frames as the vision representation.

##### Training and Evaluation

We adopt a three-stage training process for our model. For the first stage, we
pretrain the text auto-encoder. We use 32 Tesla-V100 GPUs, with a batch size
of 8 on each individual GPU, and conducted training for two epochs. For the
second stage, we train the whole model on video captioning or video
summarization using 64 Tesla-V100 GPUs, with a batch size of 8 for video
captioning and 6 for video summarization on each individual GPU. We train the
model for another 2 epochs. For the third stage, we further finetune the model
with 32 Tesla-V100 GPUs for 3 epochs on MSR-VTT and 10 epochs on Video-CSR.

Table 1: Results for video caption on MSR-VTT. w/o A means without fine-grained modality alignment. SCST means Self-Critical Sequence Training (Rennie et al., 2017). Video-Teller achieves similar performance with its counterparts but uses much less PreTraining Data. Model | #PT Data | B@4 | M | R | C
---|---|---|---|---|---
HiTeA (Ye et al., 2022) | 17M | 49.2 | 30.7 | 65.0 | 65.1
VideoCoCa (Yan et al., 2023) | 3B | 53.8 | - | 68.0 | 73.2
GIT (Wang et al., 2022a) | 0.8B | 53.8 | 32.9 | 67.7 | 73.9
GIT2 (Wang et al., 2022a) | 12.9B | 54.8 | 32.9 | 68.2 | 75.9
Video-Teller w/o A | 4.5M | 47.9 | 32.4 | 65.5 | 68.0
Video-Teller | 4.5M | 49.2 | 33.0 | 66.4 | 72.0
Video-Teller (SCST) | 4.5M | 49.4 | 33.4 | 67.0 | 74.5

###  4.2 Video Captioning

We conducted experiment on two datasets, MSR-VTT and Video-CSR. As Video-CSR
is a newly released dataset, we implement baselines with VideoCoCa (Yan et
al., 2023) by adding a similar ASR fusion module to facilitate it to extract
information from both frames and ASR text. We initialize VideoCoCa with CoCa
pretrained on LAION-5B (Schuhmann et al., 2022). We name the VideoCoCa model
with the added ASR fusion module VideoCoCa (ASR). Results on MSR-VTT can be
found in Table 1 and results on Video-CSR can be found in Table 2. All results
are reported on BLEU-4 (B@4), METEOR (M), CIDEr (C) and ROUGE-L (R).

We also test the model applying self-critical sequence training, which is a
REINFORCE algorithm that directly optimize the CIDEr metric (Rennie et al.,
2017). Those results demonstrate Video-Teller’s strong video description
though using limited videos for pre-training compared with other models.

Table 2: Results for video caption on Video-CSR. w/o A means without fine-grained modality alignment. For Zero-Shot, both models are trained 100K videos from pretraining dataset. |  | Finetuned | Zero-Shot
---|---|---|---
Model | ASR | B@4 | M | R | C | B@4 | M | R | C
VideoCoCa | No | 6.2 | 11.0 | 23.8 | 18.7 | 2.1 | 10.7 | 18.7 | 5.7
VideoCoCa (ASR) | Yes | 7.1 | 11.9 | 25.0 | 22.1 | 2.8 | 11.4 | 19.7 | 9.1
Video-Teller w/o A | Yes | 7.2 | 12.7 | 26.3 | 21.9 | 3.5 | 12.2 | 22.2 | 13.9
Video-Teller | Yes | 10.4 | 14.7 | 28.7 | 30.7 | 5.6 | 14.2 | 24.0 | 19.9

###  4.3 Video Summarization

We evaluate performance of video summarization on Video-CSR. This dataset
covers 5000 videos. We randomly choose 1200 videos for testing, while the rest
are used for fine-tuning the models. It is important to mention that the ratio
of videos with ample and limited ASR information in the test set and training
set is both approximately 1 to 2. We compare with four baseline models:
VideoCoCa, VideoCoCa (ASR), Video-LLaMA (Zhang et al., 2023b), and VideoChat
(Li et al., 2023b). Among them, VideoCoCa and Video-LLaMA only uses visual
input and VideoCoCa (ASR) and VideoChat uses both visual and ASR input. We
also evaluate both zero-shot and finetuned performance. For the metrics, we
choose BLEURT (Sellam et al., 2020) as the main metrics. We also report
results with CIDEr (C) and ROUGE-L (R). Results can be found in Table 3. After
calculating the metrics, we randomly select 20 generated sentences from
different models. We manually ranked each result to assess their level of
consistency with various indicators and find that semantic-related evaluation
metrics such as BLEURT (Sellam et al., 2020) are more suitable than metrics
based on string matching for long text evaluation. The results also indicate
that Video-Teller has achieved certain advantages in video summarization
compared to other models.

Table 3: Results for video summarization on Video-CSR. w/o A means without fine-grained modality alignment. |  | Finetuned |  | Zero-Shot |
---|---|---|---|---|---
Model | #PT Data | BLEURT | R | C | BLEURT | R | C
VideoCoCa | 0.5M | 29.6 | 19.3 | 2.9 | 28.8 | 18.6 | 3.0
VideoCoCa (ASR) | 0.5M | 36.8 | 22.4 | 9.5 | 31.0 | 20.1 | 8.1
Video-LLaMA | - | - | - | - | 39.3 | 19.2 | 2.1
VideoChat | - | - | - | - | 42.8 | 22.6 | 15.2
Video-Teller w/o A | 0.5M | 45.2 | 22.4 | 9.7 | 41.2 | 20.1 | 7.1
Video-Teller | 0.5M | 47.1 | 23.5 | 11.2 | 43.3 | 21.3 | 9.0

###  4.4 Ablation Experiments

While our model exhibits commendable performance in the text generation task,
we remain skeptical about the extent to which the inclusion of ASR and fine-
grained alignment can genuinely enhance its performance. Consequently, we
undertake ablation experiments and assess them using Video-CSR and MSR-VTT
dataset. Results on Video-CSR can be found in Table 4. Results for MSR-VTT are
in Table 4. As MSR-VTT doesn’t provid ASR, we only test the influence of
alignment and contrastive loss. From the results, we can observe that on
Video-CSR our model’s performance declines when either ASR or fine-grained
alignment is absent. This demonstrates the effectiveness of our approach on
real-world scenario datasets. Result on MSR-VTT captioning also shows the
fine-grained alignment improves the performance.

Table 4: Results for video captioning. Here w/o A means without align while w/o C means without contrastive learning. We also use w/o ASR represents without ASR. |  | Finetuned |  |  | Zero-Shot |
---|---|---|---|---|---|---
Model | B@4 | M | R | C | B@4 | M | R | C
Results on Video-CSR |  |  |  |  |  |  |  |
Video-Teller w/o ASR | 4.7 | 9.8 | 22.8 | 13.1 | 2.2 | 10.5 | 19.7 | 13.4
Video-Teller w/o A | 7.2 | 12.7 | 26.3 | 21.9 | 21.9 | 12.2 | 22.2 | 13.9
Video-Teller w/o C | 10.3 | 14.7 | 28.5 | 30.4 | 5.6 | 14.1 | 24.0 | 19.8
Video-Teller | 10.4 | 14.7 | 28.7 | 30.7 | 5.6 | 14.2 | 24.0 | 19.9
Results on MSR-VTT |  |  |  |  |  |  |  |
Video-Teller w/o A | 47.9 | 31.5 | 65.3 | 69.6 | 12.4 | 17.6 | 36.3 | 24.6
Video-Teller w/o C | 48.4 | 32.9 | 65.7 | 70.9 | 13.4 | 18.7 | 38.5 | 25.4
Video-Teller | 49.2 | 33.0 | 66.4 | 72.0 | 15.6 | 19.6 | 40.1 | 26.9

###  4.5 Video Retrieval results

Though achieving strong results on video generation task with fine-grained
modality alignment, it still needs to be verified whether the method will have
an impact on the accuracy of retrieval. Through ablation experiments, it’s
demonstrated that fine-grained modality alignment enhances the cross-modal
generation capability of the model without affecting its retrieval accuracy.
Result can be found in Table 5. The model is pre-trained with WebVid-2M (Bain
et al., 2022) and CC3M (Sharma et al., 2018).

Table 5: Results for video retrieval on MSR-VTT, where w/o A means without fine-grained alignment. |  | Finetuned | Zero-Shot
---|---|---|---
Model | #PT Data | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
Video-Teller w/o A | 4.5M | 33.1 | 57.8 | 65.9 | 41.0 | 67.1 | 77.3
Video-Teller | 4.5M | 33.5 | 57.5 | 66.1 | 40.7 | 67.5 | 77.7

From above result, we have demonstrated that fine-grained alignment can
enhance the generation capability of the model without affecting video
retrieval task.

##  5 Analysis

As shown before, we find that Video-Teller, with limited video data for pre-
training, achieves strong performance both on video summarization and video
captioning. We will analyze the improvements to the model that our proposed
method brings in terms of hallucination of description.

Similar to LLM, Video-Teller is bothered by hallucination. it tends to fill in
incorrect information, especially when generating detailed description. We
evaluated the severity of different models’ hallucination through manual
assessment. Specifically, we randomly selected 50 generated results from the
test set of Video-CSR and categorized them into three types: no hallucination,
slightly hallucination, and severe hallucination, based on the comparison
between the generated content and manually annotated content. The ratios for
each model are illustrated in Table 7. We also provide criteria for rating
different levels of hallucination in Table 6.

Table 6: Criteria of rating hallucination. Hallucination level |  Description
---|---
no hallucination |  The predicted summary delineates events that are entirely congruous with the actual video, albeit with potential omissions in its depiction.
moderate hallucination |  The predicted summary portrays events that are largely congruent with the actual video, albeit with some minor deviations in certain details.
severe hallucination |  The predicted summary depicts events that are starkly divergent from the actual video.
Table 7: Hallucination ratio of different models. w/o A means trained without fine-grained alignment. Model | no hallucination | moderate hallucination | severe hallucination
---|---|---|---
VideoCoCa (ASR) | 0.60 | 0.26 | 0.14
Video-LLaMA | 0.26 | 0.40 | 0.34
Video-Teller w/o A | 0.40 | 0.26 | 0.34
Video-Teller | 0.56 | 0.24 | 0.20

Based on the findings presented in Table 7, it is evident that models
utilizing LLMs face a more pronounced issue of hallucination. This can be
attributed to the limited information provided by the visual encoder, forcing
the LLM to heavily rely on imaginative processes to complete the description.
In contrast, VideoCoCa, which does not employ an LLM, exhibits a relatively
milder form of hallucination. This difference can be explained by VideoCoCa’s
tendency to generate shorter descriptions when faced with insufficient
information, thereby reducing the generation of extraneous content.
Conversely, the extensive prior knowledge of the LLM engenders the production
of erroneous information. With our fine-grained alignment, Video-Teller is
able to significantly reduce the rate of hallucination, with the no
hallucination rate increased from 0.40 to 0.56, and severe hallucination rate
diminished from 0.34 to 0.20. This indicates that the fine-grained alignment
enforces the encoded video tokens $\mathbf{R}_{video}$ to be more relevant to
the semantics of the target caption/summary and thus reduces hallucination. We
provide a demo which shows model trained without fine-grained alignment
suffers more from hallucination in Figure 4 in the appendix.

As we could see in Figure 4, this video belongs to the category with a high
ASR (Automatic Speech Recognition) content. Therefore, in order to generate
its summary, it is necessary to make better use of the ASR information. From
the ASR information, we can infer that this video discusses the relevant
aspects of the decline in clean energy prices, just as predicted by Video-
Teller. However, we can observe that without alignment, the model’s
description includes specific price changes that cannot be extracted from the
video.

##  6 Conclusion

In this paper, we propose Video-Teller, a robust video-text foundation model
that attains impressive performance on video-to-text tasks, encompassing both
concise and comprehensive descriptions. Video-Teller leverages the rich speech
information contained in the videos to enhance the model’s understanding of
the video. Simultaneously, it utilizes pre-trained visual models and large
language models to reduce training costs while maintaining the impressive
performance. Furthermore, we employ a standalone text auto-encoder to learn
the proper intermediate language tokens that guides the learning of the video
foundation model, which boosts the decoupling of the fused multi modality
information. Extensive experimental results demonstrate the impressive
performance of our approach with light-weighted training, effectively reducing
model hallucinations (no hallucination rate wit a gain from 40% to 56%) and
significantly improving the accuracy of model descriptions (BLEURT score
increased from 41.2 to 43.3).

## References

  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Bai et al. (2022) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, et al.  Constitutional ai: Harmlessness from ai feedback, 2022.
  * Bain et al. (2022) Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval, 2022.
  * Bao et al. (2022) Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei.  VLMo: Unified vision-language pre-training with mixture-of-modality-experts.  In _NeurIPS_ , 2022.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In _ECCV_ , 2020.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, et al.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, et al.  Palm: Scaling language modeling with pathways, 2022.
  * Dai et al. (2023) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  2020\.
  * Huang et al. (2023) Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, et al.  Language is not all you need: Aligning perception with language models, 2023.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, et al.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _ICML_ , 2021.
  * Li et al. (2022) Juncheng Li, Xin He, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang, Qi Tian, and Siliang Tang.  Fine-grained semantically aligned vision-language pre-training, 2022.
  * Li et al. (2023a) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023a.
  * Li et al. (2023b) KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao.  Videochat: Chat-centric video understanding, 2023b.
  * Li et al. (2023c) KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao.  Videochat: Chat-centric video understanding.  _arXiv preprint arXiv:2305.06355_ , 2023c.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language, 2019.
  * Li* et al. (2022) Liunian Harold Li*, Pengchuan Zhang*, Haotian Zhang*, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao.  Grounded language-image pre-training.  In _CVPR_ , 2022.
  * Li et al. (2020) Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In _ECCV_ , 2020.
  * Liu et al. (2023) Tingkai Liu, Yunzhe Tao, Haogeng Liu, Qihang Fan, Ding Zhou, Huaibo Huang, Ran He, and Hongxia Yang.  Video-csr: Complex video digest creation for visual-language models, 2023.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In _NeurIPS_ , 2019.
  * Maaz et al. (2023) Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan.  Video-chatgpt: Towards detailed video understanding via large vision and language models, 2023.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report, 2023.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Rennie et al. (2017) Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel.  Self-critical sequence training for image captioning, 2017.
  * Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  Laion-5b: An open large-scale dataset for training next generation image-text models, 2022.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur P. Parikh.  Bleurt: Learning robust metrics for text generation, 2020.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 2556–2565, 2018.
  * Shukor et al. (2022) Mustafa Shukor, Guillaume Couairon, and Matthieu Cord.  Efficient vision-language pretraining with visual concepts and hierarchical alignment.  _arXiv preprint arXiv:2208.13628_ , 2022.
  * Su et al. (2020) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  In _ICLR_ , 2020.
  * Tan & Bansal (2019) Hao Tan and Mohit Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  In _EMNLP_ , 2019.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, et al.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * Wang et al. (2022a) Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang.  Git: A generative image-to-text transformer for vision and language, 2022a.
  * Wang et al. (2022b) Peng Wang, An Yang, Rui Men, Junyang Lin, et al.  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework, 2022b.
  * Xu et al. (2023) Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, Guohai Xu, Ji Zhang, Songfang Huang, Fei Huang, and Jingren Zhou.  mplug-2: A modularized multi-modal foundation model across text, image and video.  In _ICML_ , 2023.
  * Xu et al. (2016) Jun Xu, Tao Mei, Ting Yao, and Yong Rui.  Msr-vtt: A large video description dataset for bridging video and language.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 5288–5296, 2016.
  * Yan et al. (2023) Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu.  Videococa: Video-text modeling with zero-shot transfer from contrastive captioners, 2023.
  * Yang et al. (2022) Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang.  Vision-language pre-training with triple contrastive learning.  In _CVPR_ , 2022.
  * Ye et al. (2022) Qinghao Ye, Guohai Xu, Ming Yan, Haiyang Xu, Qi Qian, Ji Zhang, and Fei Huang.  Hitea: Hierarchical temporal-aware video-language pre-training, 2022.
  * Yu et al. (2022) Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models, 2022.
  * Zeng et al. (2022) Yan Zeng, Xinsong Zhang, and Hang Li.  Multi-grained vision language pre-training: Aligning texts with visual concepts.  In _CVPR_ , 2022.
  * Zhang et al. (2023a) Ao Zhang, Hao Fei, Yuan Yao, Wei Ji, Li Li, Zhiyuan Liu, and Tat-Seng Chua.  Transfer visual prompt generator across llms, 2023a.
  * Zhang et al. (2023b) Hang Zhang, Xin Li, and Lidong Bing.  Video-llama: An instruction-tuned audio-visual language model for video understanding.  _arXiv preprint arXiv:2306.02858_ , 2023b.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_ , 2022.
  * Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_ , 2023.

##  Appendix A Detailed algorithm description

Three training stages are scheduled in our experiment. We will briefly show
this with an algorithm table. Here we take video caption task as example.
Where 1 shows stage 1 while 2 shows stage 2.

Algorithm 1 Stage 1

1: Dataset: 1.9M sentences, covering 0.5M summaries (long sentences) and 1.5M
captions (short sentences). It should be noted that no testset’s sentence are
included.

2: Model: Text Auto-Encoder, mainly covering BERT and Text Q-Former (5
layers). About 150M trainable parameters, in which 110M are shared with the
vision part.

3: Input: $text$

4: repeat

5: Feature Decoupling: $\mathbf{R}_{text}$ =
$\mathbf{QF}_{text}$(BERT($text$)), where $\mathbf{QF}_{text}$ means Text
Q-Former

6: Loss caculation:
$\mathcal{L}_{text}=\mathbf{LLM}(\mathbf{Proj}(\mathbf{R}_{text}),text).Loss$,
where $\mathbf{Proj}$ and $\mathbf{LLM}$ means linear projection and large
language model.

7: Loss back-propagation and weights updating

8: until convergence

Algorithm 2 Stage 2

1: Dataset: WebVid2M, covering 2M valid (video, text) pairs; CC3M, covering
2.5M valid (image, text) pairs which is considered as static video.

2: Model: Text Auto-Encoder, mainly covering BERT and Text Q-Former (5
layers). Video foundation model, mainly covering Image Q-Former, Video
Q-Former (5 layers) and ASR BERT. Here ASR BERT are shared by both parts.
About 307M trainable parameters, in which only 40M are exclusively belongs to
text Auto-Encoder.

3: Input: $(text,asr,frames)$, noting we use ’none.’ to represent the videos’
asr for the pre-training dataset.

4: repeat

5: # A. Text Auto-Encoder pipeline.

6: Feature Decoupling: $\mathbf{R}_{text}$ =
$\mathbf{QF}_{text}$(BERT($text$)), where $\mathbf{QF}_{text}$ means Text
Q-Former

7: Loss caculation:
$\mathcal{L}_{text}=\mathbf{LLM}(\mathbf{Proj}(\mathbf{R}_{text}),text).Loss$,
where $\mathbf{Proj}$ and $\mathbf{LLM}$ means linear projection and large
language model.

8:

9: # B. Video pipeline.

10: ASR processing: $\mathbf{R}_{asr}$ = BERT($asr$),

11: Video frames processing: $\mathbf{R}_{vision}$ =
$\mathbf{QF}_{img}$(ViT($frames$)), where $\mathbf{QF}_{img}$ means iamge
Q-Former in BLIP-2.

12: Video feature decoupling: $\mathbf{R}_{video}$ =
$\mathbf{QF}_{video}$(Concat($\mathbf{R}_{vision},\mathbf{R}_{asr}$))

13: Loss caculation: $\mathcal{L}_{contra}$ is just caculated by the first
token from $\mathbf{R}_{video},\mathbf{R}_{text}$
$\displaystyle\mathcal{L}_{video}=\mathbf{LLM}(\mathbf{Proj}(\mathbf{R}_{video}),text).Loss$
$\displaystyle\mathcal{L}_{align}=\mathrm{MSE}(\mathbf{R}_{video},\mathbf{R}_{text})$

14: # C. Loss combination.

15: Total training loss:
$\mathcal{L}_{train}=\mathcal{L}_{video}+\mathcal{L}_{contra}+\lambda_{1}\mathcal{L}_{text}+\lambda_{2}\mathcal{L}_{align}$

16: Loss back-propagation and weights updating

17: until convergence

The training stage three is the same with stage 2, where the only difference
is the dataset.

##  Appendix B Demos

Figure 4: A case shows improvement of hallucination using fine-grained
modality alignment.


More Backgrounds and Related Works

More real-world examples on open environment challenges

Autonomous vehicles. A self-driving car operates in an open environment where
it can encounter unseen scenarios during training. (1) Many factors (e.g.
traffic situation, weather) can change from time to time. The autonomous
driving model must continuously adapt to these situations, which leads to the
challenge of distribution drifts. (2) It may come across unique traffic
situations, new traffic signs, or unusual pedestrian behaviors that are not
present in its training data. These belong to the challenge of outliers or new
classes. (3) Old cameras can malfunction while new cameras can be installed,
which constitutes the challenge of incremental/decremental feature space.

Real-time health monitoring. Consider a wearable device like a smartwatch that
monitors a user’s health conditions for disease prediction. (1) The wearer’s
health condition might change, reflecting alterations in the underlying data
distributions and causing the challenge of distribution drifts. (2) Moreover,
the user might develop a medical condition that is absent during the initial
training of the model, introducing the problem of outliers or novel classes.
(3) Additionally, software updates in the device can modify the methods of
data collection, leading to potential changes in the input feature space
(incremental/decremental feature space).

Financial fraud detection. In the financial industry, detecting fraudulent
transactions is critical. (1) Over time, typical consumer behaviors may
change, causing a shift in transaction data distributions and thus, the
challenge of distribution drifts. (2) Fraudsters may devise new fraudulent
strategies that have not been captured in the existing data, introducing the
issue of outliers or novel classes. (3) With the continuous evolution of new
financial technology (e.g., mobile payment methods, cryptocurrency
transactions), the types of data collected could change, leading to changes in
the input feature space (incremental/decremental feature space).

Detailed Discussions of Incremental Learning Algorithms sec:il_detail

From the perspective of regularization, various approaches have been proposed.
Elastic Weight Consolidation (EWC) kirkpatrick2017overcoming penalizes the
change in crucial parameters from previous windows based on the Fisher
Information Matrix. PathInt zenke2017continual estimates importance by tracing
the tracks of parameter value at each window. If a parameter significantly
influences the loss of previous windows, it is assigned greater penalty in the
current window. Memory Aware Synapses (MAS) aljundi2018memory calculates the
importance factor by online accumulation of gradient magnitude. Rwalk
chaudhry2018riemannian revises EWC with KL divergence regularization for
memory efficiency and combines it with PathInt to compute each parameter’s
importance. LwF li2017learning integrates the prediction of the previous model
into the current loss function, which reduces the over-confidence towards seen
classes of the current window. LwM dhar2019learning further introduces
attention distillation loss to force the attention heat map of the current
model similar to that of the previous model in vision tasks. DMC
zhang2020class proposes to perform distillation on previous models and the
current model to balance the supervision between previous and current windows.
smith2021always improves data-free knowledge distillation to generate examples
from previous models and trains the current model with more balanced datasets.
DDE hu2021distilling refers to casual inference and proposes to regularize by
representation distance of previous models. SPB wu2021striking regularizes the
training by penalizing representation change with contrastive loss.
toldo2022bring explores the feature drifts and revives features of old classes
during training.

From the perspective of storing exemplars, iCaRL rebuffi2017icarl iteratively
selects exemplars that are close to the mean representation of corresponding
classes. Rwalk chaudhry2018riemannian chooses exemplars which are hard to
classify. BiC wu2019large applies a similar exemplar selection strategy but
uses the exemplars as a validation dataset to train a bias-correcting linear
model for the last layer. LUCIR hou2019learning utilizes exemplars to push the
embedding of current classes away from previous similar classes. It also
normalizes the last classification layer to reduce the bias towards the
current window. Der yan2021dynamically decouples the representation learning
with classifier learning. For a new window, it trains a new encoder with
encoders of previous windows fixed, and stores exemplars as a balanced dataset
to train the final classification layer. RMM liu2021rmm utilizes reinforcement
learning to allocate the memory for exemplars of different classes.

From the perspective of parameter isolation, PackNet mallya2018packnet prunes
the neural network for new windows while keeping important parameters frozen.
Expert Gate aljundi2017expert maintains the model of each window and trains a
gating function to select the appropriate expert model. However, real-world
streams can be infinite, making it unfeasible to continually expand network
architecture or isolate neurons.

Drift Detection sec:drift_detection Drift detection plays an important role in
open environment learning. By identifying the occurrence, type, and magnitude
of distribution drifts, algorithms can be modified and updated accordingly.

Drift detection methods fall into two broad categories: data drift detection
and concept drift detection. Denote features as $X$ and labels as $Y$. Data
drift estimates the change of $(X,Y)$, while concept drift estimates the
change of the learned function $f:\mathcal{X}\rightarrow\mathcal{Y}$.
Therefore, the input of data drift detector is usually the dataset itself,
while concept drift detectors mostly require the error rate of a model trained
on the data stream.

In terms of data drift detection, lu2018learning proposes to compute the
similarity between new and old data with certain distance metrics, and then
conduct a hypothesis test to decide whether a drift occurs. Frequently
employed statistical tests include the Kolmogorov-Smirnov test
berger2014kolmogorov, Wilcoxon test, and Kullback-Leibler divergence
joyce2011kullback. kifer2004detecting proposes to apply the Kolmogorov-Smirnov
test between adjacent windows. Distance metrics, such as the Hellinger
distance, are often used to determine the similarity between different
distributions. However, without an appropriate window size, potential drift
might go unnoticed. Adaptive time window-based solutions, like ADaptive
WINdowing (ADWIN) bifet2007learning, are proposed to detect distribution drift
using an adaptive window size.

In terms of concept drift detection, a typical technique is to closely observe
the model’s effectiveness. For instance, a substantial decrease in the model’s
accuracy may signal a change in the data distribution. Algorithms like the
Drift Detection Method (DDM) gama2004learning and Early Drift Detection Method
(EDDM) baena2006early signal drift when the error rate falls out of control.
However, the task of drift detection becomes complex when the ground truth is
not readily available.

Evaluating drift detection methods can be challenging due to the uncertainty
of drift occurrence in real-world datasets lu2018learning. Thus, many works
wang2015concept,losing2016knn,lu2016concept,bu2016pdf experiment on manually
designed datasets such as rotating hyperplane hulten2001mining and Usenet
katakis2008ensemble. Real-world datasets, like Electricity Prices
harries1999splice and Covertype asuncion2007uci, are also used in a
prequential setting, where data is first tested then trained in each window.

We provide a summary of 16 popular drift detection algorithms in Table
tbl:driftdetection. The comparison includes detector type, required input
data, data type (categorical or numerical), applicable task, and compatibility
with stream or batch data processing. We have made the following findings:

  * •

First, some approaches are designed to detect drifts on 1-D data, e.g. KS
statistic, ADWIN bifet2007learning and HDDM frias2014online. To adapt them to
multi-dimensional data, we have to conduct drift detection in each dimension
separately.

  * •

Second, many concept drift detectors are designed only for classification or
even binary classification tasks. Only PERM harel2014concept and EIA
baier2020handling are applicable to regression tasks. However, EIA
baier2020handling compares the current model with a simple model of predicting
with the latest target, which is not quite reasonable. Thus, to detect concept
drift in regression tasks, PERM harel2014concept is the most suitable choice.
One can also consider adapting the error rate to the regression loss in DDM
gama2004learning or other methods.

  * •

Third, some approaches like KS statistic are designed to compare batches. To
adapt to data streams, we can divide the streams into windows and run the
methods on adjacent windows.

Table : Summary of drift detection methods. Method | Detector type | Input | Data type | Applicable task | Stream | Batch
---|---|---|---|---|---|---
DDM gama2004learning | Concept drift | Error rate | Categorical / Numerical | Classification |  51 |  55
EDDM baena2006early | Concept drift | Error rate | Categorical / Numerical | Classification |  51 |  55
ADWIN accuracy bifet2007learning | Concept drift | Error rate | Categorical / Numerical | Classification |  51 |  55
FW-DDM liu2017fuzzy | Concept drift | Error rate | Categorical / Numerical | Classification |  51 |  55
ECDD ross2012exponentially | Concept drift | Error rate | Categorical / Numerical | Classification |  51 |  55
LFR wang2015concept | Concept drift | Error rate | Categorical / Numerical | Binary classification |  51 |  55
MD3 sethi2015don | Concept drift | Margin data rate | Categorical / Numerical | Binary classification |  51 |  55
PERM harel2014concept | Concept drift | Test loss | Categorical / Numerical | Classification / Regression |  55 |  51
EIA baier2020handling | Concept drift | Error intersection | Categorical / Numerical | Classification / Regression |  55 |  51
KS statistic | Data drift | 1-D data | Categorical / Numerical | Classification / Regression |  55 |  51
ADWIN bifet2007learning | Data drift | 1-D data | Numerical | Classification / Regression |  51 |  55
HDDM frias2014online | Data drift | 1-D data | Numerical | Classification / Regression |  51 |  55
CDBD lindstrom2013drift | Data drift | Confidence score | Categorical / Numerical | Classification / Regression |  55 |  51
HDDDM ditzler2011hellinger | Data drift | Multi-dimension data | Categorical / Numerical | Classification / Regression |  55 |  51
kdq-Tree dasu2006information | Data drift | Multi-dimension data | Categorical / Numerical | Classification / Regression |  55 |  51
PCA-CD qahtan2015pca | Data drift | Multi-dimension data | Numerical | Classification / Regression |  55 |  51

##  1 Additional Experiments

###  Experiments on all 55 datasets

We show the experimental results of all 55 datasets in Table 1. As we can see,
no algorithm consistently outperforms others. The recommendations on different
open environment cases are shown in the main paper.

Table : Test loss / test error of stream learning algorithms on different
characters of real-world datasets. Lower value indicates better result. We
repeat all experiments for three times with different random seeds.

2.1!  Task Drift Anomaly Missing value Dataset Naive-NN EWC LwF iCaRL SEA-NN
Naive-DT Naive-GBDT SEA-DT SEA-GBDT Classification Medium high High Low Room
occupancy 0.214$\pm$0.004 0.207$\pm$0.003 0.207$\pm$0.014 0.136$\pm$0.021
0.207$\pm$0.037 0.198$\pm$0.006 0.181$\pm$0.004 0.191$\pm$0.004
0.151$\pm$0.002 Classification Medium high Medium high Low Electricity price
0.311$\pm$0.012 0.311$\pm$0.012 0.311$\pm$0.012 0.286$\pm$0.013
0.332$\pm$0.022 0.272$\pm$0.001 0.256$\pm$0.000 0.263$\pm$0.009
0.264$\pm$0.001 Classification Medium high Medium high Low Covertype
0.320$\pm$0.006 0.320$\pm$0.004 0.320$\pm$0.003 0.349$\pm$0.006
0.455$\pm$0.008 0.382$\pm$0.001 0.380$\pm$0.000 0.432$\pm$0.006
0.422$\pm$0.000 Classification Medium high Medium low Low NOAA 0.717$\pm$0.020
0.716$\pm$0.021 0.718$\pm$0.020 0.695$\pm$0.011 0.757$\pm$0.008
0.556$\pm$0.003 0.568$\pm$0.001 0.594$\pm$0.004 0.625$\pm$0.029 Classification
Medium high Medium high Low RSSI 0.379$\pm$0.003 0.379$\pm$0.003
0.379$\pm$0.003 0.331$\pm$0.003 0.431$\pm$0.032 0.372$\pm$0.001
0.277$\pm$0.000 0.309$\pm$0.004 0.256$\pm$0.000 Classification Medium low
Medium high Low INSECTS (incremental, reoccurring, balanced) 0.269$\pm$0.006
0.269$\pm$0.006 0.269$\pm$0.006 0.306$\pm$0.005 0.321$\pm$0.007
0.329$\pm$0.001 0.306$\pm$0.000 0.291$\pm$0.004 0.291$\pm$0.002 Classification
Medium low Medium high Low INSECTS (abrupt, imbalanced) 0.202$\pm$0.003
0.202$\pm$0.003 0.202$\pm$0.003 0.234$\pm$0.001 0.202$\pm$0.002
0.366$\pm$0.000 0.328$\pm$0.000 0.252$\pm$0.001 0.256$\pm$0.001 Classification
Low Low Low Safe driver 0.036$\pm$0.000 0.036$\pm$0.000 0.036$\pm$0.000
0.039$\pm$0.000 0.036$\pm$0.000 0.081$\pm$0.000 0.036$\pm$0.000
0.040$\pm$0.000 0.036$\pm$0.000 Classification Medium low Medium high Low
INSECTS (abrupt, balanced) 0.337$\pm$0.021 0.338$\pm$0.022 0.337$\pm$0.021
0.328$\pm$0.012 0.395$\pm$0.023 0.383$\pm$0.000 0.356$\pm$0.000
0.328$\pm$0.002 0.309$\pm$0.003 Classification Medium low Low Low KDDCUP99
0.227$\pm$0.184 0.381$\pm$0.235 0.792$\pm$0.009 0.117$\pm$0.104
0.013$\pm$0.004 0.015$\pm$0.000 0.015$\pm$0.000 0.008$\pm$0.000
0.007$\pm$0.000 Classification Low Medium high Low INSECTS (out of control)
0.312$\pm$0.001 0.312$\pm$0.001 0.440$\pm$0.221 0.359$\pm$0.001
0.296$\pm$0.001 0.537$\pm$0.000 0.563$\pm$0.000 0.399$\pm$0.002
0.423$\pm$0.001 Classification Medium low Medium high Low INSECTS
(incremental, imbalanced) 0.215$\pm$0.002 0.215$\pm$0.002 0.216$\pm$0.001
0.252$\pm$0.002 0.238$\pm$0.002 0.316$\pm$0.000 0.263$\pm$0.000
0.255$\pm$0.002 0.259$\pm$0.001 Classification Medium high Medium low Low
INSECTS (incremental, balanced) 0.460$\pm$0.019 0.460$\pm$0.019
0.459$\pm$0.019 0.484$\pm$0.009 0.628$\pm$0.109 0.463$\pm$0.001
0.376$\pm$0.001 0.438$\pm$0.006 0.405$\pm$0.000 Classification Medium high
High Low INSECTS (incremental, abrupt, balanced) 0.288$\pm$0.002
0.288$\pm$0.002 0.288$\pm$0.003 0.326$\pm$0.003 0.344$\pm$0.008
0.346$\pm$0.001 0.319$\pm$0.001 0.312$\pm$0.001 0.319$\pm$0.005 Classification
Medium high Medium high Low INSECTS (gradual, imbalanced) 0.220$\pm$0.001
0.220$\pm$0.001 0.220$\pm$0.001 0.262$\pm$0.003 0.221$\pm$0.000
0.367$\pm$0.002 0.327$\pm$0.000 0.256$\pm$0.001 0.261$\pm$0.001 Classification
Medium high Medium high Low INSECTS (incremental, abrupt, imbalanced)
0.220$\pm$0.001 0.220$\pm$0.000 0.219$\pm$0.001 0.266$\pm$0.003
0.223$\pm$0.001 0.385$\pm$0.000 0.349$\pm$0.000 0.279$\pm$0.000
0.285$\pm$0.001 Classification Medium high Medium high Low INSECTS
(incremental, reoccurring, imbalanced) 0.216$\pm$0.000 0.216$\pm$0.000
0.216$\pm$0.001 0.262$\pm$0.002 0.218$\pm$0.001 0.386$\pm$0.000
0.349$\pm$0.000 0.275$\pm$0.000 0.284$\pm$0.000 Classification Medium high
Medium high Low INSECTS (gradual, balanced) 0.374$\pm$0.015 0.374$\pm$0.015
0.374$\pm$0.015 0.349$\pm$0.015 0.421$\pm$0.011 0.342$\pm$0.002
0.320$\pm$0.000 0.294$\pm$0.007 0.282$\pm$0.004 Classification High High Low
Bitcoin 0.014$\pm$0.000 0.014$\pm$0.000 0.014$\pm$0.000 0.055$\pm$0.011
0.014$\pm$0.000 0.020$\pm$0.000 0.015$\pm$0.000 0.014$\pm$0.000
0.014$\pm$0.000 Classification Medium low Low Low Airlines 0.381$\pm$0.002
0.381$\pm$0.002 0.381$\pm$0.002 0.380$\pm$0.001 0.405$\pm$0.008
0.398$\pm$0.001 0.422$\pm$0.000 0.374$\pm$0.001 0.382$\pm$0.000 Regression Low
Medium low High Air quality (Shunyi) 0.166$\pm$0.002 0.166$\pm$0.002
0.166$\pm$0.002 0.182$\pm$0.008 0.213$\pm$0.023 0.263$\pm$0.013
0.498$\pm$0.002 0.199$\pm$0.010 0.519$\pm$0.003 Regression High Medium low Low
Tetouan power consumption 0.793$\pm$0.005 0.794$\pm$0.004 0.779$\pm$0.004
0.818$\pm$0.014 0.783$\pm$0.015 1.278$\pm$0.003 0.800$\pm$0.000
0.845$\pm$0.007 0.835$\pm$0.002 Regression Medium high Medium low Low Bike
sharing 0.039$\pm$0.002 0.039$\pm$0.002 0.040$\pm$0.002 0.044$\pm$0.004
0.130$\pm$0.008 0.016$\pm$0.001 0.379$\pm$0.000 0.039$\pm$0.002
0.467$\pm$0.000 Regression Medium low Medium low High Indian weather
(Bangalore) 0.213$\pm$0.012 0.214$\pm$0.012 0.214$\pm$0.012 0.222$\pm$0.013
0.292$\pm$0.055 0.258$\pm$0.002 0.445$\pm$0.000 0.163$\pm$0.001
0.442$\pm$0.000 Regression High Medium high High Italian air quality
0.280$\pm$0.006 0.280$\pm$0.006 0.280$\pm$0.006 0.273$\pm$0.003
0.496$\pm$0.043 0.355$\pm$0.004 0.538$\pm$0.000 0.412$\pm$0.004
0.718$\pm$0.000 Regression Medium high High High 5 cities PM2.5 (Chengdu)
1.071$\pm$0.007 1.068$\pm$0.010 1.072$\pm$0.010 1.016$\pm$0.006
1.186$\pm$0.041 1.727$\pm$0.006 1.139$\pm$0.000 1.370$\pm$0.012
1.215$\pm$0.000 Regression High High Low Energy prediction 0.965$\pm$0.009
0.966$\pm$0.008 0.966$\pm$0.010 0.969$\pm$0.007 0.940$\pm$0.001
5.623$\pm$0.070 1.241$\pm$0.007 2.824$\pm$0.133 1.028$\pm$0.025 Regression
High Medium high Low Household 0.492$\pm$0.002 0.494$\pm$0.001 0.492$\pm$0.002
0.536$\pm$0.007 0.439$\pm$0.004 0.529$\pm$0.000 0.464$\pm$0.000
0.465$\pm$0.001 0.443$\pm$0.000 Regression Low Low Low Allstate claim
0.492$\pm$0.002 0.492$\pm$0.002 0.489$\pm$0.001 0.495$\pm$0.001
0.474$\pm$0.002 1.05$\pm$0.003 0.782$\pm$0.000 0.593$\pm$0.001 0.773$\pm$0.000
Regression Medium low Low Low Air quality (Wanliu) 0.186$\pm$0.009
0.186$\pm$0.009 0.185$\pm$0.008 0.208$\pm$0.015 0.226$\pm$0.040
0.308$\pm$0.005 0.527$\pm$0.002 0.239$\pm$0.008 0.556$\pm$0.001 Regression
Medium low Medium low Low Air quality (Wanshouxingong) 0.170$\pm$0.004
0.170$\pm$0.004 0.170$\pm$0.004 0.184$\pm$0.007 0.267$\pm$0.061
0.289$\pm$0.006 0.523$\pm$0.000 0.214$\pm$0.011 0.596$\pm$0.005 Regression
Medium low Medium low Low Air quality (Gucheng) 0.184$\pm$0.009
0.184$\pm$0.009 0.184$\pm$0.009 0.202$\pm$0.016 0.284$\pm$0.029
0.372$\pm$0.013 0.570$\pm$0.000 0.386$\pm$0.017 0.696$\pm$0.001 Regression
Medium low Medium low Low Air quality (Huairou) 0.174$\pm$0.008
0.174$\pm$0.008 0.174$\pm$0.008 0.197$\pm$0.01 0.236$\pm$0.031 0.307$\pm$0.009
0.519$\pm$0.003 0.224$\pm$0.013 0.490$\pm$0.004 Regression Medium low Medium
low Low Air quality (Nongzhanguan) 0.154$\pm$0.012 0.154$\pm$0.011
0.154$\pm$0.012 0.163$\pm$0.011 0.257$\pm$0.024 0.272$\pm$0.014
0.504$\pm$0.002 0.200$\pm$0.028 0.572$\pm$0.027 Regression Medium low Medium
low Low Air quality (Changping) 0.171$\pm$0.009 0.172$\pm$0.009
0.171$\pm$0.008 0.18$\pm$0.014 0.191$\pm$0.027 0.337$\pm$0.015 0.549$\pm$0.000
0.203$\pm$0.013 0.489$\pm$0.002 Regression Medium low Medium low Low Air
quality (Dingling) 0.164$\pm$0.011 0.164$\pm$0.011 0.164$\pm$0.011
0.180$\pm$0.010 0.186$\pm$0.031 0.317$\pm$0.005 0.551$\pm$0.000
0.203$\pm$0.020 0.542$\pm$0.001 Regression Medium low Medium low Low Air
quality (Aotizhongxin) 0.181$\pm$0.008 0.180$\pm$0.008 0.181$\pm$0.008
0.197$\pm$0.016 0.222$\pm$0.017 0.349$\pm$0.008 0.515$\pm$0.002
0.280$\pm$0.057 0.561$\pm$0.002 Regression Medium low Medium high Low Air
quality (Dongsi) 0.173$\pm$0.008 0.173$\pm$0.008 0.173$\pm$0.008
0.186$\pm$0.014 0.245$\pm$0.018 0.279$\pm$0.003 0.517$\pm$0.002
0.267$\pm$0.011 0.544$\pm$0.001 Regression Medium low Medium low Low Air
quality (Guanyuan) 0.168$\pm$0.011 0.167$\pm$0.011 0.168$\pm$0.011
0.18$\pm$0.019 0.239$\pm$0.032 0.265$\pm$0.004 0.518$\pm$0.000 0.214$\pm$0.015
0.571$\pm$0.001 Regression Medium low Medium high Low Air quality (Tiantan)
0.164$\pm$0.01 0.164$\pm$0.01 0.164$\pm$0.01 0.182$\pm$0.013 0.191$\pm$0.023
0.266$\pm$0.013 0.515$\pm$0.000 0.180$\pm$0.007 0.559$\pm$0.007 Regression
Medium low Low High Indian weather (Lucknow) 0.206$\pm$0.009 0.206$\pm$0.009
0.206$\pm$0.009 0.215$\pm$0.012 0.268$\pm$0.026 0.215$\pm$0.002
0.450$\pm$0.000 0.156$\pm$0.004 0.441$\pm$0.000 Regression Low Low High Indian
weather (Mumbai) 0.427$\pm$0.013 0.427$\pm$0.013 0.428$\pm$0.013
0.456$\pm$0.022 0.506$\pm$0.036 0.629$\pm$0.004 0.588$\pm$0.000
0.393$\pm$0.003 0.579$\pm$0.000 Regression Low Medium low High Indian weather
(Rajasthan) 0.213$\pm$0.012 0.213$\pm$0.012 0.214$\pm$0.012 0.222$\pm$0.013
0.292$\pm$0.055 0.258$\pm$0.002 0.445$\pm$0.000 0.163$\pm$0.001
0.442$\pm$0.000 Regression Low Low High Indian weather (Bhubhneshwar)
0.427$\pm$0.013 0.427$\pm$0.013 0.428$\pm$0.013 0.456$\pm$0.022
0.506$\pm$0.036 0.629$\pm$0.004 0.588$\pm$0.000 0.393$\pm$0.003
0.579$\pm$0.000 Regression Low Low High Indian weather (Delhi) 0.129$\pm$0.009
0.129$\pm$0.009 0.129$\pm$0.009 0.143$\pm$0.011 0.180$\pm$0.023
0.102$\pm$0.001 0.401$\pm$0.000 0.070$\pm$0.001 0.398$\pm$0.000 Regression Low
Low High Indian weather (Chennai) 0.220$\pm$0.009 0.220$\pm$0.009
0.221$\pm$0.009 0.234$\pm$0.011 0.292$\pm$0.028 0.273$\pm$0.001
0.444$\pm$0.000 0.183$\pm$0.001 0.444$\pm$0.000 Regression Medium high Medium
low Low Taxi duration 1.000$\pm$0.001 1.001$\pm$0.002 1.001$\pm$0.002
0.999$\pm$0.002 0.997$\pm$0.000 1.955$\pm$0.008 1.096$\pm$0.000
1.156$\pm$0.010 1.018$\pm$0.000 Regression Low Medium low Low Traffic volume
N/A N/A N/A N/A N/A 1.838$\pm$0.004 0.973$\pm$0.000 1.142$\pm$0.056
0.976$\pm$0.000 Regression Medium low Medium low Low News popularity
0.465$\pm$0.004 0.465$\pm$0.004 0.465$\pm$0.004 N/A 0.473$\pm$0.011
0.705$\pm$0.000 0.572$\pm$0.000 0.523$\pm$0.000 0.482$\pm$0.000 Regression
Medium high High Low Beijing PM2.5 0.830$\pm$0.008 0.836$\pm$0.012
0.830$\pm$0.006 0.857$\pm$0.012 0.988$\pm$0.164 1.464$\pm$0.002
0.930$\pm$0.000 1.265$\pm$0.006 0.960$\pm$0.000 Regression Medium high Medium
high Low Election 0.506$\pm$0.173 N/A 0.513$\pm$0.154 0.566$\pm$0.312
0.329$\pm$0.143 0.023$\pm$0.001 0.353$\pm$0.000 0.002$\pm$0.000
0.313$\pm$0.004 Regression Medium high High High 5 cities PM2.5 (Shenyang)
0.173$\pm$0.016 0.173$\pm$0.015 0.174$\pm$0.016 0.205$\pm$0.020
0.476$\pm$0.068 0.363$\pm$0.006 0.545$\pm$0.000 0.277$\pm$0.021
0.619$\pm$0.001 Regression High Medium low High 5 cities PM2.5 (Guangzhou) N/A
0.419$\pm$0.107 0.585$\pm$0.299 0.617$\pm$0.574 0.471$\pm$0.190
0.027$\pm$0.002 0.432$\pm$0.000 0.012$\pm$0.001 0.407$\pm$0.000 Regression
Medium high Medium high High 5 cities PM2.5 (Beijing) N/A N/A N/A N/A N/A
1.480$\pm$0.005 0.896$\pm$0.000 1.107$\pm$0.025 0.926$\pm$0.001 Regression
Medium high Medium low High 5 cities PM2.5 (Shanghai) 0.869$\pm$0.003
0.870$\pm$0.004 0.868$\pm$0.004 0.835$\pm$0.008 0.926$\pm$0.021
1.629$\pm$0.005 0.936$\pm$0.001 1.116$\pm$0.040 0.974$\pm$0.000

###  Effect of Larger Models

[width=colback=white,boxrule=1pt,arc=0pt,outer
arc=0pt,left=0pt,right=0pt,top=0pt,bottom=0pt,boxsep=1pt,halign=left] Finding
(6): Decision trees or lightweight neural networks already achieve good model
accuracy on real-world relational data streams. Large models do not
necessarily lead to better effectiveness. A possible reason is that large
models are prone to overfitting and hard to generalize towards new
environment. When it comes to practical relational data streams, we recommend
to consider lightweight models. In this section, we experiment with larger
neural network models, in particular training MLPs with 3, 5, and 7 layers.
The number of neurons in each layer is (32, 16, 8), (32, 32, 16, 16, 8), and
(32, 32, 32, 16, 16, 16, 8), respectively. As shown in Figure 1, larger neural
networks perform worse in most instances, since larger neural networks tend to
overfit the data of the current window. Therefore, we recommend lightweight NN
models or tree models for real-world relational data streams.

[ROOM][width=0.2]figures/exp/heavy/room.pdf
[ELECTRICITY][width=0.2]figures/exp/heavy/electricity.pdf
[INSECTS][width=0.2]figures/exp/heavy/insects.pdf
[AIR][width=0.2]figures/exp/heavy/air.pdf
[POWER][width=0.2]figures/exp/heavy/power.pdf

Figure : Test error / test loss on NN with different number of hidden layers.

###  Effect of Regularization Factor

In this section, we explore the effect of regularization factor in EWC and
LwF. For EWC, the regularization factor is varied in
$\\{10^{2},10^{3},10^{4},10^{5}\\}$ and for LwF, we test among the range
$\\{0.001,0.01,0.1,1,10\\}$. Results are shown in Figure 1. As illustrated in
Figure 1, EWC generally exhibits improved effectiveness when the
regularization factor is set at 100 or 1000, while LwF performs the best with
a regularization factor of 0.01. It is notable that a too large regularization
factor can degrade the model’s effectiveness.

[EWC][width=0.25]figures/exp/reg/ewc.pdf
[LwF][width=0.24]figures/exp/reg/lwf.pdf

Figure : Test error / test loss on different regularization weights.

###  Effect of Exemplar Buffer Size

[width=colback=white,boxrule=1pt,arc=0pt,outer
arc=0pt,left=0pt,right=0pt,top=0pt,bottom=0pt,boxsep=1pt,halign=left] Finding
(7): Large buffer size or ensemble size increase the memory and computation
costs, but do not necessarily boost effectiveness. This verifies that
memorizing more old data is not always effective to open environment learning.
In this section, we delve deeper into understanding the role of exemplar
buffer size within the iCaRL framework The buffer sizes tested are
$\\{20,50,100,200,500\\}$. Figure 1 illustrates that the exemplar buffer size
does not drastically affect the test loss. However, a considerably large
buffer size, like 500, could lead to a higher loss, thus suggesting that more
exemplars does not always translate to better effectiveness. In light of this,
we suggest a relatively smaller buffer size for improved efficiency.

[width=0.6]figures/exp/buffer-size.pdf

Figure : Test error / test loss on different buffer size of iCaRL.

###  Effect of Ensemble Size

We evaluate the effectiveness of GBDT and SEA with ensemble size varying among
$\\{5,10,20,40\\}$. ARF is excluded due to its relative inefficiency without
significant improvement in effectiveness. As shown in Figure 1, a naive GBDT
generally performs better with a larger ensemble size, whereas SEA exhibits
different trends depending on the dataset. For instance, a larger ensemble
size leads to higher loss in the INSECTS dataset, yet results in lower loss in
the AIR dataset. Therefore, a larger ensemble size does not always improve
ensemble model accuracy.

[ROOM][width=0.2]figures/exp/ensemble/room.pdf
[ELECTRICITY][width=0.2]figures/exp/ensemble/electricity.pdf
[INSECTS][width=0.2]figures/exp/ensemble/insects.pdf
[AIR][width=0.2]figures/exp/ensemble/air.pdf
[POWER][width=0.2]figures/exp/ensemble/power.pdf

Figure : Test error / test loss on NN with ensemble size.

###  Running Time

We show the running time of these algorithms in Table 1. As we can see,
decision trees are more efficient than NN-based methods. Therefore, we
recommend to apply decision trees when high throughput is required.

Table : Running time in seconds of explored stream learning algorithms on
selected real-world datasets. All experiments run on 4 CPU threads. Lower
value indicates better performance. We report the average of three runs.

2.1!  Dataset Epochs Naive-NN EWC LwF iCaRL SEA-NN Naive-DT Naive-GBDT ARF
SEA-DT SEA-GBDT 4*ROOM 1 0.16 0.38 0.32 0.33 0.38 4*0.05 4*0.23 4*43.22 4*0.37
4*0.37 5 0.72 1.64 1.31 1.38 0.84 10 1.24 3.00 2.18 2.42 1.43 20 2.75 6.20
5.54 5.55 3.21 4*ELECTRICITY 1 0.64 1.31 1.14 1.53 1.04 4*0.17 4*0.63 4*141.55
4*0.89 4*1.61 5 2.72 6.00 5.38 5.35 3.58 10 5.41 13.18 9.44 9.91 6.04 20 10.86
25.05 22.81 22.02 12.81 4*INSECTS 1 1.64 4.71 2.98 2.81 4.73 4*1.44 4*25.42
4*1831.80 4*11.14 4*46.68 5 6.49 15.05 12.27 12.37 10.31 10 13.06 30.41 21.66
23.72 15.83 20 24.12 54.45 49.81 49.67 30.75 4*AIR 1 0.81 1.41 1.29 1.55 1.07
4*0.73 4*1.08 4*N/A 4*1.68 4*2.29 5 2.64 4.68 4.18 4.20 3.09 10 4.46 9.30 6.85
7.85 4.75 20 8.42 15.12 13.54 13.92 9.01 4*POWER 1 0.63 1.39 1.21 1.20 0.75
4*0.31 4*0.39 4*N/A 4*0.44 4*0.57 5 3.16 5.64 5.25 5.11 3.32 10 5.66 10.54
8.30 9.12 5.82 20 9.77 20.25 16.73 17.03 9.94 Rank 1 5 3 4 2 1 2 5 3 4

##  2 Dataset Description

Table 2 and Table 2 present statistical information of the datasets explored
in this work. We also list the extracted statistics of three open environment
challenges (anomaly, drift and missing value), which are the three dimensions
visualized in Figure LABEL:fig:clustershow and the criteria in Table 1. We
briefly describe these datasets by categories in the following sections.

Table : Basic information of real-world relational data streams
(classification task).

2.1!  Dataset Instances Features Classes Type Anomaly Drift Missing value
Bitcoin Heist Ransomware Address 2,916,697 6 27 Commerce 0.054 0.301 0 Room
Occupancy Estimation 10,129 16 4 Others 0.106 0.213 0 Electricity Prices
45,312 7 2 Commerce 0.052 0.221 0 Airlines 539,383 6 2 Commerce 0.007 0.143 0
Forest Covertype 581,012 54 7 S&T 0.043 0.249 0 INSECTS-Abrupt (balanced)
52,848 33 6 S&T 0.041 0.131 0 INSECTS-Abrupt (imbalanced) 355,275 33 6 S&T
0.051 0.125 0 INSECTS-Incremental (balanced) 57,018 33 6 S&T 0.033 0.183 0
INSECTS-Incremental (imbalanced) 452,044 33 6 S&T 0.046 0.143 0 INSECTS-
Incremental-abrupt-reoccurring (balanced) 79,986 33 6 S&T 0.055 0.196 0
INSECTS-Incremental-abrupt-reoccurring (imbalanced) 452,044 33 6 S&T 0.044
0.218 0 INSECTS-Incremental-gradual (balanced) 24,150 33 6 S&T 0.035 0.198 0
INSECTS-Incremental-gradual (imbalanced) 143,323 33 6 S&T 0.038 0.203 0
INSECTS-Incremental-reoccurring (balanced) 79,986 33 6 S&T 0.040 0.192 0
INSECTS-Incremental-reoccurring (imbalanced) 452,044 33 6 S&T 0.044 0.202 0
INSECTS-Out-of-control 905,145 33 24 S&T 0.039 0.067 0 KDDCUP99 494,021 41 23
S&T 0.005 0.111 0 NOAA Weather 18,159 8 2 Ecology 0.028 0.199 0 Safe Driver
595,212 57 2 Commerce 0.008 0.062 0 BLE RSSI dataset for Indoor localization
Data Set 9,984 5 3 Others 0.044 0.217 0

Table : Basic information of real-world relational data streams (regression
task).

2.1!  Dataset Instances Features Target Type Anomaly Drift Missing value
Italian City Air Quality 9,358 12 PT08.S1 (CO) Ecology 0.035 0.303 0.163
Energy Prediction 19,735 25 Appliances Power 0.061 0.326 0 Beijing Multi-Site
Air-Quality Aotizhongxin 35,064 11 PM2.5 Ecology 0.022 0.145 0.066 Beijing
Multi-Site Air-Quality Changping 35,064 11 PM2.5 Ecology 0.027 0.145 0.029
Beijing Multi-Site Air-Quality Dingling 35,064 11 PM2.5 Ecology 0.026 0.146
0.040 Beijing Multi-Site Air-Quality Dongsi 35,064 11 PM2.5 Ecology 0.043
0.139 0.038 Beijing Multi-Site Air-Quality Guanyuan 35,064 11 PM2.5 Ecology
0.031 0.142 0.033 Beijing Multi-Site Air-Quality Gucheng 35,064 11 PM2.5
Ecology 0.035 0.136 0.020 Beijing Multi-Site Air-Quality Huairou 35,064 11
PM2.5 Ecology 0.029 0.134 0.057 Beijing Multi-Site Air-Quality Nongzhanguan
35,064 11 PM2.5 Ecology 0.028 0.138 0.017 Beijing Multi-Site Air-Quality
Shunyi 35,064 11 PM2.5 Ecology 0.029 0.139 0.040 Beijing Multi-Site Air-
Quality Tiantan 35,064 11 PM2.5 Ecology 0.042 0.136 0.032 Beijing Multi-Site
Air-Quality Wanliu 35,064 11 PM2.5 Ecology 0.022 0.144 0.035 Beijing Multi-
Site Air-Quality Wanshouxigong 35,064 11 PM2.5 Ecology 0.032 0.143 0.024
Beijing PM2.5 43,824 7 PM2.5 Ecology 0.052 0.184 0.012 Indian Cities Weather
Bangalore 11,894 5 Average temperature Ecology 0.025 0.081 0.120 Indian Cities
Weather Bhubhneshwar 11,894 5 Average temperature Ecology 0.020 0.044 0.157
Indian Cities Weather Chennai 11,894 5 Average temperature Ecology 0.023 0.055
0.158 Indian Cities Weather Delhi 11,894 5 Average temperature Ecology 0.021
0.115 0.146 Indian Cities Weather Lucknow 11,894 5 Average temperature Ecology
0.022 0.111 0.198 Indian Cities Weather Mumbai 11,894 5 Average temperature
Ecology 0.020 0.044 0.157 Indian Cities Weather Rajasthan 11,894 5 Average
temperature Ecology 0.025 0.081 0.120 Household Electric Consumption 2,075,259
6 Global active power Power 0.035 0.399 0.005 Metro Interstate Traffic Volume
48,204 7 Traffic volume Commerce/ 0.023 0.077 0 PM2.5 Data of Five Chinese
Cities Beijing 52,584 8 PM2.5 Ecology 0.040 0.218 0.139 PM2.5 Data of Five
Chinese Cities Chengdu 52,584 8 PM2.5 Ecology 0.067 0.186 0.197 PM2.5 Data of
Five Chinese Cities Shanghai 52,584 8 PM2.5 Ecology 0.028 0.186 0.193 PM2.5
Data of Five Chinese Cities Shenyang 52,584 8 PM2.5 Ecology 0.056 0.192 0.255
PM2.5 Data of Five Chinese Cities Guangzhou 52,584 8 PM2.5 Ecology 0.031 0.297
0.128 Power Consumption of Tetouan City 52,417 7 Zone 3 power consumption
Power 0.022 0.352 0 Bike Sharing Demand 10,886 7 Count Commerce 0.025 0.287 0
Allstate Claims Severity 188,318 130 Loss Commerce 0.013 0.045 0 Portugal
Parliamentary Election 21,843 28 Final mandates Social 0.042 0.080 0 News
Popularity 93,239 11 Popularity index Social 0.027 0.169 0.001 Taxi Trip
Duration 1,458,644 11 Duration Commerce 0.023 0.217 0

###  Ecology

Italian City Air Quality de2008field: This dataset records the hourly averaged
concentrations for air pollutants as well as meteorological data in an Italian
city from 2004 to 2005. We regard PT08.S1 (CO) as the target of the regression
task. Beijing Multi-Site Air Quality zhang2017cautionary: This dataset
contains the hourly averaged concentrations for air pollutants and
meteorological data from 12 sites in Beijing from 2013 to 2017. We regard
PM2.5 as the target of the regression task. Beijing PM2.5 liang2015assessing:
This dataset contains the hourly averaged concentrations for air pollutants
and meteorological data from 2010 to 2014. We regard PM2.5 as the target of
the regression task. PM2.5 Data of Five Chinese Cities liang2016pm2: This
dataset contains the hourly averaged concentrations for air pollutants and
meteorological data in five Chinese cities from 2010 to 2015. We regard PM2.5
as the target of the regression target. Indian Cities Weather: This dataset
contains temperature and precipitation measurements from seven cities in
India. We regard the average temperature as the regression target. NOAA
Weather ditzler2013incremental: This dataset contains meteorological
measurements in Bellevue, Nebraska for 50 years. The task is to predict
whether it rains or not.

###  Power Consumption

Energy Prediction candanedo2017data: The task of this dataset is to predict
appliance energy use. We discard the last two unrelated random variables as
they are not real-world data. Power Consumption in Tetouan City
salam2018comparison: This dataset contains power consumption and
meteorological data in 3 zones in Tetouan city. We regard zone 3 power
consumption as the regression task. Household Electric Consumption Dua:2019:
This dataset contains electric usage measurements collected in an individual
household in Sceaux from 2006 to 2010. We set global active power as the
regression target.

###  Commerce

Bitcoin Heist Ransomware Address Dataset goldsmith2020analyzing: This dataset
contains Bitcoin transaction features and the respective ransomware labels
from 2009 to 2018. The goal is to predict the ransomware label. We discard the
Bitcoin address feature since it is hard to be converted to statistics.
Electricity Prices harries1999splice: This dataset contains electricity price
information in Australian New South Wales Electricity Market over around 5
months. The task is to predict whether the electricity prices go up or down.
Airlines ikonomovska2011learning: This dataset contains flight departure and
arrival information in the USA from 1987 to 2008. The task is to predict
whether a flight will be delayed or not. Bike Sharing Demand bike-sharing-
demand: This dataset contains the bike sharing data in a city. The task is to
predict how many bikes are rented in an hour. Feature “casual” and
“registered” are discarded, since their sum is the answer. Porto Seguro’s Safe
Driver porto-seguro-safe-driver-prediction: This dataset contains information
about automobile insurance claims. The task is to predict whether the driver
will file an insurance claim in the next year. Allstate Claims Severity
allstate-claims-severity: This dataset records insurance claims filed with
Allstate, an insurance company. The regression task is to predict the amount
of insurance claims. Metro Interstate Traffic Volume: This dataset records
hourly traffic volume in a metropolitan interstate along with weather and
holiday information. We regard the traffic volume as the regression target.
Taxi Trip Duration: This dataset contains the information of taxi trips. The
regression target is to predict the duration of a taxi trip.

###  Social

Portugal Parliamentary Election moniz2019real: This dataset tracks the
evolution of the 2019 Portuguese Parliamentary Election and the related
information. The target of this dataset is to predict the final mandates. News
Popularity moniz2018multi: This dataset contains feedback and responses of
several hot topics on 3 social media platforms for 8 months. We choose the
popularity of the news item on Facebook as the regression target.

###  Science & Technology

KDDCUP99 tavallaee2009detailed: This dataset contains records of network
connections and TCP dump data in a local area network. Each record is labeled
as either normal or one of the 22 different types of attack. The task is to
predict the type of attack. Forest Covertype blackard1999comparative: This
dataset contains forest information of 30*30 meter cells in US. The task is to
predict the predominant cover type. INSECTS souza2020challenges: This dataset
is collected from biology experiments of insects where drifts are simulated by
changing the temperature. The task is to predict the insect type. It contains
11 sub-datasets.

###  Others

BLE RSSI dataset for Indoor localization Data Set mohammadi2017semi: This
dataset contains Bluetooth Low Energy (BLE) Received Signal Strength
Indications (RSSI) collected from smartphones for 20 minutes. Participants are
instructed to walk inside their limited area with activated smartphones. The
task is to predict which area the participant is in. Room Occupancy Estimation
singh2018machine: This dataset contains environmental sensor statistics in a
room. The task is to predict the number of people in a room, which varies
between 0 and 3 people. Rialto Bridge Timelapse losing2016knn: This dataset
contains images captured for 10 buildings from a fixed camera in 20 days. 27
key features are extracted for the picture of each building. The changing
weather and light can lead to distribution drifts in features. The task is to
identify the correct building.