# VTG-LLM: Integrating Timestamp Knowledge into Video LLMs for Enhanced Video
Temporal Grounding

Yongxin Guo1,2, Jingyu Liu1, Mingda Li1, Xiaoying Tang2,3,4, Xi Chen1, Bo
Zhao1
1Tencent PCG
2School of Science and Engineering, The Chinese University of Hong Kong
(Shenzhen)
3The Shenzhen Institute of Artificial Intelligence and Robotics for Society
4The Guangdong Provincial Key Laboratory of Future Networks of Intelligence

###### Abstract

Video Temporal Grounding (VTG) focuses on accurately identifying event
timestamps within a particular video based on a linguistic query, playing a
vital role in downstream tasks such as video browsing and editing. While Video
Large Language Models (video LLMs) have made significant progress in
understanding video content, they often face challenges in accurately
pinpointing timestamps within videos, which limits their performance on VTG
tasks. Therefore, to improve video LLMs’ ability to effectively locate
timestamps, we argue that two critical aspects need to be enhanced. First, it
is essential to have high-quality instructional tuning datasets that encompass
mainstream VTG tasks. Second, directly incorporating timestamp knowledge into
video LLMs is crucial, as it enables models to efficiently comprehend
timestamp information. To address these needs, we first introduce VTG-IT-120K,
a high-quality and comprehensive instruction tuning dataset that covers VTG
tasks such as moment retrieval, dense video captioning, video summarization,
and video highlight detection. Furthermore, we propose a specially designed
video LLM model for VTG tasks, VTG-LLM, which (1) effectively integrates
timestamp knowledge into visual tokens; (2) incorporates absolute-time tokens
that specifically handle timestamp knowledge, thereby avoiding concept shifts;
and (3) introduces a lightweight, high-performance slot-based token
compression method to facilitate the sampling of more video frames.
Comprehensive experiments showcase the superior performance of VTG-LLM in
comparison to other video LLM methods across various VTG tasks. Our code and
datasets are available at https://github.com/gyxxyg/VTG-LLM.

##  1 Introduction

Large language models (LLMs) (Kaplan et al., 2020; Achiam et al., 2023;
Touvron et al., 2023) have exhibited considerable potential in capturing
knowledge and tackling real-world challenges using a zero-shot approach.
Recently, research has explored integrating knowledge from other modalities,
such as vision (Liu et al., 2024; Li et al., 2023a) and audio (Ghosal et al.,
2023), to enhance the capabilities of LLMs. Within the visual modality, video
large language models (video LLMs) have emerged as a significant research area
(Lin et al., 2023a; Zhang et al., 2023; Li et al., 2023b). These models aim to
generate textual outputs based on given videos and text prompts, achieving
remarkable success in multiple video-related tasks like video captioning (Xu
et al., 2024) and question-answering (Li et al., 2023c).

Video Temporal Grounding (VTG) is a crucial component of video understanding.
It requires models to accurately pinpoint event timestamps within a video
based on the given query. This task is essential for subsequent operations
such as video browsing and editing. The sub-tasks within VTG can be
categorized based on output formats, as described by (Lin et al., 2023b). For
instance, moment retrieval and dense video captioning (Fabian Caba Heilbron &
Niebles, 2015; Zhou et al., 2018) tasks demand models to generate one or
multiple timestamp intervals accompanied by captions; video highlight
detection (Lei et al., 2021) task requires models to produce a salient score
curve; and video summarization (Song et al., 2015) task calls for models to
output a series of timestamps along with their corresponding video frames.

Despite the remarkable achievements of video LLMs in video understanding
tasks, current models struggle to accurately pinpoint timestamps, leading to
subpar performance on VTG tasks. Therefore, it is essential to enhance video
LLMs by equipping them with the ability to effectively comprehend timestamps.
To accomplish this, we contend that two key aspects need to be addressed. On
one hand, high-quality instructional tuning datasets specifically tailored for
VTG tasks are necessary. Existing datasets are either of low quality (Zellers
et al., 2021) or exhibit significant task imbalance (Ren et al., 2023; Wang et
al., 2024b). On the other hand, it is crucial to develop new models that
efficiently and explicitly incorporate timestamp knowledge into video LLMs.

In this paper, we tackle the aforementioned issues by first introducing a
high-quality instruction tuning dataset, VTG-IT-120K, which consists of 47.2K
videos and 120K annotations. This dataset encompasses most mainstream VTG
tasks, including moment retrieval (63.2K), dense video captioning (37.2K),
video summarization (15.2K), and video highlight detection (3.9K). The VTG-
IT-120K dataset is more task-balanced compared to existing datasets (Ren et
al., 2023; Wang et al., 2024b), which either contain annotations exclusively
for the moment retrieval task or provide a mere 75 samples for the video
summarization task. Moreover, the VTG-IT-120K includes 51.9K self-collected
data annotations of superior quality. Furthermore, we present a novel video
LLM model, VTG-LLM, which comprises three components for efficiently
integrating timestamp knowledge into video LLMs: (1) To ensure that the inputs
of LLMs contain sufficient timestamp information, we introduce a sequence-time
embedding method that integrates timestamp data into visual tokens; (2)
Concept shifts can occur when different input data lead to the same output
targets, which can obscure the decision boundary (Moreno-Torres et al., 2012).
For instance, the number 20 could represent seconds or number of people. To
mitigate this, we incorporate absolute-time tokens that specifically handle
timestamp knowledge, without introducing quantization errors. This ensures
that the LLM tokenizer and LLM prediction head do not confuse timestamp-
related information with other digit-related information, such as counting and
mathematics; (3) Given the limited context length of LLMs, the number of
visual tokens must be reduced to allow for more video frame sampling. As such,
we employ a lightweight slot-based token compression method to decrease the
number of visual tokens to a fixed length, enabling video LLMs to sample more
video frames. Numerical results illustrate the superior performance of the
VTG-LLM compared to state-of-the-art video LLM methods. Additionally, ablation
studies reveal that absolute-time tokens and sequence-time embedding
significantly enhance the model’s performance in accurately locating
timestamps. Furthermore, the slot-based token compression method outperforms
the naive token compression baselines. Our key contributions are summarized as
follows:

Figure 1: Overview of the VTG-LLM model. We utilize a fixed visual encoder and
a Qformer model to encode sampled video frames into visual tokens.
Subsequently, the absolute time $t_{i}$ and sequence ID $f_{i}$ are embedded
and added to the top of each visual token for frame $i$. The slot-based token
compression is designed to compress the number of tokens to a fixed number of
visual tokens. To prevent overfitting and facilitate training, we employ a
variety of prompts and well-structured answers. The timestamps with green
color are encoded using six distinct time tokens.

  * •

We introduce VTG-IT-120K, a high-quality instruction tuning dataset comprising
annotations for four distinct VTG tasks: moment retrieval (63.2K), dense video
captioning (37.2K), video summarization (15.2K), and video highlight detection
(3.9K).

  * •

We present a carefully crafted model, VTG-LLM, which incorporates timestamp
information into video LLMs, providing models with the ability to understand
timestamps. Quantitative results demonstrate the superior performance of VTG-
LLM across various datasets, such as Charades-STA, QVHighlights, and YouCook2.
Moreover, all three components are necessary for VTG-LLM to achieve the best
results.

##  2 Related Works

Video Temporal Grounding (VTG) aims to accurately locate the timestamps of
events within a given video (Lin et al., 2023b). This encompasses tasks such
as moment retrieval (Gao et al., 2017; Zala et al., 2023b; Oncescu et al.,
2021; Hendricks et al., 2018a), dense video captioning (Zellers et al., 2021;
Zala et al., 2023b; Tang et al., 2019; Fabian Caba Heilbron & Niebles, 2015),
video summarization (Song et al., 2015; Gygli et al., 2014), and video
highlight detection (Lei et al., 2021; Xiao et al., 2023). Traditional methods
mainly address VTG tasks through large-scale video-text pre-training, using
training objectives such as video-text contrastive learning (Xu et al., 2021;
Wang et al., 2022; Yan et al., 2022), video-text matching (Li et al., 2023d;
Chen et al., 2024), and masked auto-encoding (Tong et al., 2022; Zhao et al.,
2024). Although these methods have shown satisfactory results, they require
resource-consuming pre-training, lack zero-shot capabilities, and often need
further fine-tuning on many downstream tasks.

Recently, Video Large Language Models (video LLMs) have shown promise in video
understanding tasks. Traditional video LLMs mainly generate captions that
summarize videos (Zhang et al., 2023; Lin et al., 2023a; Li et al., 2023b;
Maaz et al., 2023; Zhu et al., 2023). However, these models lack the capacity
to accurately locate the timestamps of events within a given video. Some
studies have attempted to address this issue. For example, TimeChat (Ren et
al., 2023) constructs time-sensitive instruction tuning datasets and encodes
timestamp knowledge into visual tokens. VTimeLLM (Huang et al., 2023) proposes
a LLaVA-like three-stage training method. LITA (Huang et al., 2024) introduces
fast-slow visual tokens and adds time tokens to LLM tokenizers. Momentor (Qian
et al., 2024) proposes a time encoder to solve the quantization error of time
tokens. HawkEye (Wang et al., 2024b) constructs a high-quality instruction
tuning dataset based on InternVid (Wang et al., 2022) for the moment retrieval
task. However, existing instruction tuning datasets for VTG tasks are either
low-quality or exhibit extreme task imbalance, hindering model performance on
overall tasks. Therefore, we propose a high-quality dataset VTG-IT to address
this issue. Furthermore, we introduce VTG-LLM, which contains three well-
designed components to boost video LLM performance on VTG tasks.

##  3 Method

In this section, we introduce the VTG-LLM model and the VTG-IT-120K dataset.
The overview of the VTG-LLM model can be found in Figure 1.

###  3.1 VTG-LLM: Incorporating Knowledge about Timestamps into Video LLMs

In this subsection, we introduce the detailed structure of VTG-LLM, which
includes three key components designed to help video LLMs better understand
timestamps. Specifically, we propose: (1) a sequence-time embedding mechanism
to straight-forwardly incorporate timestamp information into visual tokens;
(2) the introduction of absolute-time token without quantization error to
distinguish time-related knowledge from other digit-related knowledge; and (3)
the use of slot-based token compression to enable video LLMs to effectively
handle more frames. We will introduce these components in detail in the
following parts of this section.

####  3.1.1 Sequence-Time Embedding

##### Sequence embedding.

Existing studies (Zhang et al., 2023; Ren et al., 2023) employ sequence
embedding to incorporate relative time information into visual tokens.
Specially, given $N\times M$ tokens $\\{\mathbf{z}_{i,j}|1\leq i\leq N,1\leq
j\leq M\\}$ sampled from $N$ frames, the process of adding sequence embedding
onto visual tokens can then be represented by:

$\displaystyle\hat{\mathbf{z}}_{i,j}=\mathbf{z}_{i,j}+[\mathbf{W}_{s}]_{i}\,,$
(1)

where $\mathbf{W}_{s}\in\mathbb{R}^{N\times d}$ is the weight of sequence-
embedding matrix.

##### Sequence-Time Embedding.

Although promising, the sequence embedding only contains information about the
temporal order and may not accurately represent knowledge about the absolute
timestamps. For instance, sampled frames may not be uniformly distributed
throughout the entire video (Ren et al., 2023; Huang et al., 2023), and the
start time of the first sampled frame is also unknown. This makes it difficult
to infer the timestamps of the frames. To address this issue, in addition to
the sequence embedding, we also add the absolute time embedding as shown in
the following equation

$\displaystyle\hat{\mathbf{z}}_{i,j}=\mathbf{z}_{i,j}+[\mathbf{W}_{s}]_{i}+[\mathbf{W}_{t}]_{t}\,.$
(2)

Here, $t$ is the absolute timestamp (in seconds) of the corresponding frames.
$\mathbf{W}_{t}\in\mathbb{R}^{T\times d}$ is the absolute time embedding. $T$
is the maximum timestamp (in seconds), which is set to 8192 by default,
supporting videos for more than 1 hours long theoretically. It’s important to
note that to avoid disrupting the visual tokens generated by pre-trained
visual encoders and to speed up the convergence, $\mathbf{W}_{t}$ is
initialized by setting all entries to zero. Without the zero-initialization
method, the performance of VTG-LLM significantly degraded, as shown in the "TE
Random Initialize" part of Table 2.

Due to the imbalanced nature of video lengths and the gap between training and
test data, there may be timestamps for which the absolute time embedding has
not been trained. To address this issue, we further introduce a test-time
interpolation mechanism. In detail, for the timestamp $t$ which is not trained
during training, we first find the timestamps $t_{l}$ and $t_{r}$ that
satisfy:

$\displaystyle
t_{l}=\operatorname*{arg\,max}_{t_{l}}t_{1}<t,t_{l}\in\mathcal{T}_{tr}\,,t_{r}=\operatorname*{arg\,min}_{t_{r}}t_{r}>t,t_{r}\in\mathcal{T}_{tr}\,,$
(3)

where $\mathcal{T}_{tr}$ is the set of timestamps with trained absolute-time
embedding. Then the absolute-time embedding of timestamp $t$ is given by

$\displaystyle[\mathbf{W}_{t}]_{t}=\frac{t-t_{l}}{(t_{r}-t_{l})}[\mathbf{W}_{t}]_{t_{l}}+\frac{t_{r}-t}{(t_{r}-t_{l})}[\mathbf{W}_{t}]_{t_{r}}\,.$
(4)

######  Remark 3.1 (Discussion on existing techniques that integrating
information of timestamps into visual tokens.).

In addition to incorporating time embeddings into visual tokens, there are
other techniques for integrating timestamp information into visual tokens. For
instance, adding text inputs into Qformer (Ren et al., 2023) and inserting
time tokens before visual tokens (Hua et al., 2024). However, we believe that
these techniques are orthogonal to the time embedding approach, and it is also
possible to combine these approaches in future work.

####  3.1.2 Absolute-Time Tokens

The integration of special time tokens into the tokenizer has proven
advantageous for video temporal grounding tasks, as demonstrated in several
studies (Yang et al., 2023; Qian et al., 2024; Huang et al., 2024). However,
the use of relative time tokens (frame ID) has revealed certain limitations.
Firstly, there is the issue of quantization errors, which prevent the models
from predicting timestamps outside the sampled frames. Secondly, the token
embedding requires training from scratch, and any changes to the number of
time tokens necessitate further training. As a result, our goal is to develop
a new time token mechanism that can both represent fine-grained time without
quantization errors and represent a broad range of timestamps without
requiring additional effort when sampling varying numbers of frames.

##### Absolute-time tokens for resolving the quantization errors.

To resolve the issue of quantization errors, we introduce the concept of
absolute-time tokens. As depicted in Figure 1, we have incorporated eleven
time tokens into the tokenizer. These consist of ten digit time tokens
representing the digits from 0 to 9, and an additional token for a decimal
point. All timestamps are represented using six time tokens. For instance, the
time 120.5 seconds would be formatted as
<TIME_ZERO><TIME_ONE><TIME_TWO><TIME_ZERO><TIME_DOT><TIME_FIVE>. This
formulation allows the time tokens to handle videos more than 1 hours in
length, with the precision remaining constant regardless of video length
increases. Importantly, it is essential to format all timestamps using the
same number of time tokens. As shown in the "Time Token not Formatted" section
of Table 2, we found that maintaining a consistent format for all timestamps
significantly improves model performance.

##### Initialization of token embedding for absolute-time tokens.

Although using the absolute time token resolves the quantization error, we
discovered that randomly initialized time tokens negatively impact the
original token embedding space, preventing the LLMs from learning accurate
knowledge about the time tokens. This resulting in poor performance of models,
as shown in the "Token Embedding not Initialized" part of Table 2. To tackle
this issue, our goal is to initialize the time-related knowledge using the
digit-related knowledge, as the digit-related knowledge has already been well-
trained during the LLM pretraining stage.

This is achieved by adjusting the initialization of the weights of token
embedding and LLM prediction head. Specifically, we consider the token
embedding matrix $\mathbf{W}_{token}$. When initializing the embedding for the
time token "<TIME_ONE>", we align the embedding of this time token with the
embedding of the token "1":

$\displaystyle[\mathbf{W}_{token}]_{\text{ID}(\text{'<TIME\\_ONE>'}),j}=[\mathbf{W}_{token}]_{\text{ID}(\text{'1'}),j}\,,\forall
0\leq j<d\,,$ (5)

where ID() denotes the token ID of the given token string. Similarly, for the
LM prediction head, we employ the same method to transfer the knowledge from
the digit tokens to the time tokens.

####  3.1.3 Slot-Based Token Compression

Compressing the number of visual tokens is a critical issue in video LLMs. On
one hand, using all the visual tokens generated by visual backbones preserves
all the information. However, due to the restrictions on LLM context length,
the number of sampled frames must also be reduced. On the other hand, sampling
a subset of frames may result in overlooking important information. To resolve
this issue, we propose a simple yet effective method, namely slot-based token
compression, to compress the visual tokens to a fixed number of tokens.

(a) Overview of VTG-IT

(b) Slot-Based Token Compression

Figure 2: Overview of VTG-IT and slot based sampler. As shown in Figure 2(a),
the VTG-IT contains annotations of 4 important video temporal grounding tasks.
The VTG-IT-X indicate self-collected annotations. Figure 2(b) shows the
architecture of slot-based token compression. The $N$ tokens are passed
through a trainable slot dispatcher, which assigns dispatch weights
$\bm{\omega}\in\mathbb{R}^{N}$ to each slot. The tokens are then mixed based
on these dispatch weights to obtain the slots.

As illustrated in Figure 2(b), given $N$ visual tokens
$\mathbf{z}_{1},\cdots,\mathbf{z}_{N}$, and the trainable slot dispatcher
$\mathbf{\Phi}\in\mathbb{R}^{K\times d}$, where $K$ represents the number of
slots, the visual tokens first pass through the slot dispatcher to obtain the
dispatch weights, and the tokens are mixed based on the dispatch weights to
generate the slots. For instance, the slot $k$ is generated by

$\displaystyle\mathbf{s}_{k}=\sum_{i=1}^{N}\frac{\exp(\mathbf{\Phi}_{k}^{T}\mathbf{z}_{i})}{\sum_{j=1}^{N}\exp(\mathbf{\Phi}_{k}^{T}\mathbf{z}_{j})}\mathbf{z}_{i}\,.$
(6)

The slots are subsequently fed into the visual projection layers and serve as
visual tokens.

######  Remark 3.2 (Discussion on slot mechanism).

The slot-based token compression is inspired by the slot mechanism in SoftMoE
(Puigcerver et al., 2023). However, in SoftMoE, the slots are not used to
reduce sequence length, which is distinct from our primary objective.
Moreover, some existing multi-modal LLMs compress the token number through
cross-attention (Zhang et al., 2023; Bai et al., 2023; Ren et al., 2023). Our
approach involves training only one matrix $\mathbf{\Phi}$, making it more
computationally efficient and less data-consuming, and achieve better
performance (Table 2).

###  3.2 VTG-IT-120K: Formatted Time-Sensitive Instruction Tuning Dataset

In this section, we introduce the VTG-IT-120K, which consists of 51.9K self-
collected annotations and 67.6K annotations obtained from TimeIT (Ren et al.,
2023). As shown in Figure 2(a), VTG-IT-120K primarily covers four distinct
video temporal grounding tasks, including:

  * •

Moment Retrieval (63.2K): For the moment retrieval task, we use
HiRESTgrounding (Zala et al., 2023a), QuerYD (Oncescu et al., 2021), DiDeMo
(Hendricks et al., 2018b), and self-collected VTG-IT-MR.

  * •

Dense Video Captioning (37.2K): For the dense video captioning task, we use
HiRESTstep (Zala et al., 2023a), COIN (Tang et al., 2019), ActivityNet
Captions (Fabian Caba Heilbron & Niebles, 2015), and self-collected VTG-IT-
DVC.

  * •

Video Summarization (15.2K): For the video summarization task, we use TVSum
(Song et al., 2015), SumMe (Gygli et al., 2014), and self-collected VTG-IT-VS.

  * •

Video Highlight Detection (3.9K): For the video highlight detection task, we
use self-collected VTG-IT-VHD.

The VTG-IT-X annotations for the four tasks are self-collected using 16K
videos from the YT-Temporal-180M dataset (Zellers et al., 2021).

##### Data formatting.

All the tasks are formatted to QA pairs, and we use diverse questions to help
models understand human intention. The answers are formatted for better
transferring knowledge between different tasks. In detail, we reformat the
annotations of different tasks, and use more similar answer formats among
tasks. More detailed examples can be found in Appendix B.

  * •

Moment retrieval and dense video captioning. The GPT responses for these two
tasks are formatted as ’start time - end time, description’. The moment
retrieval task will consist of a single sentence, while the dense video
captioning task may include multiple sentences.

  * •

Video summarization and video highlight detection. The responses for these two
tasks are formatted as ’timestamp, score, description’. For the video
highlight detection task, the description is the same as the query; for the
video summarization task, the descriptions use placeholders like ’the
highlight of the video’.

##  4 Experiments

###  4.1 Experiment Settings

##### Models and training configuration.

Unless otherwise specified, we employ ViT-G/14 from EVA-CLIP (Sun et al.,
2023) and the Qformer from InstructBLIP (Dai et al., 2024) as the visual
backbone. The vision projection layer and the weights of sequence embedding
are initialized using the Video-LLaMA checkpoint (Zhang et al., 2023). For the
language model backbone, we utilize LLaMA-2-7B (Touvron et al., 2023). The
number of slots is set to 256. The models are trained on the VTG-IT-120k and a
randomly sampled subset (97k) from the Valley dataset (Luo et al., 2023b). We
randomly sample 96 frames for each video. Training is carried out using 16 ATN
910B NPUs with a batch size of 64, taking approximately 40 hours to complete.
The fine-tuning also uses 16 NPUs and set the batch size to 16. We set the
learning rate to 3e-5 initially, and train the models for 10 epochs by
default.

##### Evaluation datasets, metrics, and baseline models.

We evaluate the model performance on three different tasks:

  * •

dense video captioning. We employ Youcook2 (Zhou et al., 2018) as the test
dataset, and following the evaluation settings in TimeChat (Ren et al., 2023).
Metrics including CIDEr (Vedantam et al., 2015), which assessing the quality
of the captions; (2) SODA_c (Fujita et al., 2020) for story-level evaluation;
(3) F1 score to measure the model’s ability to accurately locate timestamps;

  * •

Moment retrieval. We utilize test set of Charades-STA (Gao et al., 2017) for
the moment retrieval task and report the recall at IOU thresholds of 0.5 and
0.7.

  * •

Video highlight detection. We employ the validation set of the QVHighlights
dataset (Lei et al., 2021) and report the mean average precision (mAP) with
IOU thresholds of 0.5 and 0.75, as well as the HIT@1, which represents the hit
ratio of the highest scored clip.

For baseline models, we select Valley (Luo et al., 2023b), VideoChat (Li et
al., 2023b), Video-ChatGPT (Maaz et al., 2023), and Video-LLaMA (Zhang et al.,
2023) as examples of traditional video LLMs. For video LLMs specifically
designed for VTG tasks, we choose TimeChat (Ren et al., 2023), VTimeLLM (Huang
et al., 2023), Momentor (Qian et al., 2024), and HawkEye (Wang et al., 2024b)
111Although related, LITA (Huang et al., 2024) employs self-built evaluation
sets, making fair comparison difficult..

###  4.2 Results

##### Zero-shot performance of the VTG-LLM on various VTG tasks.

In Table 1, we evaluate the performance of the VTG-LLM on three VTG tasks,
including dense video captioning (Youcook2), moment retrieval (Charades-STA),
and video highlight detection (QVHighlights). The results indicate that: (1)
The VTG-LLM model outperforms all existing 7B video LLM models on the zero-
shot setting across all three datasets; (2) The performance of the VTG-LLM on
the Charades-STA dataset for IOU=0.7 surpasses that of the VTimeLLM-13B model,
indicating the superior performance of the VTG-LLM in accurately locating
timestamps. (3) The improvement of the VTG-LLM on metrics that evaluate
timestamp location accuracy, such as the F1 score for the Youcook2 dataset and
Recall for the Charades-STA dataset, is more pronounced. This suggests that
the performance gain of the VTG-LLM primarily stems from more accurate
timestamp location. Additionally, performance on other metrics like CIDEr and
SODA_c also improves.

Table 1: Zero-shot performance of algorithms over various tasks. We evaluated
the performance of VTG-LLM using the Youcook2, Charades-STA, and QVHighlights
datasets. We highlight the best results for each block using bold font. The
Valley, VideoChat-Embed, Video-ChatGPT, and Video-LLaMA results are elaborated
from previous studies (Ren et al., 2023; Huang et al., 2023; Qian et al.,
2024). The results with  gray color indicates unfair comparison (13B). The
VTG-LLM samples 96 frames during training and testing stages.

Model | Youcook2 | Charades-STA | QVHighlights
---|---|---|---
SODA_c | CIDEr | F1 Score | $\text{R@1}_{\text{(IOU=0.5)}}$ | $\text{R@1}_{\text{(IOU=0.7)}}$ | mAP | HIT@1
Traditional Video LLMs |  |  |  |  |  |  |
Valley (7B) | 0.1 | 0.0 | 1.5 | 4.7 | 1.6 | 10.9 | 15.2
VideoChat (7B) | 0.2 | 0.6 | 3.4 | 3.2 | 1.4 | 13.1 | 18.1
Video-LLaMA (7B) | 0.0 | 0.0 | 0.1 | 2.7 | 1.2 | 11.3 | 15.6
Video-ChatGPT (7B) |  |  |  | 7.7 | 1.7 | 3.8 |
Temporal Grounding Video LLMs |  |  |  |  |  |  |
TimeChat (7B) | 1.2 | 3.4 | 12.6 | 32.2 | 13.4 | 14.5 | 23.9
VTimeLLM (7B) |  |  |  | 27.5 | 11.4 |  |
VTimeLLM (13B) |  |  |  | 34.3 | 14.7 |  |
Momentor (7B) |  |  |  | 26.6 | 11.6 | 7.6 |
HawkEye (7B) |  |  |  | 31.4 | 14.5 |  |
VTG-LLM (7B) | 1.5 | 5.0 | 17.5 | 33.8 | 15.7 | 16.5 | 33.5

##### Quality Analysis.

In Figure 3, we compare the outputs of VTG-LLM and TimeChat in a zero-shot
manner on the Youcook2 dataset. We observe that VTG-LLM’s outputs are more
closely aligned with human intention. For instance, there are no repeated
sentences with the same meaning.

Figure 3: Zero-shot case study. We compare the model outputs of VTG-LLM and
TimeChat (Ren et al., 2023) on Youcook2 dataset in the zero-shot manner. The
colored texts indicating repeated captions. More case studies can be found in
Appendix C.

###  4.3 Ablation Studies

In this section, we aim to show that all the designed components are necessary
for VTG-LLM to achieve the best results.

Table 2: Ablation studies on VTG-LLM. We perform ablation studies on the three
components of the algorithm. For the Sequence-Time Embedding section, "SE
Only" refers to using only sequence embedding. "TE Random Initialize"
represents the random initialization of time embedding rather than the zero
initialization we recommend. In the Time Token component, "No Time Token"
signifies not using the time token, while "Time Token not Formatted" implies
not formatting timestamps to the same length. "Token Embedding not
Initialized" indicates randomly initialize the token embedding and LLM
prediction head for time tokens. Lastly, for the Token Compression aspect, we
compare slot-based token compression to several straightforward baselines. We
highlight the best results of ablation studies using underline.

Model | Youcook2 | Charades-STA | QVHighlights
---|---|---|---
SODA_c | CIDEr | F1 Score | $\text{R@1}_{\text{(IOU=0.5)}}$ | $\text{R@1}_{\text{(IOU=0.7)}}$ | mAP | HIT@1
Ablation Studies on STE |  |  |  |  |  |  |
SE Only | 1.4 | 4.4 | 17.2 | 33.6 | 14.8 | 16.4 | 32.8
SE + TE (Random Initialize) | 1.3 | 4.2 | 16.8 | 21.4 | 10.5 | 14.2 | 22.5
Ablation Studies on Time Token |  |  |  |  |  |  |
No Time Token | 1.7 | 5.5 | 17.4 | 29.5 | 13.6 | 16.2 | 33.9
Time Token not Formatted | 1.5 | 5.1 | 16.8 | 27.0 | 12.4 | 15.0 | 28.8
Token Embedding not Initialized | 1.4 | 4.8 | 16.1 | 15.6 | 6.5 | 14.0 | 21.9
Ablation Studies on Token Compression |  |  |  |  |  |  |
Entropy Sampling | 1.3 | 4.2 | 16.8 | 21.1 | 9.6 | 14.0 | 20.1
Diverse Sampling | 1.3 | 4.4 | 16.8 | 22.4 | 10.5 | 14.2 | 22.7
Cross Attention | 1.3 | 4.2 | 17.6 | 19.9 | 8.8 | 13.7 | 21.0
Original VTG-LLM | 1.5 | 5.0 | 17.5 | 33.8 | 15.7 | 16.5 | 33.5

##### Ablation studies on types of token compression methods.

In the "Ablation Studies on Token Compression" section of Table 2, we present
the performance of VTG-LLM using different token compression methods and
include some naive baselines. The "Entropy Sampling" method involves sampling
the number of tokens to 256 by selecting tokens with the maximum entropy,
where the entropy is estimated using k-nearest neighbor distances (Kozachenko
& Leonenko, 1987). The "Diverse Sampling" method involves sampling tokens to
256 using k-means++ (Arthur et al., 2007) to select tokens with the maximum
distance. The "Cross Attention" method involves using a cross-attention layer
to compress the number of tokens. The results indicate that: (1) The slot-
based sampling achieves better performance among all compression methods; (2)
Sampling-based methods perform poor compared to slot-based token compression
method. We conjecture that this is because the sampling methods drop too much
necessary information; (3) The cross-attention method performs even worse than
sampling-based methods, which might be attributed to the attention mechanism
requiring a large amount of data to perform well.

##### Ablation studies on sequence-time embedding.

In the "Ablation Studies on STE" section of Table 2, we show the performance
of VTG-LLM when only using sequence embedding ("SE Only") and using sequence-
time embedding but randomly initializing the weights of time-embedding ("TE
Random Initialize"). The results indicate that: (1) the time-embedding is
necessary for VTG-LLM to achieve the best result. Without the time-embedding,
the performance of VTG-LLM on all datasets decreases; (2) Randomly
initializing the time embedding significantly hurts the performance of VTG-
LLM, making its performance worse than "SE Only".

##### Ablation studies on time-token.

In the "Ablation Studies on Time Token" section of Table 2, we show the
performance of VTG-LLM when not using special time tokens ("No Time Token"),
when not formatting the length of timestamps to six time tokens ("Time Token
not Formatted"), and when do not initialize the token embedding and LLM head
for time tokens ("Token Embedding not Initialized"). The results indicate that

  * •

Using time tokens significantly boosts the model’s capacity to accurately
locate the timestamps, improving the score on metrics like F1 score,
$\text{R@1}_{\text{(IOU=0.5)}}$, $\text{R@1}_{\text{(IOU=0.7)}}$, and mAP.
However, the caption-quality related metrics like SODA_c and CIDEr slightly
decrease. This suggests that there exists a trade-off between caption quality
and timestamp accuracy when using time token.

  * •

We found that the "No Time Token" setting tends to predict fewer timestamps on
QVHighlights dataset, which may lead to better HIT@1 performance but a worse
mAP score.

  * •

Formatting the timestamps using the same number of time tokens significantly
enhances the model performance ("Time Token not Formatted"). We believe that
this is because the formatted setting simplifies the task, making the models
needless to identify the number of tokens in timestamps.

  * •

Initialize the weights of token embedding and LLM prediction head is essential
for time token to work well.

In summary, we recommend using special time tokens to achieve significant
improvements in the accuracy of locating timestamps, which is the primary goal
of this paper. Moreover, investigating the trade-off between caption quality
and timestamp location accuracy in future work would be interesting.

Table 3: Fine-tuned performance of algorithms over various tasks. We fine-tune
the Algorithm on Youcook2, Charades-STA, and QVHighlights datasets with a
batch size of 16. We use  gray color for traditional non-generalist models and
task-specific models. Vid2Seq (Yang et al., 2023) fully training the LLM
models and using 1B video data for pretraining.

Model | Youcook2 | Charades-STA | QVHighlights
---|---|---|---
SODA_c | CIDEr | F1 Score | $\text{R@1}_{\text{(IOU=0.5)}}$ | $\text{R@1}_{\text{(IOU=0.7)}}$ | mAP | HIT@1
Non-Generative Models |  |  |  |  |  |  |
InternVideo2-6B (Wang et al., 2024a) |  |  |  | 70.0 | 49.0 |  |
VDI (Luo et al., 2023a) |  |  |  | 52.3 | 31.4 |  |
Moment-DETR (Lei et al., 2021) |  |  |  | 55.7 | 34.2 | 37.4 | 60.2
HL-CLIP (Han et al., 2024) |  |  |  |  |  | 41.9 | 70.6
Task-Specific Models |  |  |  |  |  |  |
Vid2Seq (Yang et al., 2023) | 7.9 | 47.1 | 27.3 |  |  |  |
Vid2Seq (Visual Only) (Yang et al., 2023) | 5.7 | 25.3 | 23.5 |  |  |  |
HawkEye |  |  |  | 58.3 | 28.8 |  |
Generalist Models |  |  |  |  |  |  |
TimeChat | 3.4 | 11.0 | 19.5 | 46.7 | 23.7 | 21.7 | 37.9
VTG-LLM | 3.6 | 13.4 | 20.6 | 57.2 | 33.4 | 24.1 | 41.3

##### Fine-tuned model performance of VTG-LLM.

In Table 3, we fine-tune the VTG-LLM models on Youcook2, Charades-STA, and
QVHighlights datasets. From the results, we make the following observations:
(1) The performance of VTG-LLM is significantly better than other generalist
models across all three tasks; (2) The performance of VTG-LLM on the moment
retrieval task (Charades-STA) is comparable to task-specific models like
HawkEye, as well as non-generative models like VDI and Moment-DETR. (3) The
task-specific model Vid2Seq (Yang et al., 2023) achieves superior performance
on the Youcook2 dataset by fully training the LLM models and using 1B video
data for pretraining. In contrast, we only fine-tune the LLM using LoRA and
employ 120K instruction tuning data.

##  5 Conclusion, Limitation, and Future Works

In this paper, we introduce VTG-IT-120K, a high-quality instruction tuning
dataset that covers four video temporal grounding (VTG) tasks, including
moment retrieval, dense video captioning, video highlight detection, and video
summarization. Furthermore, we propose the VTG-LLM, an improved video LLM
model for integrating knowledge about timestamps, enhancing the model
performance on VTG tasks. Extensive numerical results demonstrate the superior
zero-shot performance of VTG-LLM over state-of-the-art video LLM models on
Charades-STA, QVHighlights, and Youcook2 datasets, with each proposed
component contributing to individual performance gains.

However, there are still some constraints with the VTG-LLM that warrant
further research. For instance, the VTG-LLM’s capacity for dense video
captioning and highlight detection remains subpar compared to non-generative
or task-specific models. Therefore, exploring more advanced methods to rectify
this issue is a worthwhile endeavor. Additionally, while this study primarily
focuses on the accuracy of timestamp location, it would also be beneficial to
investigate techniques to enhance other aspects, such as the precision of
salient scores and the quality of captions. Moreover, we did not incorporate
the audio elements of the videos in this study. Integrating these in the
future could potentially enhance the overall performance.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Arthur et al. (2007) David Arthur, Sergei Vassilvitskii, et al.  k-means++: The advantages of careful seeding.  In _Soda_ , volume 7, pp. 1027–1035, 2007.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.  Qwen-vl: A frontier large vision-language model with versatile abilities.  _arXiv preprint arXiv:2308.12966_ , 2023.
  * Chen et al. (2024) Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu.  Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Dai et al. (2024) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Fabian Caba Heilbron & Niebles (2015) Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 961–970, 2015.
  * Fujita et al. (2020) Soichiro Fujita, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata.  Soda: Story oriented dense video captioning evaluation framework.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16_ , pp. 517–531. Springer, 2020.
  * Gao et al. (2017) Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia.  Tall: Temporal activity localization via language query.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 5267–5275, 2017.
  * Ghosal et al. (2023) Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria.  Text-to-audio generation using instruction-tuned llm and latent diffusion model.  _arXiv preprint arXiv:2304.13731_ , 2023.
  * Gygli et al. (2014) Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Luc Van Gool.  Creating summaries from user videos.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VII 13_ , pp. 505–520. Springer, 2014.
  * Han et al. (2024) Donghoon Han, Seunghyeon Seo, Eunhwan Park, Seong-Uk Nam, and Nojun Kwak.  Unleash the potential of clip for video highlight detection.  _arXiv preprint arXiv:2404.01745_ , 2024.
  * Hendricks et al. (2018a) Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell.  Localizing moments in video with temporal language.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , 2018a.
  * Hendricks et al. (2018b) Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell.  Localizing moments in video with temporal language.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , 2018b.
  * Hua et al. (2024) Hang Hua, Yunlong Tang, Chenliang Xu, and Jiebo Luo.  V2xum-llm: Cross-modal video summarization with temporal prompt instruction tuning.  _arXiv preprint arXiv:2404.12353_ , 2024.
  * Huang et al. (2023) Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu.  Vtimellm: Empower llm to grasp video moments.  _arXiv preprint arXiv:2311.18445_ , 2(3):9, 2023\.
  * Huang et al. (2024) De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz.  Lita: Language instructed temporal-localization assistant.  _arXiv preprint arXiv:2403.19046_ , 2024.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.
  * Kozachenko & Leonenko (1987) Lyudmyla F Kozachenko and Nikolai N Leonenko.  Sample estimate of the entropy of a random vector.  _Problemy Peredachi Informatsii_ , 23(2):9–16, 1987.
  * Lei et al. (2021) Jie Lei, Tamara L Berg, and Mohit Bansal.  Detecting moments and highlights in videos via natural language queries.  _Advances in Neural Information Processing Systems_ , 34:11846–11858, 2021.
  * Li et al. (2023a) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In _International conference on machine learning_ , pp. 19730–19742. PMLR, 2023a.
  * Li et al. (2023b) Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao.  Videochat: Chat-centric video understanding.  _arXiv preprint arXiv:2305.06355_ , 2023b.
  * Li et al. (2023c) Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao.  Mvbench: A comprehensive multi-modal video understanding benchmark, 2023c.
  * Li et al. (2023d) Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao.  Unmasked teacher: Towards training-efficient video foundation models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 19948–19960, 2023d.
  * Lin et al. (2023a) Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan.  Video-llava: Learning united visual representation by alignment before projection.  _arXiv preprint arXiv:2311.10122_ , 2023a.
  * Lin et al. (2023b) Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou.  Univtg: Towards unified video-language temporal grounding.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 2794–2804, 2023b.
  * Liu et al. (2024) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _Advances in neural information processing systems_ , 36, 2024.
  * Luo et al. (2023a) Dezhao Luo, Jiabo Huang, Shaogang Gong, Hailin Jin, and Yang Liu.  Towards generalisable video moment retrieval: Visual-dynamic injection to image-text pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 23045–23055, 2023a.
  * Luo et al. (2023b) Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei.  Valley: Video assistant with large language model enhanced ability.  _arXiv preprint arXiv:2306.07207_ , 2023b.
  * Maaz et al. (2023) Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan.  Video-chatgpt: Towards detailed video understanding via large vision and language models.  _arXiv preprint arXiv:2306.05424_ , 2023.
  * Moreno-Torres et al. (2012) Jose G Moreno-Torres, Troy Raeder, Rocío Alaiz-Rodríguez, Nitesh V Chawla, and Francisco Herrera.  A unifying view on dataset shift in classification.  _Pattern recognition_ , 45(1):521–530, 2012.
  * Oncescu et al. (2021) Andreea-Maria Oncescu, Joao F Henriques, Yang Liu, Andrew Zisserman, and Samuel Albanie.  Queryd: A video dataset with high-quality text and audio narrations.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 2265–2269. IEEE, 2021.
  * Puigcerver et al. (2023) Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby.  From sparse to soft mixtures of experts.  _arXiv preprint arXiv:2308.00951_ , 2023.
  * Qian et al. (2024) Long Qian, Juncheng Li, Yu Wu, Yaobo Ye, Hao Fei, Tat-Seng Chua, Yueting Zhuang, and Siliang Tang.  Momentor: Advancing video large language model with fine-grained temporal reasoning, 2024.
  * Ren et al. (2023) Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou.  Timechat: A time-sensitive multimodal large language model for long video understanding.  _arXiv preprint arXiv:2312.02051_ , 2023.
  * Song et al. (2015) Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes.  Tvsum: Summarizing web videos using titles.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 5179–5187, 2015.
  * Sun et al. (2023) Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao.  Eva-clip: Improved training techniques for clip at scale.  _arXiv preprint arXiv:2303.15389_ , 2023.
  * Tang et al. (2019) Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou.  Coin: A large-scale dataset for comprehensive instructional video analysis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 1207–1216, 2019.
  * Tong et al. (2022) Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.  Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  _Advances in neural information processing systems_ , 35:10078–10093, 2022.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4566–4575, 2015.
  * Wang et al. (2022) Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al.  Internvideo: General video foundation models via generative and discriminative learning.  _arXiv preprint arXiv:2212.03191_ , 2022.
  * Wang et al. (2024a) Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al.  Internvideo2: Scaling video foundation models for multimodal video understanding.  _arXiv preprint arXiv:2403.15377_ , 2024a.
  * Wang et al. (2024b) Yueqian Wang, Xiaojun Meng, Jianxin Liang, Yuxuan Wang, Qun Liu, and Dongyan Zhao.  Hawkeye: Training video-text llms for grounding text in videos.  _arXiv preprint arXiv:2403.10228_ , 2024b.
  * Xiao et al. (2023) Yicheng Xiao, Zhuoyan Luo, Yong Liu, Yue Ma, Hengwei Bian, Yatai Ji, Yujiu Yang, and Xiu Li.  Bridging the gap: A unified video comprehension framework for moment retrieval and highlight detection.  _arXiv preprint arXiv:2311.16464_ , 2023.
  * Xu et al. (2021) Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer.  Videoclip: Contrastive pre-training for zero-shot video-text understanding.  _arXiv preprint arXiv:2109.14084_ , 2021.
  * Xu et al. (2024) Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng.  Pllava: Parameter-free llava extension from images to videos for video dense captioning.  _arXiv preprint arXiv:2404.16994_ , 2024.
  * Yan et al. (2022) Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu.  Videococa: Video-text modeling with zero-shot transfer from contrastive captioners.  _arXiv preprint arXiv:2212.04979_ , 2022.
  * Yang et al. (2023) Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid.  Vid2seq: Large-scale pretraining of a visual language model for dense video captioning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10714–10726, 2023.
  * Zala et al. (2023a) Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas Oguz, Yashar Mehdad, and Mohit Bansal.  Hierarchical video-moment retrieval and step-captioning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 23056–23065, 2023a.
  * Zala et al. (2023b) Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas Oğuz, Yashar Mehdad, and Mohit Bansal.  Hierarchical video-moment retrieval and step-captioning.  In _CVPR_ , 2023b.
  * Zellers et al. (2021) Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi.  Merlot: Multimodal neural script knowledge models.  In _Advances in Neural Information Processing Systems 34_ , 2021.
  * Zhang et al. (2023) Hang Zhang, Xin Li, and Lidong Bing.  Video-llama: An instruction-tuned audio-visual language model for video understanding.  _arXiv preprint arXiv:2306.02858_ , 2023.
  * Zhao et al. (2024) Long Zhao, Nitesh B Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, et al.  Videoprism: A foundational visual encoder for video understanding.  _arXiv preprint arXiv:2402.13217_ , 2024.
  * Zhou et al. (2018) Luowei Zhou, Chenliang Xu, and Jason Corso.  Towards automatic learning of procedures from web instructional videos.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_ , 2023.

###### Contents of Appendix

  1. A Experiments
  2. B Data Annotation Examples
  3. C Additional Case Studies

##  Appendix A Experiments

###  A.1 Detailed Experiment Settings

In this section, we present the detailed experimental settings used in our
study. We employ ViT-G/14 from EVA-CLIP (Sun et al., 2023) and Qformer from
InstructBLIP (Dai et al., 2024) as the visual backbone. The number of queries
for Qformer is set to 32. The vision projection layer and the weights of the
sequence embedding are initialized using the Video-LLaMA checkpoint (Zhang et
al., 2023). To address the mismatched size of the sequence embedding, we use
interpolation to extend the size from 32 to 96. The time-embedding (8192) is
initialized with zeros. We set the number of slots to 256.

For the language model backbone, we utilize LLaMA-2-7B (Touvron et al., 2023).
The LoRA training is conducted on "q_proj", "k_proj", "v_proj", and "o_proj".
Special time tokens are added to the LLaMA tokenizer, and we set the
prediction head of LLM and the weights of token embedding to be trainable. The
maximum text length is set to 2048.

The models are trained on the VTG-IT-120k and a randomly sampled subset (97k)
from the Valley dataset (Luo et al., 2023b). We randomly sample 96 frames for
each video. Training is carried out using 16 ATN 910B GPUs with a batch size
of 64 and takes approximately 40 hours to complete. The training employs DDP
and requires about 30GB of GPU storage for each GPU. Fine-tuning also uses 16
GPUs and sets the batch size to 16. We fine-tune the VTG-LLM model for 10
epochs on Charades dataset, 16 epochs for Youcook2 dataset, and 20 epochs for
QVHighlights dataset. We set the initial learning rate to 3e-5 and train the
models for 10 epochs by default. The weight decay is set to 0.05, and the
warm-up steps use $0.6E$, where $E$ represents the number of iterations for
each epoch.

##  Appendix B Data Annotation Examples

We present annotation examples for moment retrieval (Figure 4), dense video
captioning (Figure 5), video highlight detection (Figure 6), and video
summarization (Figure 7). It should be noted that in the VTG-IT-VHD dataset,
only clips with high scores are annotated. This leads to a relatively
consistent score distribution among the annotated clips. As a result, the VTG-
IT-VHD dataset assists models in locating relevant video clips; however, the
significant scores may not be accurately predicted in a zero-shot manner due
to differing scoring criteria. This issue can be addressed by fine-tuning the
models on target datasets, which helps the models understand the specific
scoring criteria.

Figure 4: Annotation example of moment retrieval task. Figure 5: Annotation
example of dense video captioning task. Figure 6: Annotation example of dense
video highlight detection task. It should be noted that in the VTG-IT-VHD
dataset, only clips with high scores are annotated. This leads to a relatively
consistent score distribution among the annotated clips. Figure 7: Annotation
example of dense video summarization task.

##  Appendix C Additional Case Studies

We show the additional zero-shot case studies on dense video captioning
(Figure 8), moment retrieval (Figure 9), and video highlight detection (Figure
10) tasks.

Figure 8: Zero-shot case study of dense video captioning task. Figure 9: Zero-
shot case study of moment retrieval task. Figure 10: Zero-shot case study of
video highlight detection task.


[affiliation=1,2*]WilliamRavenscroft [affiliation=1]GeorgeClose
[affiliation=1]StefanGoetze [affiliation=1]ThomasHain [affiliation=2]
MohammadSoleymanpour [affiliation=2]AnuragChowdhury [affiliation=2]Mark C.Fuhs

# Transcription-Free Fine-Tuning of Speech Separation Models for Noisy and
Reverberant Multi-Speaker Automatic Speech Recognition ††thanks:  This work
was supported by the Centre for Doctoral Training in Speech and Language
Technologies (SLT) and their Applications funded by UK Research and Innovation
[grant number EP/S023062/1]. This work was also supported in part by Solventum
and Toshiba Cambridge Research Laboratory.
* Work done while an intern at Solventum, formerly part of 3M.

###### Abstract

One solution to automatic speech recognition (ASR) of overlapping speakers is
to separate speech and then perform ASR on the separated signals. Commonly,
the separator produces artefacts which often degrade ASR performance.
Addressing this issue typically requires reference transcriptions to jointly
train the separation and ASR networks. This is often not viable for training
on real-world in-domain audio where reference transcript information is not
always available. This paper proposes a transcription-free method for joint
training using only audio signals. The proposed method uses embedding
differences of pre-trained ASR encoders as a loss with a proposed modification
to permutation invariant training (PIT) called guided PIT (GPIT). The method
achieves a $6.4$% improvement in word error rate (WER) measures over a signal-
level loss and also shows enhancement improvements in perceptual measures such
as short-time objective intelligibility (STOI).

###### keywords:

speech recognition, speech separation, multi-speaker, adaptation, fine-tuning

##  1 Introduction

While significant progress has been made in recent years in multi-speaker ASR
research, it remains a challenging problem [1, 2]. Many different approaches
have been proposed to solving this problem [3, 4, 5]. These methods include
entirely end-to-end single models [4] and modular approaches [6, 2, 5].

Modular approaches often include a variety of sub-components such as speech
separation [7], speaker diarization [8] and ASR [6, 5]. Sometimes these sub-
components, or modules, can be combined in an end-to-end fashion that allows
for joint training of the modules [9, 3]. More recently, serialized output
training (SOT) was proposed, a technique that enables the design of explicitly
multispeaker ASR models with a single output sequence containing subsequences
separated by a unique token for each speaker [4]. There is a benefit to having
a simplified model that requires no fine-tuning on sub-components but this
results in lower interpretability. A commonality of both approaches to multi-
speaker ASR is that they are reliant on the existence of ground truth
transcriptions of the speech of each speaker in the mixture. Whilst this is
easily obtainable for simulated mixtures, it is often impossible to obtain for
`real-world' in-domain speech mixtures.

In [10, 11], computing embedding differences between self-supervised speech
representations was used as a loss for training speech enhancement networks to
improve perceptual speech quality. Furthermore, in [12] it was shown that
computing the difference between senone representations of ASR models can be
used to adapt speech enhancement networks for robust ASR. Recent work [13, 14]
has shown that features derived from ASR models can capture quality and
intelligibility-related information.

In this work, a novel method for transcription-free fine-tuning of a separate-
and-recognize modular approach is proposed. Embedding differences from a pre-
trained ASR encoder are used to compute a loss from the reference speech and
separated speech signals. Furthermore, a variant of the the standard PIT [15]
algorithm, guided permutation invariant training (GPIT) is proposed to
properly address speaker permutation solving of the ASR encoder embeddings
within the proposed loss function. This recognition-based loss term is then
used to fine-tune a speech separator. The proposed method notably improves
multi-speaker WER performance over traditional signal-level losses across
multiple ASR models, and also shows improved performance in intrusive
perceptual measures. Another benefit of the proposed approach is that it
allows for fine-tuning separators using truncated audio signals [16] as the
full transcription is not required, potentially resulting in accelerated
training, lower memory requirements and reduced computational expenditure. The
proposed method also makes training on real-world data, with pseudo-reference
audio used as input to the proposed loss function, possible [17] .

The remainder of this paper proceeds as follows: in Section 2, the proposed
transcription-free fine-tuning method is described in detail. In Section 3,
the experimental setup and data are described. Results and conclusions are
given in Section 4 and Section 5, respectively.

##  2 Transcription-Free Fine-Tuning Method

In this section, the main components of the proposed transcription-free fine-
tuning method are described. A modularized approach to multi-speaker ASR is
used; this approach is referred to here as the separate-and-recognize
approach. A noisy reverberant mixture $x[i]$ of $C$ speech signals $s_{c}[i]$,
$c\in\\{1,\ldots,C\\}$ for time index $i$ is defined as

$x[i]=\sum_{c=1}^{C}h_{c}[i]\ast s_{c}[i]+\nu[i],$ (1)

where $\ast$ denotes the convolution operator, $h_{c}[i]$ is the room impulse
response corresponding to speaker $c$ and $\nu[i]$ denotes additive noise. In
the separate-and-recognize approach, a separation model firstly separates the
mixture signal $x[i]$ in $C$ estimated speech signals $\hat{s}_{c}[i]$ and
each separated signal is fed into an ASR module to attain predicted
transcription of discrete character tokens $\hat{\mathbf{t}}_{c}$ [2].

The proposed fine-tuning method assumes the existence of pre-trained speech
separation and ASR networks, where the speech separation model has been
trained using a conventional signal-level objective function such as scale-
invariant signal-to-distortion ratio (SISDR) [18]. In the proposed method,
these two modules go through an additional number of addtional training
epochs, whereby the parameters of the ASR model are frozen and the embedding
differences of the ASR encoder are backpropagated through the separation
network where the parameters are updated at each step. The proposed approach
is shown in Figure 1 compared to the baseline model trained with a purely
signal-level SISDR loss.

Figure 1: (a) Baseline approach to training speech separators without ASR-
based fine-tuning. (b) Proposed fine-tuning approach without using reference
transcriptions. Solid lines indicate information flow; dashed lines the
direction of gradient backpropagation. Figure exemplary for $C=2$ speakers.

###  2.1 ASR Encoder Loss

This section describes the proposed ASR encoder (AE) loss used for fine-tuning
the speech separation model. In this work the ASR encoder used is a
connectionist temporal classification (CTC)-based model [19] with a discrete
number of possible output symbols $N$. The encoder network is defined as a
function

$\mathcal{V}:\mathbb{R}^{L_{x}}\mapsto\mathbb{R}^{L\times N}$ (2)

where $L_{x}$ is the length of an input speech signal $s_{c}[i]$, $L$ is the
output sequence length and $N$ is the number of possible symbols, i.e. the
number of characters that can be interpreted by a decoder or decoding function
(minimum of $26$ plus a word boundary token and a blank symbol for the Latin
alphabet).

The proposed AE loss computes the difference between the encoder output of the
predicted speech signals $\hat{s}_{c}$ and reference speech signal ${s}_{c}$,
$\forall c\in\\{1,\ldots,C\\}$. The loss is the mean square error (MSE) of the
two output sequences defined as

$\mathcal{L}_{\mathrm{AE}}\left(\mathbf{s}_{c},\hat{\mathbf{s}}_{c}\right)=\frac{1}{LN}\sum_{\ell=1}^{L}\sum_{w=1}^{N}\left(\mathcal{V}(\hat{\mathbf{s}}_{c})_{\ell,n}-\mathcal{V}(\mathbf{s}_{c})_{\ell,n}\right)^{2}$
(3)

where ${s}_{c}$ is the target audio and $\hat{s}_{c}$ is the predicted output
audio of the seperator for speaker $c$. The $\mathcal{L}_{\mathrm{AE}}$ loss
is analogous to a conventional time-frequency based loss, but instead of
comparing references and outputs in terms of continuous time/frequency bins,
the proposed loss instead compares in terms of discrete time/character label
logits.

###  2.2 Guided Permutation Invariant Training (GPIT)

Empirically it was found that the standard PIT algorithm [20] was not robust
enough against distortions in the ASR encoder features when applying the
proposed AE loss function in (3), resulting in inaccuracies in resolving
speaker permutations. This resulted in models trained using the AE loss
diverging and giving $100\%$ WER in evaluations. Thus, a modified scheme
referred to as guided permutation invariant training (GPIT) (not to be
confused with Graph-PIT [21]) is proposed that uses an alternate signal-level
loss to guide the permutation solving, and then the AE loss is applied to the
minimum permutation estimates and references. The permutation-solving formula
is defined the same as for PIT. The predicted permutation $\hat{\phi}$ is
determined by

$\hat{\phi}=\operatorname*{arg\,min}_{\phi\in{\mathbf{\Phi}}}\sum_{c=1}^{C}\mathcal{L}_{\mathrm{guide}}\left(\hat{\mathbf{s}}_{c}^{(\phi)},\mathbf{s}_{c}\right)$
(4)

where $\phi$ is a permutation in the set $\mathbf{\Phi}$ of $\Phi=C!$ possible
permutations and $\hat{\mathbf{s}}_{c}^{(\phi)}$ denotes the $\phi$th possible
permutation of $\hat{\mathbf{s}}_{c}$. $\mathcal{L}_{\mathrm{guide}}$ denotes
the permutation guiding loss. In this paper
$\mathcal{L}_{\mathrm{guide}}=\mathcal{L}_{\mathrm{SISDR}}$ is chosen. Using
the predicted permutation the desired loss, in this case
$\mathcal{L}_{\mathrm{AE}}(\cdot)$ from (3), is calculated s.t. the final loss
value is

$\mathcal{L}_{\mathrm{GPIT}}\left(\mathbf{s}_{c},\hat{\mathbf{s}}_{c},\hat{\phi}\right)=\sum_{c=1}^{C}\mathcal{L}_{\mathrm{AE}}\left(\mathbf{s}_{c},\hat{\mathbf{s}}_{c}^{(\hat{\phi})}\right).$
(5)

Table 1: ASR and speech enhancement performance comparison of the AE loss in (3) to baseline SISDR loss models as well as oracle signal $s_{c}[i]$ and mixture signal $x[i]$ after a set number of ATEs. $\Delta$ indicates improvement over the mixture signal. Bold font indicates the best result for each metric. Arrows indicate performance increase. [HTML]C0C0C0 ASR Input | Loss | ATEs |  CP-WER $\downarrow$ |  $\Delta$ $\uparrow$ |  ORC-WER $\downarrow$ |  $\Delta$ $\uparrow$ |  SISDR $\uparrow$ |  PESQ $\uparrow$ |  STOI $\uparrow$ |  SRMR $\uparrow$
---|---|---|---|---|---|---|---|---|---|---
Oracles $s_{c}$ | - | - | 7.7 | - | 7.7 | - | - | - | - | -
Mixture $x$ | - | - | 85.7 | - | 81.8 | - | -7.2 dB | 1.05 | 27.1 | 2.99
Estimates $\hat{s}_{c}$ | $\mathcal{L}_{\mathrm{SISDR}}$ | - | 44.2 | 41.5 | 44.2 | 37.6 | 5.5 dB | 1.41 | 69.3 | 7.14
Estimates $\hat{s}_{c}$ | $\mathcal{L}_{\mathrm{SISDR}}$ | 30 | 43.5 | 42.3 | 43.4 | 38.4 |  5.6 dB | 1.44 | 71.9 | 7.21
Estimates $\hat{s}_{c}$ |  $\mathcal{L}_{\mathrm{AE}}$ (proposed) | 30 | 37.1 | 48.7 | 37.1 | 44.8 | 5.1 dB | 1.54 | 75.0 | 7.27

##  3 Experimental Setup

###  3.1 Data

The WHAMR [22] corpus is used for all experiments in this work. This corpus is
an extension of the WSJ0-2Mix corpus [23] which in turn is an extension of the
WSJ0 corpus [24]. WHAMR is a corpus of noisy, reverberant, two-speaker
mixtures created from overlapping two artificially reverbed speaker utterances
from WSJ0 at mixing signal-to-noise ratios of $0$-$5$ dB and then adding an
ambient noise source at an SNR of $-6$-$3$ dB relative to the loudest speaker.
The $16$kHz max (non-truncated) configuration of the corpus is used. Dynamic
mixing [25] is also used, whereby the training set is uniquely simulated at
each epoch to improve data diversity and model generalization. Signal lengths
are also limited to $4$s in length to speed up training as well as reduce
computational expenditure and memory consumption [16]. Further analysis of
training signal length (TSL) limits is given in Section 4.4.

###  3.2 Speech Separator

The TD-Conformer-XL model is chosen for speech separation as it gives close to
state-of-the-art performance on the WHAMR benchmark and is open source111TD-
Conformer training recipe available at https://github.com/jwr1995/PubSep [26].
The configuration is based on the best-performing model on WHAMR in [26] with
$S=1$ subsampling layers, a kernel size of $P=64$ and feature dimension
$B=1024$. To compensate for the larger sampling rate used in this paper
($8$kHz vs $16$kHz here), the number of subsampling layers is set to $S=2$.
This model is pre-trained on $4$s audio clips with a learning rate of
$\eta_{\mathrm{PT}}=5\times 10^{-5}$ over 150 epochs. The learning rate is
halved after $90$ epochs if there is no improvement in $3$ epochs.

###  3.3 Speech Recognizers

The large SSSR-based Wav2Vec2 model [27] fine-tuned on 960 hours of
LibriSpeech data with a CTC loss is chosen222Torchaudio Wav2Vec2 Large model
available at
https://pytorch.org/audio/0.10.0/pipelines.html#torchaudio.pipelines.WAV2VEC2_ASR_LARGE_960H
as the primary model used to obtain the embeddings $\mathcal{V}(\cdot)$ in the
AE loss in (3), as well as for evaluating ASR performance. The Wav2Vec2 model
is a large transformer SSSR model that is initially trained to predict speech
representations from time-domain signals via self-supervision where the model
is trained to predict contextualized representations across masked portions of
the input signal from the unmasked signal context [27]. The fine-tuning of
Wav2Vec2 involves training an additional linear layer to map these
representations to a CTC specific representations, in this case, grapheme-
based representations where each logit represents a possible grapheme. Note
that the fine-tuning in this model is unrelated to the proposed fine-tuning
method described in Section 2. In the large Wav2Vec2 model used in the
following experiments, the number of labels is $N=30$ ($28$ characters
interpreted verbatim plus word boundary token ``|" and blank symbol ``-"). In
the CTC decoder of the Wav2Vec2 ASR model, an open-source 4-gram Librispeech
language model is used333Librispeech language model and other resources
available at: https://www.openslr.org/11/..

In addition, the large Whisper model444In this work, Whisper ’large-v2’ is
used, which can be downloaded from: https://github.com/openai/whisper. [28] is
used as an unseen ASR evaluation model for evaluating ASR performance on a
model that was not used in the fine-tuning of the separator. Whisper is a
weakly supervised speech foundation model that uses multi-task training. The
weakly supervised and multi-task approaches are designed to make the ASR model
generalisation well across many acoustic conditions and speaker types [28].
Thus this well-generalising ASR model is thought to make it more challenging
for the proposed approach to achieve improvements over the baseline SISDR
models.

###  3.4 Fine-Tuning

For fine-tuning with the AE loss and GPIT, a learning rate of
$\eta_{\mathrm{FT}}=2\times 10^{-7}$ is used. Fine-tuning is performed over
$30$ addtional training epochs (ATEs) with the learning rate fixed for all
epochs. Exploring different learning rate strategies and optimizing this
hyperparameter is beyond the scope of this paper and left to future work.

###  3.5 Evaluation Metrics

Several options are available for multi-speaker WER evaluation [29]. In this
paper, two such definitions are chosen: concatenated minimum-power word error
rate (CP-WER) [6] and optimal reference combination word error rate (ORC-WER)
[3]. The key difference between the two measures is that CP-WER penalizes
output speaker channel switches and ORC-WER is unconcerned with whether a
given speaker is output on a single channel or multiple channels, so long as
the ASR model is still able to estimate the word accurately. CP-WER is thus
the more important measure as, ideally, in the proposed system, the goal is to
have one speaker for each output channel. However, the addition of ORC-WER
provides additional insight into the overall intelligibility of the speech
regardless of which channel(s) a given speaker gets output to. Intrusive
speech enhancement measures are also used to observe any benefit gained in
these using the proposed approach. STOI [30] is used to measure speech
intelligibility, perceptual evaluation of speech quality (PESQ) [31] is used
to measure speech quality and speech-to-reverberation modulation energy ratio
(SRMR) [32] is used to assess reverberant effects.

##  4 Results

###  4.1 Results on clean targets

The performance of the AE loss function (3) using the large Wav2Vec2 model is
shown in Table 1. The TD-Conformer separation network trained with the
proposed AE loss for $30$ ATEs is compared to the TD-Conformer separator
before fine-tuning and the TD-Conformer trained with an additional $30$
epochs, but using the standard signal-level SISDR loss [26, 18]. The model
trained with the proposed AE loss significantly outperforms both models in ASR
performance (CP-WER and ORC-WER). The AE loss-based model also outperforms the
others in terms of the speech enhancement metrics PESQ, STOI and SRMR. This is
a powerful finding, as often improved perceptual performance leads to degraded
ASR performance and vice-versa. It is consistent however with prior work [10]
which similarly compares SSSR output representations in a speech enhancement
system loss function.

###  4.2 Generalization to an Unseen ASR System

Table 2: ASR performance for the re-evaluation of models in Table 1 using the large Whisper model [28]. Bold indicates the best-performing trained model. $\Delta$ indicates improvement over the mixture signal. [HTML]C0C0C0 ASR Input | Loss | ATEs |  CP-WER $\downarrow$ |  ORC-WER $\downarrow$
---|---|---|---|---
Oracles $s_{c}$ | - | - | 10.9 | 10.9
Mixture $x$ | - | - | 63.3 | 60.0
Estimates $\hat{s}_{c}$ | $\mathcal{L}_{\mathrm{SISDR}}$ | - | 29.5 | 29.4
Estimates $\hat{s}_{c}$ | $\mathcal{L}_{\mathrm{SISDR}}$ | 30 | 29.3 | 29.2
Estimates $\hat{s}_{c}$ | $\mathcal{L}_{\mathrm{AE}}$ | 30 | 26.8 | 26.7

The generalization of the improvements found using the proposed AE loss in
Table 1 is analysed in this subsection by reevaluating models on the large
Whisper ASR model [28] that was not used in the fine-tuning stages. The
results in Table 2 show a consistent improvement in both CP-WER and ORC-WER
for the AE loss over the SISDR loss trained with additional epochs
demonstrating that at least some of the improvements can generalise from one
ASR model to another.

###  4.3 Joint SISDR Loss Weighting

To further assess the impact of the proposed AE loss
$\mathcal{L}_{\mathrm{AE}}$ against the SISDR loss
$\mathcal{L}_{\mathrm{SISDR}}$ [26, 18], a series of experiments fine-tuning
the separator with a joint weighted loss of the two different loss types is
carried out. The joint loss is defined as

$\mathcal{L}_{\mathrm{Joint}}=(1-\alpha)\mathcal{L}_{\mathrm{AE}}+\alpha\mathcal{L}_{\mathrm{SISDR}}$
(6)

where $\alpha$ controls the weighting of the two loss terms. Five models are
fine-tuned with values of $\alpha\in\\{0.0,0.2,0.4,0.6,0.8,1.0\\}$ for $30$
ATEs.

Figure 2: ASR performance (CP-WER) and objective perceptual quality of test
audio for models trained with differing weight $\alpha$ between loss terms in
(6).

Figure 2 shows the performance of the models fine-tuned using (6) in terms of
CP-WER and PESQ on the WHAMR test set. The ASR performance of the models
trained with lower values of $\alpha$ is significantly better than those
trained with higher values, as the influence of the proposed
$\mathcal{L}_{\mathrm{AE}}$ function on the models' training is higher for
lower values of $\alpha$. The biggest jump in improvement is between
$\alpha=1.0$ (i.e no use of $\mathcal{L}_{\mathrm{AE}}$) and $\alpha=0.8$,
suggesting that even a small weighting of the proposed loss can significantly
improve ASR performance. Perceptual quality in terms of PESQ does not change
as uniformly or significantly for lower $\alpha$ values, but a trend is still
apparent; models with a greater weighting of the proposed loss
$\mathcal{L}_{\mathrm{AE}}$ produce higher quality audio.

###  4.4 Training Signal Length Analysis

A benefit of the transcription-free method is the ability to train on
arbitrary signal lengths. In transcription-dependant losses, this is non-
trivial due to the requirement of truncating the transcription in alignment
with the audio. The impact of applying TSL limits [16, 33] is analysed in
Table 3.

Table 3: Comparison of the impact of the TSL limit on ASR performance of models fine-tuned using the proposed AE loss over 30 ATEs. Best results are shown in bold. [HTML]C0C0C0 TSL limit (s) |  CP-WER $\downarrow$ |  ORC-WER $\downarrow$ |  SISDR $\uparrow$
---|---|---|---
2.0 | 38.3 | 38.4 | 5.0
4.0 | 37.1 | 37.1 | 5.1
8.0 | 36.3 | 36.2 | 5.2

The results show that using longer signal lengths gives slightly better
performance, most likely due to the ASR Encoder having more context. This is
opposite to the SISDR-based evaluations in [16] where improved SISDR
performance could be obtained on WHAMR using shorter signal lengths.

###  4.5 ASR Encoder Visualization

A visual comparison of Wav2Vec2 ASR Encoder output representations
$\mathcal{V}(\cdot)$ used in (3) of audio outputs from models trained using
SISDR loss (middle) and AE loss (bottom) can be seen in Figure 3. The ASR
encoder output representation $\mathcal{V}(\cdot)$ of the respective reference
audio is visualized in the top panel.

Figure 3: Wav2Vec2 ASR Encoder output representations for reference audio
$\mathcal{V}(s[n])$ (top), and $\mathcal{V}(\hat{s}[n])$ for models with
baseline $\mathcal{L}_{\mathrm{SISDR}}$ (middle) and proposed
$\mathcal{L}_{\mathrm{AE}}$ fine-tuning (bottom) losses

Notably, the AE loss reduces the uncertainty in the logits when compared to
the SISDR loss. In both representations of the estimated signals, there is
some bleed-through of interfering noise and speakers at $L\gtrapprox 180$, but
this effect is reduced in the AE loss suggesting that the additional context
provided by the encoder network may improve separation performance and reduce
more general distortions.

##  5 Conclusions

This paper presented a novel method for fine-tuning speech separation models
for multi-speaker ASR without the need for reference transcriptions. It was
shown that the proposed loss leveraging pre-trained ASR encoder
representations for fine-tuning separators results in improved ASR performance
over a standard signal-level-based loss. It was also demonstrated that the
proposed method produces improved performance in speech enhancement metrics
such as PESQ and STOI which is often not the case for many other approaches.
Finally, it was shown that the performance gained by the proposed method
generalises to other ASR models that weren't used in the fine-tuning stage.

## References

  * [1] R. Haeb-Umbach, J. Heymann, L. Drude, S. Watanabe, M. Delcroix, and T. Nakatani, ``Far-field automatic speech recognition,'' _Proceedings of the IEEE_ , vol. 109, no. 2, pp. 124–148, 2021.
  * [2] T. Cord-Landwehr, C. Boeddeker, T. Von Neumann, C. Zorilă, R. Doddipatla, and R. Haeb-Umbach, ``Monaural source separation: From anechoic to reverberant environments,'' in _IWAENC 2022_ , Sep. 2022, pp. 1–5.
  * [3] I. Sklyar, A. Piunova, X. Zheng, and Y. Liu, ``Multi-turn rnn-t for streaming recognition of multi-party speech,'' in _ICASSP 2022_ , May 2022, pp. 8402–8406.
  * [4] N. Kanda, Y. Gaur, X. Wang, Z. Meng, and T. Yoshioka, ``Serialized Output Training for End-to-End Overlapped Speech Recognition,'' in _Proc. Interspeech 2020_ , 2020, pp. 2797–2801.
  * [5] T. von Neumann, C. Boeddeker, T. Cord-Landwehr, M. Delcroix, and R. Haeb-Umbach, ``Meeting recognition with continuous speech separation and transcription-supported diarization,'' 2023. [Online]. Available: https://arxiv.org/abs/2309.16482
  * [6] S. Watanabe, M. Mandel, J. Barker, E. Vincent, A. Arora, X. Chang, S. Khudanpur, V. Manohar, D. Povey, D. Raj, D. Snyder, A. S. Subramanian, J. Trmal, B. B. Yair, C. Boeddeker, Z. Ni, Y. Fujita, S. Horiguchi, N. Kanda, T. Yoshioka, and N. Ryant, ``CHiME-6 Challenge: Tackling Multispeaker Speech Recognition for Unsegmented Recordings,'' in _CHiME 2020_ , May 2020, pp. 1–7.
  * [7] W. Ravenscroft, S. Goetze, and T. Hain, ``Att-TasNet: Attending to Encodings in Time-Domain Audio Speech Separation of Noisy, Reverberant Speech Mixtures,'' _Frontiers in Signal Processing_ , vol. 2, 2022.
  * [8] J. Clarke, Y. Gotoh, and S. Goetze, ``Improving audiovisual active speaker detection in egocentric recordings with the data-efficient image transformer,'' in _ASRU 2023_ , Dec. 2023, pp. 1–8.
  * [9] X. Chang, W. Zhang, Y. Qian, J. L. Roux, and S. Watanabe, ``MIMO-Speech: End-to-End Multi-Channel Multi-Speaker Speech Recognition,'' in _ASRU 2019_ , Dec. 2019, pp. 237–244.
  * [10] G. Close, W. Ravenscroft, T. Hain, and S. Goetze, ``Perceive and predict: Self-supervised speech representation based loss functions for speech enhancement,'' in _ICASSP 2023_ , Jun. 2023, pp. 1–5.
  * [11] ——, ``Multi-CMGAN+/+: Leveraging Multi-Objective Speech Quality Metric Prediction for Speech Enhancement,'' in _ICASSP 2024_ , Apr. 2024, pp. 351–355.
  * [12] D. Bagchi, P. Plantinga, A. Stiff, and E. Fosler-Lussier, ``Spectral feature mapping with mimic loss for robust speech recognition,'' in _ICASSP 2018_ , Apr. 2018, pp. 5609–5613.
  * [13] Z. Tu, N. Ma, and J. Barker, ``Exploiting Hidden Representations from a DNN-based Speech Recogniser for Speech Intelligibility Prediction in Hearing-impaired Listeners,'' in _Proc. Interspeech 2022_ , Sep. 2022, pp. 3488–3492.
  * [14] R. Mogridge, G. Close, R. Sutherland, T. Hain, J. Barker, S. Goetze, and A. Ragni, ``Non-intrusive speech intelligibility prediction for hearing-impaired users using intermediate asr features and human memory models,'' in _ICASSP 24_ , May 2024, pp. 306–310.
  * [15] D. Yu, M. Kolbæk, Z.-H. Tan, and J. Jensen, ``Permutation invariant training of deep models for speaker-independent multi-talker speech separation,'' in _ICASSP 2017_ , Mar. 2017, pp. 241–245.
  * [16] W. Ravenscroft, S. Goetze, and T. Hain, ``On Data Sampling Strategies for Training Neural Network Speech Separation Models,'' in _EUSIPCO 2023_ , Sep. 2023, pp. 331–335.
  * [17] J. Kalda, C. Pagés, R. Marxer, T. Alumäe, and H. Bredin, ``PixIT: Joint training of speaker diarization and speech separation from real-world multi-speaker recordings,'' 2024. [Online]. Available: https://arxiv.org/abs/2403.02288
  * [18] Y. Luo and N. Mesgarani, ``Tasnet: Time-domain audio separation network for real-time, single-channel speech separation,'' in _ICASSP 2018_ , Apr. 2018, pp. 696–700.
  * [19] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, ``Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,'' in _ICML 2006_ , Jun. 2006, p. 369–376.
  * [20] M. Kolbaek, D. Yu, Z.-H. Tan, J. Jensen, M. Kolbaek, D. Yu, Z.-H. Tan, and J. Jensen, ``Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks,'' _IEEE/ACM Trans. Audio, Speech and Lang. Proc._ , vol. 25, no. 10, p. 1901–1913, Oct. 2017.
  * [21] T. von Neumann, K. Kinoshita, C. Boeddeker, M. Delcroix, and R. Haeb-Umbach, ``Graph-PIT: Generalized Permutation Invariant Training for Continuous Separation of Arbitrary Numbers of Speakers,'' in _Proc. Interspeech 2021_ , 2021, pp. 3490–3494.
  * [22] M. Maciejewski, G. Wichern, E. McQuinn, and J. L. Roux, ``WHAMR!: Noisy and Reverberant Single-Channel Speech Separation,'' in _ICASSP 2020_ , May 2020, pp. 696–700.
  * [23] Y. Isik, J. L. Roux, Z. Chen, S. Watanabe, and J. R. Hershey, ``Single-Channel Multi-Speaker Separation Using Deep Clustering,'' in _Proc. Interspeech 2016_ , Sep. 2016, pp. 545–549.
  * [24] D. B. Paul and J. M. Baker, ``The design for the Wall Street Journal-based CSR corpus,'' in _Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992_ , 1992. [Online]. Available: https://aclanthology.org/H92-1073
  * [25] N. Zeghidour and D. Grangier, ``Wavesplit: End-to-end speech separation by speaker clustering,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 2840–2849, 2021.
  * [26] W. Ravenscroft, S. Goetze, and T. Hain, ``On time domain conformer models for monaural speech separation in noisy reverberant acoustic environments,'' in _ASRU 2023_ , 2023, pp. 1–7.
  * [27] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 12 449–12 460.
  * [28] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, ``Robust speech recognition via large-scale weak supervision,'' in _ICML 2023_ , Jul. 2023, pp. 28 492–28 518.
  * [29] T. von Neumann, C. Boeddeker, K. Kinoshita, M. Delcroix, and R. Haeb-Umbach, ``On word error rate definitions and their efficient computation for multi-speaker speech recognition systems,'' in _ICASSP 2023_ , Jun. 2023, pp. 1–5.
  * [30] C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, ``A short-time objective intelligibility measure for time-frequency weighted noisy speech,'' in _ICASSP 2010_ , Mar. 2010, pp. 4214–4217.
  * [31] A. Rix, J. Beerends, M. Hollier, and A. Hekstra, ``Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,'' in _ICASSP 2001_ , May 2001, pp. 749–752.
  * [32] T. H. Falk, C. Zheng, and W.-Y. Chan, ``A non-intrusive quality and intelligibility measure of reverberant and dereverberated speech,'' _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 18, no. 7, pp. 1766–1774, 2010.
  * [33] W. Ravenscroft, S. Goetze, and T. Hain, ``Receptive field analysis of temporal convolutional networks for monaural speech dereverberation,'' in _EUSIPCO 2022_ , Aug. 2022, pp. 80–84.

  *[ASR]: automatic speech recognition
  *[PIT]: permutation invariant training
  *[WER]: word error rate
  *[STOI]: short-time objective intelligibility
  *[SOT]: serialized output training
  *[GPIT]: guided permutation invariant training
  *[SISDR]: scale-invariant signal-to-distortion ratio
  *[AE]: ASR encoder
  *[CTC]: connectionist temporal classification
  *[MSE]: mean square error
  *[ATEs]: addtional training epoch
  *[SNR]: signal-to-noise ratio
  *[TSL]: training signal length
  *[SSSR]: self-supervised speech representation
  *[CP-WER]: concatenated minimum-power word error rate
  *[ORC-WER]: optimal reference combination word error rate
  *[PESQ]: perceptual evaluation of speech quality
  *[SRMR]: speech-to-reverberation modulation energy ratio