# Open-vocabulary Video Question Answering: A New Benchmark for
Evaluating the Generalizability of Video Question Answering Models

Dohwan Ko Ji Soo Lee Miso Choi
Jaewon Chu Jihwan Park Hyunwoo J. Kim
Department of Computer Science and Engineering, Korea University
{ikodoh, simplewhite9, miso8070, allonsy07, jseven7071,
<EMAIL_ADDRESS>Corresponding author.

###### Abstract

Video Question Answering (VideoQA) is a challenging task that entails complex
multi-modal reasoning. In contrast to multiple-choice VideoQA which aims to
predict the answer given several options, the goal of open-ended VideoQA is to
answer questions without restricting candidate answers. However, the majority
of previous VideoQA models formulate open-ended VideoQA as a classification
task to classify the video-question pairs into a fixed answer set, i.e.,
closed-vocabulary, which contains only frequent answers (e.g., top-1000
answers). This leads the model to be biased toward only frequent answers and
fail to generalize on out-of-vocabulary answers. We hence propose a new
benchmark, Open-vocabulary Video Question Answering (OVQA), to measure the
generalizability of VideoQA models by considering rare and unseen answers. In
addition, in order to improve the model’s generalization power, we introduce a
novel GNN-based soft verbalizer that enhances the prediction on rare and
unseen answers by aggregating the information from their similar words. For
evaluation, we introduce new baselines by modifying the existing (closed-
vocabulary) open-ended VideoQA models and improve their performances by
further taking into account rare and unseen answers. Our ablation studies and
qualitative analyses demonstrate that our GNN-based soft verbalizer further
improves the model performance, especially on rare and unseen answers. We hope
that our benchmark OVQA can serve as a guide for evaluating the
generalizability of VideoQA models and inspire future research. Code is
available at https://github.com/mlvlab/OVQA.

##  1 Introduction

(a)

(b)

(c)

Figure 1: MSRVTT-QA statistics of three answer groups. Illustration of three
different answer groups: the 1000 most frequent answers in the training set
($\sim$ Top-1000), the remaining answers in the training set (Top-1000
$\sim$), and unseen answers which do not exist during training but appear in
the test set (Unseen). (a) shows the proportion of the number of unique
answers in each group. (b) shows the proportion of the number of samples in
each group. (c) shows the distribution of the number of samples for each
sorted answer. Note that the red lines distinguish each group and the y-axis
is an exponential scale.

Video question answering (VideoQA) is a multi-modal understanding task that
requires complex reasoning between two modalities to find the correct answer
given a video-question pair. There are usually two task types in VideoQA,
multiple-choice and open-ended. The multiple-choice VideoQA requires the model
to select the correct answer among several options. On the other hand, in
open-ended VideoQA, the model needs to predict the answer without restricting
candidate vocabulary.

However, most existing VideoQA models [lei2021less, wang2022all, li2020hero,
fu2021violet, zellers2021merlot, le2020hierarchical, yang2022zero] formulate
the open-ended VideoQA task as a classification problem with a fixed set of
answer candidates which frequently appear in the training set, e.g., top-1000.
Therefore, the out-of-vocabulary answers, not used during training, are
automatically regarded as incorrect without any thorough consideration during
evaluation. Fig. 1(a) highlights that the top-1000 answer categories cover
about 17.8% of the answer candidates while they possess about 90.2% of the
total samples overwhelming those of other answer categories in Fig. 1(b). This
suggests that previous models may show seemingly good performance only with
top-k answer candidates, yet they, in fact, fail to generalize to rare and
unseen answers by ignoring the underrepresented out-of-vocabulary answers.
Such problems have been overlooked since these models have been evaluated in
terms of overall performance only. In other words, the conventional benchmark
of open-ended VideoQA does not measure the generalizability and thus leads the
model to neglect the realistic setting of class imbalance and unseen answers.
Therefore, a comprehensive benchmark that handles long-tail distribution with
unseen answers is necessary.

A long-tail distribution with rare and unseen answers requires few-shot and
zero-shot generalization. Recently, prompt-tuning [radford2021learning,
zhou2022learning, chen2022prompt, jia2022visual, zhou2022conditional] with
large-scale pretrained models has drawn attention due to its significant
performance gain on zero-shot and few-shot learning. A line of work
[schick2021exploiting, liu2021gpt, gao2021making, cui2022prototypical,
shin2020autoprompt, schick2020automatically, holtzman2021surface,
hu2022knowledgeable] enables fine-tuning the model in a parameter-efficient
manner by retaining the Masked Language Modeling (MLM) objective leveraged in
the pretraining phase. In other words, the model is asked to fill in [MASK]
tokens for its downstream objectives. Subsequently, the concept of verbalizer
was introduced by [schick2021exploiting] to manually bridge the original label
and its corresponding words to be filled in [MASK], e.g., filling the word
‘great’ in [MASK] to predict the label POSITIVE in sentiment classification.
To reduce the human labor, search-based verbalizers [gao2021making,
schick2020automatically, shin2020autoprompt] have been proposed. Current works
[cui2022prototypical, hambardzumyan2021warp, zhang2021differentiable] adopt
soft verbalizers which consist of learnable tokens to find optimal embeddings
during training. However, verbalizers for unseen answers have been less
explored in the literature.

To this end, we introduce a new benchmark of open-ended VideoQA, named Open-
vocabulary Video Question Answering (OVQA), to define the task under a more
real-world setting with rare and unseen answers. In contrast to previous
approaches which focus only on frequent answers, OVQA requires the model to
predict rare and out-of-vocabulary answers. In OVQA, to address the problem of
bias towards frequent answers, we propose a novel graph neural network
(GNN)-based soft verbalizer to smooth the original answer embeddings by
aggregating the information of similar words from an external knowledge base.
Specifically, the GNN-based soft verbalizer learns how to smooth the original
answers with their neighborhood words in the training phase and is adapted to
the test phase based on the learned smoothing function during training to
enhance the prediction for the unseen answers.

In our experiments, on four benchmark open-ended VideoQA datasets (MSVD-QA,
ActivityNet-QA, TGIF-QA, and MSRVTT-QA), we develop OVQA baseline models with
an additional answer encoder and improve their performances by taking into
account rare and unseen answers as well. Also, our extensive ablation studies
demonstrate that GNN-based soft verbalizer is generally adaptable to various
backbone models and effectively reduces the bias towards frequent answers.

To sum up, our contributions are as follows:

  * •

We propose a new benchmark of open-ended VideoQA, OVQA, to evaluate models’
generalizability under a long-tail distribution, including unseen answers.

  * •

We also present a novel GNN-based soft verbalizer to smooth answers on the
answer graphs augmented with an external knowledge base.

  * •

Our experiments show that baselines are consistently improved by our simple
modification with an additional answer encoder to handle out-of-vocabulary
answers.

  * •

Extensive ablation studies and qualitative analyses demonstrate that GNN-based
soft verbalizer is broadly applicable and alleviates the bias problem toward
frequent answers.

##  2 Open-vocabulary video question answering

In this section, we introduce a new benchmark, Open-vocabulary Video Question
Answering (OVQA), to tackle the problem of a common practice that formulates
open-ended VideoQA as a classification task with fixed answer candidates.

(a) Closed-vocabulary VideoQA (CVQA)

(b) Open-vocabulary VideoQA (OVQA)

Figure 2: Comparison of CVQA and OVQA. (a) The output feature of [CLS] token
is fed to an MLP to calculate the logits over the fixed top-k answer
candidates (closed-vocabulary) thus it fails to select the out-of-vocabulary
answers in the test phase. (b) On the other hand, in our OVQA setting, the
model chooses the answer based on the similarities between the output feature
of [MASK] token and the answer embeddings. Therefore, the model can predict
the answer although the answer is unseen at the training phase.

###  2.1 Open-ended VideoQA

Unlike multiple-choice VideoQA where a model needs to choose one answer among
the given five options, the open-ended VideoQA task aims to predict the answer
without any candidate answers. However, previous works [lei2021less,
wang2022all, li2020hero, fu2021violet, zellers2021merlot, le2020hierarchical,
yang2022zero] formulate open-ended VideoQA as a classification problem with a
predefined answer set containing fixed candidate answers. We call this setting
Closed-vocabulary Video Question Answering (CVQA) for the rest of our paper.
Usually, in CVQA, they construct an answer vocabulary based on the frequencies
of answers in the training set, e.g., top-1000 answers. As a result, the out-
of-vocabulary answers not used for training will be considered incorrect
during evaluation. In other words, previous models learn to predict only the
top-k answers that frequently appear in the training set and ignore rare or
unseen answers. This leads the model to be biased toward frequent answers and
fail to generalize on rare and unseen answers, i.e., they memorize the answers
rather than generalize.

We first categorize all the answers from four benchmark datasets (MSVD-QA,
ActivityNet-QA, TGIF-QA, and MSRVTT-QA) based on how many $\langle$video,
question, answer$\rangle$ triplets from the training set they appear in:
unseen (0 times), rare (1 $\sim$ 10), common (11 $\sim$ 100), and base (101
$\sim$). The unseen answers are only present in the test set while the answers
of other categories are seen in the training set but may or may not appear in
the test set. Tab. 1 shows the number of unique answers for each category. For
an example of MSRVTT-QA, in CVQA, top-1000 answers only include base and
common answers. Therefore, we propose a new benchmark of open-ended VideoQA to
provide an opportunity to consider the rare and even unseen answers.

MSVD-QA MSRVTT-QA TGIF-QA ActivityNet-QA Base (101 $\sim$) 41 205 38 26 Common
(11 $\sim$ 100) 333 937 210 275 Rare (1 $\sim$ 10) 1,478 2,858 1,292 1,353
Unseen (0) 391 1,632 206 1,378 Total 2,243 5,632 1,746 3,032

Table 1: Answer statistics. We report the number of answers for each category:
base, common, rare, and unseen.

###  2.2 Task definition

We here introduce a new benchmark, Open-vocabulary Video Question Answering
(OVQA), which considers not only the frequent answers but also the rare or
unseen answers. Prior studies in CVQA have calculated logits with an MLP on
video-question multi-modal features for each class label that corresponds to
the individual answer candidate as shown in Fig. 2(a). Nevertheless, they fail
to determine the logit scores of the out-of-vocabulary answers that are unseen
in the training set. To consider all the answer vocabularies in OVQA, we also
introduce new baselines which further encode the answer features and calculate
the similarity between the video-question features and the encoded answer
features. This enables the open-vocabulary setting which is capable of
handling unseen answers as illustrated in Fig. 2(b). As a result, unlike
previous CVQA models memorizing only frequent answers, the goal of OVQA is to
consider all the open-vocabulary answers and evaluate the model performance
and its generalizability without ignoring rare or unseen answers.

Similar to the CVQA evaluation metric, we use the accuracy (%) metric for
OVQA. Yet, we report the total accuracy as well as the accuracy for each
answer category (base, common, rare, and unseen). We also introduce a mean
accuracy (mAcc), averaging the accuracy for each unique answer, to assess the
generalizability of the model.

###  2.3 Comparison with other benchmarks

There have been several attempts to evaluate the visual question answering
models under out-of-distribution (OOD) settings since a number of studies have
revealed that most existing models rely extremely on dataset bias to answer
questions [agrawal2018don, kervadec2021roses, niu2021counterfactual,
ramakrishnan2018overcoming, cadene2019rubi]. For example, in Visual Question
Answering, [agrawal2018don] proposed VQA-CP v2, a new split of VQA v2
[goyal2017making], by changing the answer distribution for each question type
between train and test splits, and pointed out that previous models are
vulnerable to such distribution shifts. Also, GQA-OOD [kervadec2021roses] re-
organized GQA dataset [hudson2019gqa] and introduced a new benchmark with more
comprehensive evaluation metrics (e.g., acc-tail and acc-head). However, these
benchmarks did not investigate the unseen answers, which cannot assess the
models’ zero-shot adaptability. In Video Question Answering, NExT-QA
[xiao2021next] introduced open-form video question answering which requires
the model to generate the answer, i.e., a generation problem, without fixed
answer candidates.

In contrast to previous efforts, our OVQA aims to assess the models’
generalizability under a long-tail distribution including out-of-vocabulary
answers, i.e., few-shot and zero-shot adaptability. The term ‘open-vocabulary’
means that a model is required to predict answers that are unseen during
training by comparing the similarity between the video-question feature and
the answer feature. With a sufficiently large number of unseen vocabulary, we
define Open-vocabulary VideoQA.

##  3 GNN-based soft verbalizer

By adopting an additional answer encoder to extract answer embeddings to
enable OVQA, it is worth designing a way to fine-tune the answer embeddings.
To achieve this, we propose a novel GNN-based soft verbalizer. The goal of our
framework is learning to smooth the original answer candidates with their
similar words augmented by an external knowledge base (e.g., GloVe
[pennington2014glove] and ConceptNet [speer2017conceptnet]). Thus it helps the
model enhance the prediction of rare or unseen answers and improves its
generalizability by aggregating information from their neighborhoods. The
overall architecture is illustrated in Fig. 3. We first briefly summarize the
basic concepts of the verbalizer and GNNs, and then delineate our framework.

###  3.1 Preliminaries

Verbalizer. Large-scale foundation models like BERT [devlin2018bert], CLIP
[radford2021learning], and GPT [brown2020language] have shown remarkable
performance on various domains and tasks, and thus ways to fine-tune those
effectively and efficiently have also gained attention. For example, when
fine-tuning on sentiment classification, a common practice is to predict the
label (POSITIVE or NEGATIVE) with a task-specific classification head (usually
MLP) on [CLS] token of a given sentence. Nonetheless, this scheme does not
fully leverage the pretraining objective, i.e., MLM, and its pretrained layer.
It discards the MLM head and newly adopts the classification head, which would
be trained from scratch with a classification loss, on top of [CLS] token.

To effectively utilize the pretrained MLM head, [schick2021exploiting]
reformulated an input sentence into a cloze form and implemented prediction by
filling in the [MASK] token. In this literature, the mapping from the label
space (POSITIVE or NEGATIVE) to the vocabulary (‘great’ or ‘terrible’) to be
filled into the [MASK] token is called the verbalizer. Recent studies
[hu2022knowledgeable, wang2022automatic] about the verbalizer have proposed
one-to-many mapping with similar words from the external knowledge base, e.g.,
(POSITIVE $\rightarrow$ ‘great’, ‘perfect’, ‘fun’, and ‘brilliant’) and
(NEGATIVE $\rightarrow$ ‘terrible’, ‘awful’, ‘disappointing’, and ‘not’).
Also, to deal with the limitations of such hard verbalizers that use discrete
label words, [cui2022prototypical, hambardzumyan2021warp,
zhang2021differentiable] introduced soft verbalizers by adopting learnable
label embeddings.

Remarks. Unlike prompt-tuning which maps the word to embedding by appending
several learnable tokens at the input-level, the soft verbalizer maps the word
feature to word feature in the embedding space, while the hard verbalizer maps
the word to word in the word-level.

Graph Neural Networks (GNNs). A graph is denoted as
$\mathcal{G}=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}$ is a set of nodes
and $\mathcal{E}$ is a set of edges. Each node $i\in\mathcal{V}$ has a node
feature vector $v_{i}\in\mathbb{R}^{D}$. A set of neighborhoods of the $i$-th
node including itself is defined as
$\mathcal{N}_{i}=\\{i\\}\cup\\{j\in\mathcal{V}|(i,j)\in\mathcal{E}\\}$. The
majority of current GNNs [gilmer2017neural, kipf2016semi] use message-passing
frameworks to train graph-structured data as:

$\mathbf{h}_{i}^{(l)}=\sigma\left(\mathbf{W}^{(l)}\cdot\text{AGGREGATE}\left(\mathbf{h}_{j}^{(l-1)}:j\in\mathcal{N}_{i}\right)\right),$
(1)

where $\mathbf{h}_{i}^{(l)}$ is a hidden representation of the $i$-th node on
the $l$-th layer, $\mathbf{h}_{i}^{(0)}$ is an input feature of the $i$-th
node, and $\mathbf{W}^{(l)}$ is a learnable weight matrix on the $l$-th layer.
AGGREGATE is an aggregation function defined differently by the model, and
$\sigma$ is a non-linear activation function. $L$-layer GNN is conducted by
propagating the input features through Eq. (1) $L$ times.

Latest studies [li2018deeper, wu2019simplifying] have shown that most existing
GNNs such as GCN [kipf2016semi] and GAT [velivckovic2017graph] effectively
learn to propagate information and capture meaningful patterns in the graph
when the connected nodes have similar characteristics. We hence adopt GNN to
learn how to smooth the original answer with its similar words and apply it to
the test vocabulary answers to adequately handle the rare or unseen answers by
smoothing them with their neighborhoods.

Figure 3: Overall architecture. (a) Video-question encoding: a video-question
pair is first encoded through a backbone architecture and the output feature
of [MASK] token, $\mathbf{m}\in\mathbb{R}^{D}$, is extracted. (b) GNN-based
soft verbalizer: an answer graph is constructed with both original answers and
their augmented words from an external knowledge base, and GNN aggregates
their information. (c) Similarity calculation: we finally calculate the
similarity (denoted as $\otimes$) between smoothed answer embeddings
$\mathbf{H}_{\text{train}}$ (or $\mathbf{H}_{\text{test}}$) and [MASK] token
output feature $\mathbf{m}$.

###  3.2 Overall architecture

Our model is based on FrozenBiLM [yang2022zero] consisting of three
components: a video encoder, a text encoder, and a cross-modal encoder.

Video encoder. Each input video is divided into $T$ frames and each frame is
fed into CLIP ViT-L/14 [radford2021learning, dosovitskiy2020image] to extract
the features denoted as
$\mathbf{X}=\\{\mathbf{x}_{t}\\}_{t=1}^{T}\in\mathbb{R}^{T\times D}$, where
$D$ is a feature dimension.

Input prompt and text tokenizer. The input text prompt for OVQA is formulated
as a cloze form [schick2021exploiting, taylor1953cloze], i.e., the model is
expected to fill in a mask token in the input prompt. [CLS] and [SEP] tokens
are inserted at the beginning and the end of each sequence. Textual subtitles
attained from automatic speech recognition (ASR) can be optionally appended.
The prompt is as follows: “[CLS] Question: <Question>? Answer: [MASK].
Subtitles: <Subtitles> [SEP]”. Each prompt sequence is tokenized to
$\mathbf{Y}=\\{\mathbf{y}_{n}\\}_{n=1}^{N}\in\mathbb{R}^{N\times D}$ by
DeBERTa [he2020deberta] tokenizer, where $N$ is the number of tokens.

Cross-modal encoder. The visual feature $\mathbf{X}$ and text feature
$\mathbf{Y}$ are forwarded to the cross-modal encoder. The model is optimized
by the masked language modeling (MLM) objective and we especially denote the
output feature of [MASK] token as $\mathbf{m}\in\mathbb{R}^{D}$. Then, our
model compares the similarity between $\mathbf{m}$ and the answer features
also encoded by DeBERTa tokenizer. Fig. 3 illustrates our overall
architecture.

In contrast to CVQA whose train and test vocabulary sets are consistent with
each other (top-k frequent answers), we consider two different vocabulary sets
$\mathcal{V}_{\text{train}}$ and $\mathcal{V}_{\text{test}}$ respectively
where the former covers the entire answers from the training set and the
latter contains the answers even unseen at the training phase. We further
develop several OVQA baselines by modifying a classfication head. In details,
instead of using MLP as the classification head, we replace it with the
similarity calculation between video-question multi-modal features and answer
embeddings.

###  3.3 Answer graph construction

We first construct an answer graph from an external knowledge base to be used
for a GNN-based soft verbalizer. We denote a neighborhood construction
function of the original answer $a$ as $n(a)$. Note that $n(a)$ may be
considered as an one-to-many mapping verbalizer introduced in Sec. 3.1. $n(a)$
is composed of the nearest neighborhood words of $a$ from GloVe
[pennington2014glove]. Then, we augment them into one node set as:

$\begin{split}&\mathcal{V}_{\text{train}}^{(k)}=\\{j|j\in
n(i)\>\>\text{and}\>\>i\in\mathcal{V}_{\text{train}}^{(k-1)}\\}\cup\mathcal{V}_{\text{train}}^{(k-1)}\\\
&\mathcal{V}_{\text{test}}^{(k)}=\\{j|j\in
n(i)\>\>\text{and}\>\>i\in\mathcal{V}_{\text{test}}^{(k-1)}\\}\cup\mathcal{V}_{\text{test}}^{(k-1)},\end{split}$
(2)

where $\mathcal{V}_{\text{train}}^{(0)}=\mathcal{V}_{\text{train}}$ and
$\mathcal{V}_{\text{test}}^{(0)}=\mathcal{V}_{\text{test}}$, i.e., original
train and test vocabulary sets. Also, the set of edges is defined as:

$\begin{split}&\mathcal{E}_{\text{train}}^{(k)}=\\{(j,i)|j\in
n(i)\>\>\text{and}\>\>i\in\mathcal{V}_{\text{train}}^{(k-1)}\\}\\\
&\mathcal{E}_{\text{test}}^{(k)}=\\{(j,i)|j\in
n(i)\>\>\text{and}\>\>i\in\mathcal{V}_{\text{test}}^{(k-1)}\\}.\end{split}$
(3)

Then, the answer graph is as follows:

$\mathcal{G}_{\text{train}}^{(K)}=(\mathcal{V}_{\text{train}}^{(K)},\mathcal{E}_{\text{train}}^{(K)}),\>\>\>\mathcal{G}_{\text{test}}^{(K)}=(\mathcal{V}_{\text{test}}^{(K)},\mathcal{E}_{\text{test}}^{(K)}).$
(4)

Note that $\mathcal{G}_{\text{train}}^{(K)}$ and
$\mathcal{G}_{\text{test}}^{(K)}$ take into account $K$-hop neighborhoods for
each answer, and we use $K=2$ to consider up to 2-hop neighborhoods. Also, the
edges directly connected in-between the original answers are dropped.