#  TARexp: A Python Framework for
Technology-Assisted Review Experiments

Eugene Yang  HLTCOE, Johns Hopkins UniversityBaltimoreMDUSA
<EMAIL_ADDRESS>and  David D. Lewis  Redgrave DataChantillyVAUSA
<EMAIL_ADDRESS>
(2022)

###### Abstract.

Technology-assisted review (TAR) is an important industrial application of
information retrieval (IR) and machine learning (ML). While a small TAR
research community exists, the complexity of TAR software and workflows is a
major barrier to entry. Drawing on past open source TAR efforts, as well as
design patterns from the IR and ML open source software, we present an open
source Python framework for conducting experiments on TAR algorithms. Key
characteristics of this framework are declarative representations of workflows
and experiment plans, the ability for components to play variable numbers of
workflow roles, and state maintenance and restart capabilities. Users can draw
on reference implementations of standard TAR algorithms while incorporating
novel components to explore their research interests. The framework is
available at https://github.com/eugene-yang/tarexp.

reproducible experiments, technology-assisted review, eDiscovery, systematic
review, opensource

††journalyear: 2022††copyright: acmlicensed††conference: Proceedings of the
45th International ACM SIGIR Conference on Research and Development in
Information Retrieval; July 11–15, 2022; Madrid, Spain††booktitle: Proceedings
of the 45th International ACM SIGIR Conference on Research and Development in
Information Retrieval (SIGIR ’22), July 11–15, 2022, Madrid, Spain††price:
15.00††doi: 10.1145/3477495.3531663††isbn: 978-1-4503-8732-3/22/07††ccs:
Information systems Information retrieval

⬇

component.PerfectLabeler(),

component.RelevanceSampler(),

component.FixedRoundStoppingRule(max_round=20))()

workflow = tarexp.OnePhaseTARWorkflow(dataset, setting, seed_doc=[1023],
batch_size=200, random_seed=123)

recording_metrics = [ir_measures.RPrec,
tarexp.OptimisticCost(target_recall=0.8, cost_structure=(25,5,5,1))]

for ledger in workflow:

print(”Round {}: found {} positives in total”.format(ledger.n_rounds,
ledger.n_pos_annotated))

print(”metric:”, workflow.getMetrics(recording_metrics))

Figure 1. Sample Python Snippet for Running One-Phase TAR Workflow. Please
refer to the online Google Colab Notebook demo page for a full working
example.

##  1\. Introduction

Technology-assisted review (TAR) is the use of information retrieval (IR) and
machine learning (ML) technologies to reduce the cost and increase the
effectiveness of manual review of large text collections. Application areas
include legal discovery (Baron et al., 2016), systematic literature review in
medicine (Wallace et al., 2010), construction of evaluation collections
(Lawrie et al., 2022), and responses to sunshine law requests (Mcdonald et
al., 2020).

Workshops such as DESI111users.umiacs.umd.edu/~oard/desi7,
SIRE222http://users.umiacs.umd.edu/~oard/sire11/,
LegalAIIA333https://sites.google.com/view/legalaiia-2021/home, and
ALTARS444http://altars2022.dei.unipd.it/ have brought these applications to
the awareness of the research community. Shared evaluation efforts such as the
TREC Legal Track (Baron et al., 2006; Tomlinson et al., 2007; Oard et al.,
2008; Hedin et al., 2009; Cormack et al., 2010), the TREC Total Recall Track
(Roegiest and Cormack, 2015; Grossman et al., 2016), and the CLEF eHealth
Technology-Assisted Review Tasks (Kanoulas et al., 2017, 2018, 2019) have made
data sets and formalized evaluation approaches available to researchers.

However, the inherent complexity of TAR tasks, even when abstracted to
research data sets, still imposes a substantial barrier to research. Many
research questions in TAR focus on optimizing interactions between cost and
effectiveness during an evolving review process. Testing a new TAR approach
requires exploring variations of, and capturing rich data from, iterative
active learning processes and multiple review stages. Further, the dynamics of
these algorithms varies strongly not only across tasks (whether real or
simulated) but even across choices of starting conditions and random seeds.
Expensive large scale experiments are therefore necessary to derive meaningful
generalizations. Finally, sample-based effectiveness estimation is itself an
object of study in TAR (driven largely by the needs of legal applications)
(Lewis et al., 2021). This raises the stakes for consistency and replicability
of evaluation.

TARexp is an open-source Python framework intended to reduce barriers to entry
for TAR research. We draw on design patterns from operational TAR software and
past open source TAR efforts, as well as those from the broader machine
learning and information retrieval open source ecosystem, including libact
(Yang et al., 2017c) , pyTorch (Paszke et al., 2019), pyTerrier (Macdonald et
al., 2021), ir-datasets (MacAvaney et al., 2021), and ir-measures (MacAvaney
et al., 2022). TARexp allows configuration and dispatching of experimental
runs with support for parallel processing, resumption and extension of runs,
and reproducibility. It incorporates reference implementations of key
components such as supervised learning, active learning, stopping rules,
sample-based estimation of effectiveness, and TAR-specific cost visualizations
(Yang et al., 2021c). Interfaces in the form of abstract classes for these
components aid researchers in implementing and studying their own approaches.
The framework is also compatible with Jupyter Notebooks for running
exploratory experiments and visualizing results. The framework is available at
https://github.com/eugene-yang/tarexp, and a live demo is available on Google
Colab555https://colab.research.google.com/github/eugene-
yang/tarexp/blob/main/examples/exp-demo.ipynb.

##  2\. Background

Shared software resources in IR research date back to the 1960s (Salton and
Lesk, 1965). Numerous open source research-oriented retrieval libraries are in
use, including Anserini (Yang et al., 2017a), pyTerrier (Macdonald et al.,
2021), Indri (Strohman et al., 2005), Galago (Cartright et al., 2012), and
Patapsco (Costello et al., 2022). An even larger ecosystem supports research
on ML and natural language processing (NLP), including core IR tasks such as
text categorization, summarization, question answering, and recommendation
(Amiri et al., 2019; Zacharias et al., 2018; Kalyanathaya et al., 2019).

However, the needs of TAR research are most similar to those in active
learning, an area that has seen less open source activity. We have drawn on
ideas from libact (Yang et al., 2017c) 666https://github.com/ntucllab/libact ,
the most notable open source active learning framework. The structure of
TARexp is inspired by its modularization of learning algorithms, sampling
strategies, and labelers, but we take a more flexible approach. In particular,
we allow components that perform multiple roles in a workflow to support, for
example, stopping rules that draw on training sample-based estimates.

Providing open source code with published research papers is increasingly
common, including in several recent TAR studies (Yang et al., 2021b; Li and
Kanoulas, 2020; Yang et al., 2019). These reference implementations are useful
in replicating particular studies, but have been less useful at supporting TAR
research more broadly.

There have been a few open source efforts focused on broader support for TAR
experimentation. The TAR Evaluation
Toolkit777https://cormack.uwaterloo.ca/tar-toolkit/ enables simulation of a
fixed set of active learning workflows on a labeled version of the Enron
collection, and was used in several research studies by the tool’s authors
(Cormack and Grossman, 2016a; Zhang et al., 2016; Cormack and Grossman,
2016b). The Baseline Model Implementation (Cormack and Grossman, 2015) (BMI)
is a successor to the TAR Evaluation Toolkit that is wrapped with a VirtualBox
virtual machine that provides an interface for users to run TAR interactively.
It was used in the TREC Total Recall Tracks as baseline systems(Roegiest and
Cormack, 2015; Grossman et al., 2016). HiCAL888https://hical.github.io embeds
BMI in a Django-based framework along with the Indri search engine (Strohman
et al., 2005) and an interface for interactive assessment (Abualsaud et al.,
2018). Components communication through HTTP APIs and so HiCAL has more
potential for modification than the previous efforts. It was been used in
annotation of the HC4 collections (Lawrie et al., 2022).

FreeDiscovery999https://tryfreediscovery.com/ wraps a REST API around selected
scikit-learn IR and ML learning functionality, as well as providing new
algorithms for eDiscovery tasks such as email threading and duplication
detection. It does not incorporate support for active learning experiments
itself, but has been used as a component in active learning experiments (Yang
et al., 2017b).

Numerous open source or web-based tools are available for carrying out
systematic reviews101010http://systematicreviewtools.com/, but most provide
little support for algorithmic experimentation. One exception is
ASReview111111https://github.com/asreview/asreview , an open source tool
implemented in Python and Javascript (van de Schoot et al., 2021). It includes
a simulation mode that allows running experiments on labeled data sets, and
supports user configuration of supervised learning, active learning, and
feature extraction methods121212https://asreview.nl/blog/simulation-mode-
class-101.

##  3\. Structure of TARexp

Figure 2. Structure overview of TARexp.

A major advance of TARexp over previous TAR research software is the ability
to declaratively specify TAR workflows. Users can create components defined
using a standard interface and combine them with TARexp components in
workflows of their design. This includes incorporating different simulations
of human-in-the-loop reviewing, or even embedding in systems using actual
human review (though we have not done the latter).

Execution of declaratively specified review workflows is supported by a
workflow object (Sec 3.1). The changes in the labeling state of the document
collection are recorded in the the ledger (Sec 3.2). During the iterative
process, the workflow reaches out to a set of workflow components (Sec 3.3),
each of which can play one or more roles in the workflow. Finally, an
experiment (Sec 3.5) object defines a set of experiments and dispatches them
sequentially or in parallel. Figure 1 is a code snippet that demonstrates how
each element combines to form a working TAR process, Figure 2 is a general
overview diagram of TARexp.

###  3.1. Workflow

An object of class workflow executes the user’s declarative specification of a
TAR workflow. In doing so, it reaches out to components for services specified
in the declarative specification such as creating training batches, scoring
and ranking the collection, and testing for stopping conditions.

After an optional initial seed round where the user can specify a starting set
of labeled training data, the workflow is executed as a sequence of training
rounds. Each round consists of selecting a batch of training documents (using
a sampler object), looking up labels for those documents (using the labeler
object), training a model and scoring and ranking the collection documents
(using the ranker object).

TARexp supports specifications of both one and two-phase TAR workflows, as
described in Yang et al. (2021c). One-phase workflows
(tarexp.OnePhaseTARWorkflow in code) can be run for a fixed number of training
rounds, or until all documents have been reviewed. Two-phase reviews also use
a stopping rule to determine when to end training, but then follow that by
ranking the collection with the final trained model and reviewing to a
statistically determined cutoff.

The workflow object maintains only enough state in-memory to work through a
training round including the seed for random number generators. Besides the
optionally written document scores, the rest of the state of the workflow is
recorded in the ledger, which is written to secondary storage at user-
configurable intervals. This allows easy restarting of crashed runs with
minimal redundant work.

The workflow object is implemented as a Python iterator, allowing procedures
defined outside the workflow to execute at each round. The iterator yields a
frozen ledger (see next Section). The user can define a custom per-round
evaluation process or record information for later analysis.

###  3.2. Ledger

Any aspect of the history of a batch-based workflow can, if necessary, be
reproduced from a record of which documents were labeled on which training
rounds (including any initial seed round). The ledger object records this
state in memory, and writes it to disk at user-specified intervals to enable
restarts.

The persisted ledger for a complete run can be used to execute TARexp in
frozen mode where no batch selection, training, or scoring is done. Frozen
mode supports efficient testing of new components that do not change training
or scoring, e.g. non-interventional stopping rules (Lewis et al., 2021),
effectiveness estimation methods, etc. Evaluating stopping rules for two-phase
reviews also requires persisting scores of all documents at the end of each
training round, an option the user can specify.

###  3.3. Components

Figure 3. Cost dynamic graphs on topic GPRO in RCV1-v2 with different cost
structured targeting 80% recall produced by the helper function. The height of
the color blocks indicate the cost on each part spent at the respective round.
The faded section indicates the rounds that pass the optimal stopping point
and the grey vertical dashed line indicates the round where the one phase TAR
workflow would reach the recall target.

TARexp implements algorithms via components. A component is an object that is
declared to serve one or more roles in a workflow, e.g. the stopping rule, the
training batch sampler, the ranker, or the labeler. Components communicate
only through the workflow. The association of components with multiple roles
is important when implementing algorithms where, for instance, the stopping
rule interacts tightly with a particular batch selection method (e.g. AutoStop
(Li and Kanoulas, 2020)). The current release of TARexp defines the interface
of multi-role components, but release of the particular multi-role components
we have implemented is waiting on a paper under review (Yang et al., 2021a).

TARexp supports classification models implemented in Scikit-learn through
component.SklearnRanker wrapper. However, any supervised learning model that
can produce a score for each document in the collection can be integrated into
TARexp. We have tested an initial implementation of Transformer-based models
for TAR Yang et al. (2022), but have not yet integrated this code into the
released version of TARexp.

TARexp provides reference implementations of a variety of TAR-specific
algorithms, to aid reproducibility and reduce experimenter work. For instance,
uncertainty sampling (Lewis and Gale, 1994), relevance feedback (Rocchio,
1971), and simple random sampling batch selection algorithms are provided.

Stopping rules are a particular focus of TAR research. TARexp provides
implementation of the Knee and Budget Rules (Cormack and Grossman, 2016a,
2015), a configurable bath precision rule, the 2399 Rule (Roitblat, 2007; Yang
et al., 2021b), fixed numbers of training rounds (Wallace et al., 2010), the
Quant and QuantCI Rules (Yang et al., 2021b), and others.

A Labeler object simulates human review. For most TAR experiments, we assume
we simply look up the gold label of each document using
component.PerfectLabeler. Random errors can be introduced using
component.SuccessProbLabeler.

###  3.4. Evaluation

Consistent implementation of effectiveness metrics, including tricky issues
like tiebreaking is critical to TAR experiments. This is true both for
evaluation, and because stopping rules may incorporate effectiveness estimates
based on small samples. We provide all metrics from the open source package
ir-measures131313https://ir-measur.es/en/latest/ through the
tarexp.Workflow.getMetrics method. Metrics are computed on both the full
collection and unreviewed documents to support both finite population and
generalization perspectives (Yang et al., 2019).

In addition to standard IR metrics, TARexp implements Optimis-
ticCost to support the idealized end-to-end cost analysis for TAR proposed in
Yang et al. (2021c). Such analysis requires specifying a target recall and a
cost structure associated with the TAR process (Line 8 in Figure 1).
TARexpalso provides helper functions for plotting cost dynamics graphs
(Section 4.1).

###  3.5. Experiments

TAR inherits both the large topic-to-topic variability of IR tasks, and the
strong dependence on initial conditions and random seeds of active learning
processes. Multiple collections, topics, and runs are necessary to reliably
demonstrate that one approach dominates another. Inspired by pyTerrier
(Macdonald et al., 2021), TARexp supports of a declarative representation for
experiments, as shown in this example:

⬇

’./experiment_output/’,

random_seed=123, max_round_exec=50,

metrics=[ir_measures.RPrec, ir_measures.P@10],

tasks=tarexp.TaskFeeder(

dataset,

labels[[’GPRO’, ’GOBIT’]]),

components=setting,

workflow=tarexp.OnePhaseTARWorkflow,

batch_size=[200, 50])

results = exp.run(n_processes=2,

n_nodes=2, node_id=0,

resume=True, dump_frequency=5)

where an example of defining the variable setting is seen in Figure 1. The
object tarexp.TaskFeeder creates a stream of tasks, each of which corresponds
to a different labeling of the same document collection. In the example, GPRO
and GOBIT are categories in RCV1 (Lewis et al., 2004). components parameter
specify the kind of components for experimenting, which can be either a single
one, e.g., the one in Figure 1, or a list of such. Given a task,
tarexp.TARExperiment then creates one workflow object for each combination of
component, hyperparameter, and random seed choices. The example above will
yield four workflows since we have two tasks and two specified batch sizes,
and only one alternative was specified for each component.

We support spawning multiple processes both across machines on a network, and
in multiple threads on appropriate hardware. The method .run dispatches the
TAR tasks with runtime settings. In the above example, experiments will run on
the first of the two machines with two processes on each, resulting in all
four tasks being run simultaneously. The .run method returns the per-round
metric values of all the experiment tasks running on the node.

##  4\. Helper Functions and Integration

Besides the core functionality of executing experiments, TARexp provides
several tools to aid in analyzing results. Most are included in our Google
Colab notebook.

###  4.1. Experiment Analysis and Visualization

The experiment results that tarexp.TARExperiment returns are in generic Python
dictionaries. Through createDFfromResults, the results are transformed into
Pandas (Wes McKinney, 2010) DataFrames for further analysis. The resulting
DataFrame contains a multi-level index of the experimental parameters and
multi-level columns containing the values of effectiveness and cost metrics.

We also provide visualization tools to produce cost dynamics graphs, such as
Figure 3, through both Python and command-line interfaces. The following is an
example command for creating a graph with two runs and three cost structures.

⬇

–runs GPRO=location/to/GPRO \

GOBIT=location/to/GOBIT \

–cost_structures 1-1-1-1 10-10-10-1 25-5-5-1 \

–y_thousands –with_hatches

###  4.2. Jupyter Notebook and Google Colab

Figure 4. Screenshot of running TAR experiments using TARexp on Google Colab.

Our framework is fully integrated with Jupyter Notebook (Kluyver et al., 2016)
, a browser-based tool for running python interactively. Users can also run
TARexp on Google Colab141414https://colab.research.google.com/, a cloud
version of Jupyter Notebook powered by Google, by installing TARexp through
Pypi151515https://pypi.org/project/tarexp/, the online Python package
distribution repository. Figure 4 is a screenshot of running TAR experiments
on the Google Colab.

##  5\. Conclusion

This paper introduces a new Python framework TARexp for conducting TAR
experiments, an area of IR involving particularly complex software and
workflows. TARexp allows flexible combination of new methods under study with
reference implementations for many commonly used TAR algorithms. Large scale,
deterministically reproducible experiments are supported. We hope that TARexp
will reduce the barriers to entry for researchers to study the many exciting
research problems in TAR, and increase comparability of results.

###### Acknowledgements.

The authors would like to thank Sean MacAvaney and Ophir Frieder for providing
valuable feedback on the design of the framework.

## References

  * (1)
  * Abualsaud et al. (2018) Mustafa Abualsaud, Nimesh Ghelani, Haotian Zhang, Mark D Smucker, Gordon V Cormack, and Maura R Grossman. 2018.  A System for Efficient High-Recall Retrieval. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_. ACM, 1317–1320.
  * Amiri et al. (2019) Saba Amiri, Sara Salimzadeh, and A.S.Z. Belloum. 2019\.  A Survey of Scalable Deep Learning Frameworks. In _2019 15th International Conference on eScience (eScience)_. 650–651.  https://doi.org/10.1109/eScience.2019.00102
  * Baron et al. (2016) J.R. Baron, R.C. Losey, and M.D. Berman. 2016.  _Perspectives on Predictive Coding: And Other Advanced Search Methods for the Legal Practitioner_.  American Bar Association, Section of Litigation.  https://books.google.com/books?id=TdJ2AQAACAAJ
  * Baron et al. (2006) Jason R Baron, David D Lewis, and Douglas W Oard. 2006\.  TREC 2006 Legal Track Overview.. In _TREC_. Citeseer.
  * Cartright et al. (2012) Marc-Allen Cartright, Samuel Huston, and Henry Feild. 2012\.  Galago: A Modular Distributed Processing and Retrieval System.. In _OSIR@ SIGIR_. Citeseer, 25–31.
  * Cormack and Grossman (2015) Gordon V Cormack and Maura R Grossman. 2015.  Autonomy and reliability of continuous active learning for technology-assisted review.  _arXiv preprint arXiv:1504.06868_ (2015).
  * Cormack and Grossman (2016a) Gordon V. Cormack and Maura R. Grossman. 2016a.  Engineering Quality and Reliability in Technology-Assisted Review. In _SIGIR_. ACM Press, Pisa, Italy, 75–84.  https://doi.org/10.1145/2911451.2911510 00024\.
  * Cormack and Grossman (2016b) Gordon V Cormack and Maura R Grossman. 2016b.  Scalability of continuous active learning for reliable high-recall text classification. In _Proceedings of the 25th ACM international on conference on information and knowledge management_. 1039–1048.
  * Cormack et al. (2010) Gordon V Cormack, Maura R Grossman, Bruce Hedin, and Douglas W Oard. 2010.  Overview of the TREC 2010 Legal Track.. In _TREC_.
  * Costello et al. (2022) Cash Costello, Eugene Yang, Dawn Lawrie, and James Mayfield. 2022.  Patapasco: A Python Framework for Cross-Language Information Retrieval Experiments.  (2022).  https://arxiv.org/abs/2201.09996
  * Grossman et al. (2016) Maura R. Grossman, Gordon V. Cormack, and Adam Roegiest. 2016\.  TREC 2016 Total Recall Track Overview.. In _TREC_.
  * Hedin et al. (2009) Bruce Hedin, Stephen Tomlinson, Jason R Baron, and Douglas W Oard. 2009.  _Overview of the TREC 2009 legal track_.  Technical Report. NATIONAL ARCHIVES AND RECORDS ADMINISTRATION COLLEGE PARK MD.
  * Kalyanathaya et al. (2019) Krishna Prakash Kalyanathaya, D Akila, and P Rajesh. 2019\.  Advances in natural language processing–a survey of current research trends, development tools and industry applications.  _International Journal of Recent Technology and Engineering_ 7, 5C (2019), 199–202.
  * Kanoulas et al. (2017) Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2017\.  CLEF 2017 Technologically Assisted Reviews in Empirical Medicine Overview. In _CEUR workshop proceedings_ , Vol. 1866. 1–29.
  * Kanoulas et al. (2018) Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2018\.  CLEF 2018 Technologically Assisted Reviews in Empirical Medicine Overview.  _CEUR Workshop Proceedings_ 2125 (July 2018).  https://strathprints.strath.ac.uk/66446/
  * Kanoulas et al. (2019) Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2019\.  CLEF 2019 technology assisted reviews in empirical medicine overview. In _CEUR workshop proceedings_ , Vol. 2380.
  * Kluyver et al. (2016) Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Safia Abdalla, and Carol Willing. 2016\.  Jupyter Notebooks – a publishing format for reproducible computational workflows. In _Positioning and Power in Academic Publishing: Players, Agents and Agendas_ , F. Loizides and B. Schmidt (Eds.). IOS Press, 87 – 90.
  * Lawrie et al. (2022) Dawn Lawrie, James Mayfield, Douglas W. Oard, and Eugene Yang. 2022.  HC4: A New Suite of Test Collections for Ad Hoc CLIR.  (2022).  https://arxiv.org/abs/2201.09992
  * Lewis and Gale (1994) David D. Lewis and William A Gale. 1994.  A sequential algorithm for training text classifiers. In _SIGIR 1994_. 3–12.
  * Lewis et al. (2021) David D. Lewis, Eugene Yang, and Ophir Frieder. 2021\.  Certifying One-Phase Technology-Assisted Reviews (Under review).  (2021).
  * Lewis et al. (2004) David D. Lewis, Yiming Yang, Tony G. Rose, and Fan Li. 2004\.  RCV1: A New Benchmark Collection for Text Categorization Research.  _JMLR_ 5 (2004), 361–397.
  * Li and Kanoulas (2020) Dan Li and Evangelos Kanoulas. 2020.  When to Stop Reviewing in Technology-Assisted Reviews: Sampling from an Adaptive Distribution to Estimate Residual Relevant Documents.  _ACM Transactions on Information Systems (TOIS)_ 38, 4 (2020), 1–36.
  * MacAvaney et al. (2022) Sean MacAvaney, Craig Macdonald, and Iadh Ounis. 2022\.  Streamlining Evaluation with ir-measures. In _ECIR_.  https://arxiv.org/abs/2111.13466
  * MacAvaney et al. (2021) Sean MacAvaney, Andrew Yates, Sergey Feldman, Doug Downey, Arman Cohan, and Nazli Goharian. 2021\.  Simplified Data Wrangling with ir_datasets. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_.  https://doi.org/10.1145/3404835.3463254
  * Macdonald et al. (2021) Craig Macdonald, Nicola Tonellotto, Sean MacAvaney, and Iadh Ounis. 2021.  PyTerrier: Declarative Experimentation in Python from BM25 to Dense Retrieval. In _30th ACM International Conference on Information and Knowledge Management_.
  * Mcdonald et al. (2020) Graham Mcdonald, Craig Macdonald, and Iadh Ounis. 2020\.  How the Accuracy and Confidence of Sensitivity Classification Affects Digital Sensitivity Review.  _ACM Transactions on Information Systems (TOIS)_ 39, 1 (2020), 1–34.
  * Oard et al. (2008) Douglas W Oard, Bruce Hedin, Stephen Tomlinson, and Jason R Baron. 2008.  _Overview of the TREC 2008 legal track_.  Technical Report. MARYLAND UNIV COLLEGE PARK COLL OF INFORMATION STUDIES.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Advances in Neural Information Processing Systems 32_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024–8035.  http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * Rocchio (1971) J. J. Rocchio. 1971\.  Relevance feedback in information retrieval.  In _The Smart retrieval system - experiments in automatic document processing_ , G. Salton (Ed.). Englewood Cliffs, NJ: Prentice-Hall, 313–323.
  * Roegiest and Cormack (2015) Adam Roegiest and Gordon V. Cormack. 2015.  TREC 2015 Total Recall Track Overview. In _TREC_.
  * Roitblat (2007) Herbert L Roitblat. 2007\.  Search and information retrieval science. In _Sedona Conf. J._ , Vol. 8. HeinOnline, 225.
  * Salton and Lesk (1965) G. Salton and M. E. Lesk. 1965.  The SMART Automatic Document Retrieval Systems—an Illustration.  _Commun. ACM_ 8, 6 (jun 1965), 391–398.  https://doi.org/10.1145/364955.364990
  * Strohman et al. (2005) Trevor Strohman, Donald Metzler, Howard Turtle, and W Bruce Croft. 2005.  Indri: A language model-based search engine for complex queries. In _Proceedings of the international conference on intelligent analysis_ , Vol. 2. Citeseer, 2–6.
  * Tomlinson et al. (2007) Stephen Tomlinson, Douglas W Oard, Jason R Baron, and Paul Thompson. 2007.  Overview of the TREC 2007 Legal Track.. In _TREC_. Citeseer.
  * van de Schoot et al. (2021) Rens van de Schoot, Jonathan de Bruin, Raoul Schram, Parisa Zahedi, Jan de Boer, Felix Weijdema, Bianca Kramer, Martijn Huijts, Maarten Hoogerwerf, Gerbrich Ferdinands, et al. 2021\.  An open source machine learning framework for efficient and transparent systematic reviews.  _Nature Machine Intelligence_ 3, 2 (2021), 125–133.
  * Wallace et al. (2010) Byron C Wallace, Thomas A Trikalinos, Joseph Lau, Carla Brodley, and Christopher H Schmid. 2010\.  Semi-automated screening of biomedical citations for systematic reviews.  _BMC bioinformatics_ 11, 1 (2010), 55.
  * Wes McKinney (2010) Wes McKinney. 2010\.  Data Structures for Statistical Computing in Python. In _Proceedings of the 9th Python in Science Conference_ , Stéfan van der Walt and Jarrod Millman (Eds.). 56 – 61.  https://doi.org/10.25080/Majora-92bf1922-00a
  * Yang et al. (2017b) Eugene Yang, David Grossman, Ophir Frieder, and Roman Yurchak. 2017b.  Effectiveness results for popular e-discovery algorithms. In _Proceedings of the 16th edition of the International Conference on Articial Intelligence and Law_. 261–264.
  * Yang et al. (2019) Eugene Yang, David D Lewis, and Ophir Frieder. 2019\.  A Regularization Approach to Combining Keywords and Training Data in Technology-Assisted Review. In _Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law_. 153–162.
  * Yang et al. (2021a) Eugene Yang, David D. Lewis, and Ophir Frieder. 2021a.  $\beta$-stop: Stopping Fixed-Batch Technology-Assisted Reviews (Under review).  (2021).
  * Yang et al. (2021b) Eugene Yang, David D. Lewis, and Ophir Frieder. 2021b.  Heuristic Stopping Rules For Technology-Assisted Review. In _Proceedings of the 21st ACM Symposium on Document Engineering_.
  * Yang et al. (2021c) Eugene Yang, David D. Lewis, and Ophir Frieder. 2021c.  On Minimizing Cost in Legal Document Review Workflows. In _Proceedings of the 21st ACM Symposium on Document Engineering)_.
  * Yang et al. (2022) Eugene Yang, Sean MacAvaney, David D. Lewis, and Ophir Frieder. 2022.  Goldilocks: Just-Right Tuning of BERT for Technology-Assisted Review.  (2022).  https://arxiv.org/abs/2105.01044
  * Yang et al. (2017a) Peilin Yang, Hui Fang, and Jimmy Lin. 2017a.  Anserini: Enabling the use of Lucene for information retrieval research. In _Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval_. 1253–1256.
  * Yang et al. (2017c) Yao-Yuan Yang, Shao-Chuan Lee, Yu-An Chung, Tung-En Wu, Si-An Chen, and Hsuan-Tien Lin. 2017c.  _libact: Pool-based Active Learning in Python_.  Technical Report. National Taiwan University.  https://github.com/ntucllab/libact available as arXiv preprint https://arxiv.org/abs/1710.00379.
  * Zacharias et al. (2018) Jan Zacharias, Michael Barz, and Daniel Sonntag. 2018\.  A survey on deep learning toolkits and libraries for intelligent user interfaces.  _arXiv preprint arXiv:1803.04818_ (2018).
  * Zhang et al. (2016) Haotian Zhang, Jimmy Lin, Gordon V. Cormack, and Mark D. Smucker. 2016.  Sampling Strategies and Active Learning for Volume Estimation. In _Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Pisa, Italy) _(SIGIR ’16)_. Association for Computing Machinery, New York, NY, USA, 981–984.  https://doi.org/10.1145/2911451.2914685


11institutetext: School of Engineering and Applied Science, Ahmedabad
University, India

# Recent Advances in Video Question Answering: A Review of Datasets and
Methods

Devshree Patel  Ratnam Parikh  Yesha Shastri

###### Abstract

Video Question Answering (VQA) is a recent emerging challenging task in the
field of Computer Vision. Several visual information retrieval techniques like
Video Captioning/Description and Video-guided Machine Translation have
preceded the task of VQA. VQA helps to retrieve temporal and spatial
information from the video scenes and interpret it. In this survey, we review
a number of methods and datasets for the task of VQA. To the best of our
knowledge, no previous survey has been conducted for the VQA task.

###### Keywords:

Video question answering Visual information retrieval Multimodal co-memory
networks Spatio-temporal reasoning Attention networks.

##  1 Introduction

Computer Vision is one of the sub-fields of Artificial Intelligence. It
enables computers to interpret and analyze real-world scenes as humans do.
Some of the computer vision research tasks are segmentation, object detection/
tracking, visual captioning and VQA. Out of them, VQA is a more challenging
task since it requires an understanding of visual data based on the question
asked in natural language. Since, most of the visual information we face in
the real world is either in the form of images or videos, the task of VQA will
help to extract crucial information for building real-life applications.

The task of VQA can be bifurcated into two sub-tasks: multiple-choice QA and
open-ended QA. The latter is more challenging since it does not have a fixed
answer set. Also, an in-depth understanding of the video scene is required to
produce an answer from the pool of all possible answers. Open-ended QA finds
its applicability in the real-world since it helps to understand complex scene
and generate relevant information without the constraints of choices.

The collection of data for image-based QA is relatively easier as compared to
video-based QA. The creation of datasets for VQA is a burdensome task as it
requires huge human efforts for creating QA pairs and accurate annotations by
looking at the visual content. Additionally, video content must have varied
actions, scenes and object interactions in order to build a robust VQA model.
In general, it is a difficult task to derive interpretations from videos since
they comprise of varied length having multiple frames.

The models for VQA have been created by extending the models for image
question answering. Earlier approaches for constructing models of VQA only
considered the spatial information from the videos. However, VQA requires to
find clues accurately on both spatial and temporal dimensions. Proceeding with
the methods, [3] introduced the approach for extracting temporal context
information from the videos. Further, [5] studied the problem of modeling
temporal dynamics with frame-level attention by proposing an attribute
augmented attention network for VQA. In [7], authors exploit information
relating to appearance and motion residing in the video with an optimized
attention mechanism. In contrast to the aforementioned approaches, [16]
presents a novel approach of positional self-attention by replacing RNNs with
self-attention. [12] proposes an architecture based on Multimodal Dual
Attention Memory (MDAM) to extract the latent concepts from video scenes and
captions.Unlike the previous approaches which apply spatio-temporal attention,
[21] improves the approach by introducing location and relations among object
interactions by creating a location-aware graph network. In [27] authors
leverage the task of VQA by using Bidirectional Encoder Representations (BERT)
for capturing the attention of every word from both the directions.

##  2 Datasets

Data acquisition for VQA has been a challenging task due to requirement of
large video corpus with varied content. One of the first datasets for VQA is
YouTube2Text [48] which comprises of 1987 videos and 122,708 descriptions of
those in natural language. Many existing models [5, 9, 13, 21] address the
task of VQA with the help of YouTube2Text [48]. The LSMDC 2016 description
dataset [47] enables multi-sentence description of the videos which is a
unique property from the previous approaches of datasets. It comprises of
$\sim 128$K clips taken as a combination from M-VAD and MPII-MD datasets.
Based on the LSMDC 2016 description dataset [47], a fill-in-the-blank dataset
(MovieFIB) [42] was proposed. This was a simple benchmark dataset consisting
of 300,000 fill-in-the-blank question-answer and video pairs. To answer these
fill-in-the-blank questions a model is expected to have basic understanding of
scenes by detecting activities, objects, people and their interactions. Table
5 shows dataset statistics for existing VQA datasets.

###  2.1 MovieQA

MovieQA [36] is one of the unique benchmark datasets for VQA as it contains
multiple sources of information. It comprises of video clips, subtitles, plots
and scripts. It is aimed to perform automatic story comprehension from both
text and video. The dataset includes 408 movies and 14,944 questions with
semantic variations. [8,12,20,23] attempt to tackle the VQA task with MovieQA
dataset [36]. The original paper [36] obtains an accuracy of $\sim 35\%$ which
is gradually improved and reached to $\sim 45\%$ by [23]. A comparison of
various models using MovieQA dataset is shown in Table 1.

###  2.2 YouTube2TextQA

YouTube2TextQA was one of the first dataset VQA datasets proposed by [48]. The
data is taken from the YouTube2Text dataset. It comprises of 1987 videos and
122,708 automatically generated QA pairs. The dataset can address multiple-
choice questions. Initially, an accuracy of 61.4% was obtained with r-ANL [5]
method and it reached to 83.9% with one of the latest models L-GCN [21].

Table 1: Best Test Accuracies of models using MovieQA and Youtube2TextQA
datasets

Datasets YouTube2TextQA MovieQA Model r-ANL [5] DLAN [9] Fan et al. [13] L-GCN
[21] DEMN [8] MDAM [12] PAMN [20] AMN [23] Accuracy 61.4 36.33 82.5 83.9 30
41.41 42.53 45.31

###  2.3 MSRVTT-QA and MSVD-QA

MSRVTT-QA and MSVD-QA were first used by [7] based on MSRVTT [45] and MSVD
[49] video datasets. MSRVTT-QA consists of $\sim 10$K video clips and $\sim
243$K QA pairs. MSVD-QA is mainly used in video captioning experiments but due
to its large data size it is also used for VQA. It has 1970 video clips and
$\sim 50.5$K QA pairs. A comparison of various models using MSRVTT-QA and
MSVD-QA datasets is shown in Table 2.

Table 2: Accuracies of models using MSRVTT-QA(Left) and MSVD-QA(Right)

MSRVTT-QA MSVD-QA Model What Who How When Where All What Who How When Where
All Xu et al. [7] 26.2 43 80.2 72.5 30 32.5 20.6 47.5 83.5 72.4 53.6 32 Fan et
al. [13] 26.5 43.6 82.4 76 28.6 33 22.4 50.1 73 70.7 42.9 33.7 CAN [22] 26.7
43.7 83.7 75.3 35.2 33.2 21.1 47.9 84.1 74.1 57.1 32.4 Resnet+CDC(AA)[15] 27.3
44.2 86.3 73.9 34.4 34.1 - - - - - - QueST [28] 27.9 45.6 83 75.7 31.6 34.6
24.5 52.9 79.1 72.4 50 36.1 TSN [17] 27.9 46.1 84.1 77.8 37.6 35.4 25 51.3
83.8 78.4 59.1 36.7 Jin et al. [19] 29.5 45 83.2 74.7 42.4 35.4 24.2 49.5 83.8
74.1 53.6 35 Jiang et al. [31] 29.2 45.7 83.5 75.2 34 35.5 23.5 50.4 83 72.4
46.4 34.7 HCRN [29] - - - - - 35.6 - - - - - 36.1 MHMAN [30] 28.7 47.1 85.1
77.1 35.2 35.6 23.3 50.7 84.1 72.4 53.6 34.6 Zhang et al. [15] - - - - - -
21.3 48.3 82.4 70.7 53.6 32.6

###  2.4 VideoQA

VideoQA dataset [4] consists of the questions in free-form natural language as
opposed to previous fill-in-the-blank questions [42]. The dataset is made from
the internet videos having user-curated descriptions. QA pairs are generated
automatically from the descriptions by using a state-of-the-art question
generation method. A collection of 18,100 videos and 175,076 candidate QA
pairs make up this large-scale dataset for VQA.

###  2.5 Pororo-QA

In [8], authors construct a large-scale VQA dataset - PororoQA from cartoon
video series. The dataset is made up of videos having simple but a coherent
story structure as compared to videos in previous datasets. The dataset
includes $\sim 16$K dialogue-scene pairs, $\sim 27$K descriptive sentences and
$\sim 9$K multiple-choice questions. DEMN [8] achieved best accuracy of 68% at
the time of release. A comparison of various models using PororoQA dataset is
shown in Table 3.

###  2.6 TVQA+ and TVQA

TVQA+ dataset [41] is built upon one of the largest VQA datasets - TVQA [40].
TVQA [40] is a dataset built on natural video content, collected from 6 famous
TV series. The videos have rich dynamic interactions and the QA pairs are
formed by the people considering both the videos and the accompanying
dialogues. The provision of temporal annotations by the TVQA dataset [40] is
one of the key properties. However, this dataset does not consider the spatial
annotations which are equally important as temporal. Hence, TVQA+ [41] is a
spatio-temporally grounded VQA dataset made by adding bounding boxes to the
essential regions of the videos from a subset of the TVQA dataset [40]. It
comprises of $\sim 29.4$K multiple-choice questions annotated in both spatial
and temporal domains. A comparison of various models using TVQA dataset is
shown in Table 3.

###  2.7 ActivityNet-QA

[35] introduces ActivityNet-QA, a fully manually annotated large-scale
dataset. It consists of $\sim 58$K QA pairs based on $\sim 5.8$K web videos
taken from the popular ActivityNet dataset [50]. The dataset consists of open-
ended questions. A comparison of various models using ActivityNet-QA dataset
is shown in Table 3.

Table 3: Best Test Accuracies of models using PororoQA, TVQA and ActivityNetQA
datasets

Datasets PororoQA TVQA ActivityNetQA Model DEMN [8] MDAM [12] Yang et al. [27]
PAMN [20] AMN [23] Yang et al. [27] CAN [22] MAR-VQA(a+m) [25] Accuracy 68.0
48.9 53.79 66.77 70.70 73.57 35.4 34.6

###  2.8 TGIF-QA

[44] proposes a TGIF-QA dataset from the earlier (Tumblr GIF) TGIF dataset
which was used for video captioning task. TGIF-QA dataset [44] contains $\sim
165$K QA pairs from $\sim 72$K animated GIFs taken from the TGIF dataset. The
TGIF dataset was ideal for VQA task as it comprised of attractive animated
GIFs in a concise format for the purpose of story telling. A comparison of
various models using TGIF-QA dataset is shown in Table 4 (Lower the count
value, better is the model performance).

Table 4: Models using TGIF-QA dataset

Model Action Trans Frame Count MAR-VQA(a+m) [25] 71.1 71.9 56.6 5.35 PSAC [16]
70.4 76.9 55.7 4.27 STA(R) [18] 72.3 79 56.6 4.25 Jin et al. [19] 72.7 80.9
57.1 4.17 Co-Mem [10] 68.2 74.3 51.5 4.1 Jiang et al. [31] 75.4 81 55.1 4.06
Fan et al. [13] 73.9 77.8 53.8 4.02 L-GCN [21] 74.3 81.1 56.3 3.95 HCRN [29]
75 81.4 55.9 3.82

###  2.9 LifeQA

The LifeQA dataset from [33] considers the real life scenarios in its video
clips unlike the movies or TV shows which contain the edited content. In real-
life QA systems, this dataset will be useful because of its relevance to day-
to-day lives. It comprises of 275 videos and 2328 multiple-choice questions.

###  2.10 DramaQA

[43] proposes DramaQA dataset which focuses upon character-centred
representations with richly annotated 217,308 images taken from the TV drama
”Another Miss Oh”. The annotations consider the aspects of behaviour and
emotions of the characters. In total, the dataset consists of $\sim 23$K video
clips and $\sim 16$K QA pairs.

###  2.11 Social-IQ

Social-IQ dataset [38] also referred as Social Intelligence Queries dataset
targets to analyze the unscripted social situations which comprise of natural
interactions. The dataset is prepared extensively from the YouTube videos as
they are rich in dynamic social situations. This dataset comprising of 1,250
social videos, 7,500 questions and a total of 52,500 answers including 30,000
correct and 22,500 incorrect answers is created with an aim to benchmark
social intelligence in AI systems.

###  2.12 MarioQA

The main motivation behind creating MarioQA dataset [37] was to understand
temporal relationships between video scenes to solve VQA problems. Before
MarioQA [37], all the datasets required excessive reasoning through single
frame inferences. The dataset was synthesized from super mario gameplay
videos. It consists of $\sim 92$K QA pairs from video sets as large as 13
hours.

###  2.13 EgoVQA

The EgoVQA dataset proposed by [34] presents a novel perspective of
considering first-person videos as most of the previous VQA datasets [MSRVTT-
QA [7], MSVD-QA [7], Youtube2TextQA [48]] focus on third-person video
datasets. Answering questions about first-person videos is a unique and
challenging task because the normal action recognition and localization
techniques based on the keywords in questions will not be effective here. To
address the same, 600 question-answer pairs with visual contents are taken
across 5,000 frames from 16 ﬁrst-person videos.

###  2.14 Tutorial-VQA

Tutorial-VQA [32] was prepared to address the need for a dataset comprising of
non-factoid and multi-step answers. The technique for data acquisition was
same as MovieQA [36] and VideoQA [4] as it was build from movie-scripts and
news transcripts. The answers of MovieQA [36] had a shorter span than the
answers collected in Tutorial-VQA [32]. In VideoQA dataset [4], questions are
based on single entity in contrast to instructional nature of Tutorial-VQA
[32]. It consists of 76 videos and 6195 QA pairs.

###  2.15 KnowIT-VQA

KnowIT VQA (knowledge informed temporal VQA) dataset [39] tries to resolve the
limited reasoning capabilities of previous datasets by incorporating external
knowledge. External knowledge will help reasoning beyond the visual and
textual content present in the videos. The collected dataset comprises of
videos annotated with knowledge-based questions and answers from the popular
TV show - The Big Bang Theory. There are a total of $\sim 12$K video clips and
$\sim 24$K QA pairs, making it one of the largest knowledge-based human-
generated VQA dataset.

Table 5: Dataset statistics for existing video question answering datasets Dataset | Videos | QA Pairs | QA Type | Source | QA Tasks
---|---|---|---|---|---
TutorialVQA [32] | 76 | 6195 | Manual | Screencast Tutorials | Open-ended
LifeQA [33] | 275 | 2328 | Manual | Daily life videos | Multiple-choice
EgoVQA [34] | 520 | 600 | Manual | Egocentric video studies |  | Open-ended and
---
multiple-choice
ActivityNetQA [35] | 5800 | 58,000 | Manual | ActivityNet dataset | Open-ended
MovieQA [36] | 6771 | 6462 | Manual | Movies | Multiple-choice
PororoQA [8] | 16,066 | 8913 | Manual | Cartoon Videos | Multiple-choice
MarioQA [37] | - | 187,757 | Automatic | Gameplay videos | Multiple-choice
TGIF-QA [44] | 71,741 | 165,165 |  | Automatic and
---
Manual
| TGIF(internet animated
---
GIFs)
| Open-ended and
---
multiple-choice
TVQA [40] | 21,793 | 152,545 | Manual |  | Sitcoms,medical and
---
criminal drama
Multiple-choice
TVQA+ [41] | 4198 | 29,383 | Manual |  | Sitcoms,medical and
---
criminal drama
Multiple-choice
Know-IT-VQA [39] | 12,087 | 24,282 | Manual | Big Bang Theory | Multiple-choice
Movie-FIB [42] | 118,507 | 348,998 | Automatic | LSMDC 2016 | Open-ended
VideoQA [4] | 18,100 | 175,076 | Automatic | Online internet videos | Open-ended
DramaQA [43] | 23,928 | 16,191 |  | Automatic and
---
Manual
TV Drama | Multiple-choice
Social-IQ [38] | 1250 | 7500 | Manual | YouTube Videos | Multiple-choice
MSRVTT-QA [7] | 10,000 | 243,680 | Automatic | MSRVTT(Web videos) | Open-ended
MSVD-QA [7] | 1970 | 50,505 | Automatic | MSVD(Web videos) | Open-ended
Youtube2TextQA [48] | 1987 | 122,708 | Automatic | Youtube2Text | Multiple-choice

##  3 Methods

VQA is a complex task which that combines the domain knowledge of both
computer vision and natural language processing. The methods implemented for
VQA tasks have been extended from the methods of Image-QA. VQA is more
challenging compared to Image-QA because it expects spatial as well as
temporal mapping and requires understanding of complex object interactions in
the video scenes. One of the first works on VQA was proposed by [1]. They
build a query-answering system based on a joint parsing graph from videos as
well as texts. As an improvement over the model proposed by [1], [3] proposes
a flexible encoder-decoder architecture using RNN, which covers a wide range
of temporal information. Leveraging the task of VQA, [4] proposes a model
which is capable of generating QA pairs from learned video descriptions. [4]
constructs a large-scale VQA dataset with $\sim 18$K videos and $\sim 175$K
candidate QA pairs. In comparison to previous baseline methods of Image-QA,
the model proposed in [4] proves to be efficient.

Following the aforementioned approaches, several methods have evolved in the
literature to solve the complex task of VQA. The first sub-section presents
the methods based on spatio-temporal approaches. The second sub-section deals
with memory-based methods. Methods using attention mechanism are listed in the
third sub-section. The fourth sub-section details the multimodal attention
based approaches. In addition to these, various novel approaches for VQA are
described in the later section. A summary of results from all the described
models is abridged in tables 1, 2, 3 and 4.

###  3.1 Spatio-Temporal Methods

Joint reasoning of spatial and temporal structures of a video is required to
accurately tackle the problem of VQA. The spatial structure will give
information about in-frame actions and temporal structure will analyze the
sequence of actions taking place in the videos. Methods which work on the same
motivation are described in detail in the below sections.

####  3.1.1 r-STAN (Zhao et al., 2017)

A Structured Two-stream Attention network (STA) is developed by [18] with the
help of a structured segment component and the encoded text features. The
structured two-stream attention component focuses on the relevant visual
instances from the video by neglecting the unimportant information like
background. The fusion of video and question aware representations generates
the inferred answer. Evaluation of the model is conducted on the famous TGIF-
QA dataset.

####  3.1.2 STA(R) (Gao et al., 2019)

A Structured Two-stream Attention network (STA) is developed by [18] with the
help of a structured segment component and the encoded text features. The
structured two-stream attention component focuses on the relevant visual
instances from the video by neglecting the unimportant information like
background. The fusion of video and question aware representations generate
the inferred answer. Evaluation of the model is conducted on the famous TGIF-
QA dataset.

####  3.1.3 L-GCN (Huang et al., 2020)

Along with the extraction of spatio-temporal features, taking into account the
information about object interactions in the videos helps to infer accurate
answers for VQA. Considering this, [21] proposes a novel and one of the first
graph model network called Location-Aware Graph Network. In this approach, an
object is associated to each node by considering features like appearance and
location. Inferences like category and temporal locations are obtained from
the graph with the help of graph convolution. The final answer is then
generated by merging the outputs of the graph and the question features. The
proposed method [21] is evaluated on three datasets - YouTube2Text-QA [48],
TGIF-QA [44], and MSVD-QA [7].

####  3.1.4 QueST (Jiang et al., 2020)

Often more significance is given to spatial and temporal features derived from
the videos but less to questions, therefore, accounting to this, the authors
of [28] propose a novel Question-Guided Spatio-Temporal contextual attention
network (QueST) method. In QueST [28], semantic features are divided into
spatial and temporal parts, which are then used in the process of constructing
contextual attention in spatial and temporal dimensions. With contextual
attention, video features can be extracted from both spatio-temporal
dimensions. Empirical results are obtained by evaluating the model on TGIF-QA
[44], MSVD-QA [7], and MSRVTT-QA [7] datasets.

###  3.2 Memory-based Methods

To answer questions for VQA, models need to keep track of the past and future
frames as the answers would require inferences from multiple frames in time.
Below are descriptions of some of the approaches which implemented memory-
based models.

####  3.2.1 DEMN (Kim et al., 2017)

A novel method based on video-story learning is developed by [8] which uses
Deep Embedded Memory Network (DEMN) to reconstruct stories from a joint scene
dialogue video stream. The model uses a latent embedding space of data. Long
short term memory (LSTM) based attention model is used to store video stories
for answering the questions based on specific words. The model consists of
three modules: video story understanding, story selection and answer
selection. Baseline models such as Bag-of-Words (BOW), Word2vec(W2V), LSTM,
end to end memory networks, and SSCB are used to compare with the proposed
model. In addition to the proposed model, authors also create PororoQA dataset
which is the first VQA dataset having a coherent story line throughout the
dataset.

####  3.2.2 Co-Mem (Gao et al., 2018)

A novel method of motion appearance co-memory networks is developed by [10]
which considers the following two observations. First, motion and appearance
when correlated becomes an important feature to be considered for the VQA
task. Second, different questions have different requirements of the number of
frames to form answer representations. The proposed method uses two stream
models that convert the videos into motion and appearance features. The
extracted features are then passed to a temporal convolutional and
deconvolutional neural network to generate the multi-level contextual facts.
These facts are served as input to the co-memory networks which are built upon
the concepts of Dynamic Memory Networks (DMN). Co-memory networks aid in joint
modeling of the appearance and motion information with the help of a co-memory
attention mechanism. The co-memory attention mechanism takes the appearance
cues for generating motion attention and motion cues for generating appearance
attention. The proposed model is evaluated on the TGIF-QA [44] dataset.

####  3.2.3 AHN (Zhao et al., 2018)

To tackle the problem of long-form VQA, the authors of [11] create an adaptive
hierarchical reinforced network (AHN). The encoder-decoder network comprises
adaptive encoder network learning and a reinforced decoder network learning.
In adaptive encoder learning, the attentional recurrent neural networks
segment the video to obtain the semantic long-term dependencies. It also aids
the learning of joint representation of the segments and relevant video frames
as per the question. The reinforced decoder network generates the open-ended
answer by using the segment level LSTM networks. A new large scale long-form
VQA dataset, created from the ActivityNet data [50], is also contributed by
[11]. For the long-form VQA task, the proposed model is compared with state-
of-the-art methods [STAN [6], MM+ [4] and ST-VQA [44]] on which the new model
outperforms.

####  3.2.4 Fan et al., 2018

[13] proposes a novel heterogeneous memory component to integrate appearance
and motion features and learn spatio-temporal attention simultaneously. A
novel end-to-end VQA framework consisting of three components is proposed. The
first component includes heterogeneous memory which extracts global contextual
information from appearance and motion features. The second component of
question memory enables understanding of complex semantics of questions and
highlights queried subjects. The third component of multimodal fusion layer
performs multi-step reasoning by attending relevant textual and visual clues
with self-attention. Finally, an attentional fusion of the multimodal visual
and textual representations helps infer the correct answer. [13] outperforms
existing state-of-the-art models on the four benchmark datasets - TGIF-QA
[44], MSVD-QA [7], MSRVTT-QA [7], and YouTube2Text-QA [48].

####  3.2.5 MHMAN (Yu et al., 2020)

Humans give attention only to certain important information to answer
questions from a video rather than remembering the whole content. Working on
the same motivation, [30] proposes a framework comprising of two heterogeneous
memory sub-networks. The top-guided memory network works at a coarse-grained
level which extracts filtered-out information from the questions and videos at
a shallow level. The bottom memory networks learn fine-grained attention by
guidance from the top-guided memory network that enhances the quality of QA.
Proposed MHMAN [30] model is evaluated on three datasets - ActivityNet-QA
[35], MSRVTT-QA [7] and MSVD-QA [7]. Results demonstrate that MHMAN [30]
outperforms the previous state-of-the-art models such as CoMem [10] and HME
[13].

###  3.3 Attention-based Methods

Attention mechanism plays an important role in VQA as it helps to extract
answers efficiently from the videos by giving more substance to certain
relevant words from the questions. The approaches mentioned below develop
models by exploiting the different aspects of attention mechanism.

####  3.3.1 r-ANL (Ye et al., 2017)

As an improvement to the earlier presented approach [3] which considered
temporal dynamics, [5] also incorporates frame-level attention mechanism to
improve the results. [5] proposes an attribute-augmented attention network
that enables the joint frame-level attribute detection and unification of
video representations. Experiments are conducted with both multiple-choice and
open-ended questions on the YouTube2Text dataset [50]. The model is evaluated
by extending existing Image-QA methods and it outperforms the previous Image-
QA baseline methods.

####  3.3.2 Xu et al., 2017

In [7], authors exploit the appearance and motion information resided in the
video with a novel attention mechanism. To develop an efficient VQA model, [7]
proposes an end-to-end model which refines attention over the appearance and
motion features of the videos using the questions. The weight representations
and the contextual information of the videos are used to generate the answer.
[7] also creates two datasets - MSVD-QA and MSRVTT-QA from the existing video-
captioning datasets - MSVD [49] and MSRVTT [45].

####  3.3.3 DLAN (Zao et al., 2017)

In [9], authors study the problem of VQA from the viewpoint of Hierarchical
Dual-Level Attention Network (DLAN) learning. On the basis of frame-level and
segment-level feature representations, object appearance and movement
information from the video is obtained. The object appearance is obtained from
frame-level representations using a 2D-ConvNet whereas movement information
based on segment-level features is done using a 3D-ConvNet. Dual-level
attention mechanism is used to learn the question-aware video representations
with question-level and word-level attention mechanisms. The authors also
construct large scale VQA datasets from existing datasets - VideoClip [51] and
YouTube2Text [48].

####  3.3.4 Resnet+CDC (AA) (WenqiaoZhang et al., 2019)

Inspired from previous contributions based on attention mechanisms in VQA
systems, [15] studies the problem related to unidirectional attention
mechanism which fails to yield a better mapping between modalities. [15] also
works on the limitation of previous works that do not explore high-level
semantics at augmented video-frame level. In the proposed model, each frame
representation with context information is augmented by a feature extractor-
(ResNet+C3D variant). In order to yield better joint representations of video
and question, a novel alternating attention mechanism is introduced to attend
frame regions and words in question in multi-turns. Furthermore, [15] proposes
the first-ever architecture without unidirectional attention mechanism for
VQA.

####  3.3.5 PSAC (Li et al., 2019)

In contrast to previous works like [10], [16] proposes Positional Self
Attention with Co-attention (PSAC) architecture without recurrent neural
networks for addressing the problem of VQA. Despite the success of various RNN
based models, sequential nature of RNNs pose a problem as they are time-
consuming and they cannot efficiently model long range dependencies. Thus,
positional self attention is used to calculate the response at each position
by traversing all positions of sequence and then adding representations of
absolute positions. PSAC [16] exploits the global dependencies of temporal
information of video and question, executing video and question encoding
processes in parallel. In addition to positional attention, co-attention is
used to attend relevant textual and visual features to guarantee accurate
answer generation. Extensive experimentation on TGIF-QA [44] shows that it
outperforms RNN-based models in terms of computation time and performance.

####  3.3.6 TSN (Yang et al., 2019)

The authors of [17] work on the failure of existing methods that take into
account the attentions on motion and appearance features separately. Based on
the motivation to process appearance and motion features synchronously, [17]
proposes a Question-Aware Tube Switch Network (TSN) for VQA. The model
comprises of two modules. First, correspondence between appearance and motion
is extracted with the help of a mix module at each time slice. This
combination of appearance and motion representation helps achieve fine-grained
temporal alignment. Second, to choose among the appearance or motion tube in
the multi-hop reasoning process, a switch module is implemented. Extensive
experiments for the proposed method [17] have been carried out on the two
benchmark datasets - MSVD-QA [7] and MSRVTT-QA [7].

####  3.3.7 PAMN (Kim et al., 2019)

[20] proposes the Progressive Attention Memory Network (PAMN) for movie story
QA. PAMN [20] is designed to overcome 2 main challenges of movie story QA:
finding out temporal parts for discovering relevant answers in case of movies
that are longer in length and to fulfill the need to infer the answers of
different questions on the basis of different modalities. PAMN [20] includes 3
main features: progressive attention mechanism which uses clues from video and
question to find the relevant answer, dynamic modality fusion and belief
correction answering scheme that improves the prediction on every answer. PAMN
[20] performs experiments on two benchmark datasets: MovieQA [36] and TVQA
[40].

####  3.3.8 CAN (Yu et al., 2019)

Earlier works in the domain of VQA have achieved favorable performance on
short-term videos. Those approaches fail to work equally well on the long-term
videos. Considering this drawback, [22] proposes a two-stream fusion strategy
called compositional attention networks (CAN) for the VQA tasks. The method
comprises of an encoder and a decoder. The encoder uses a uniform sampling
stream (USS) which extracts the global semantics of the entire video and the
action pooling stream (APS) is responsible for capturing the long-term
temporal dynamics of the video. The decoder utilizes a compositional attention
module (CAN) to combine the two-stream features generated by encoder using
attention mechanism. The model is evaluated on two-short term (MSRVTT-QA [7]
and MSVD-QA [7]) and one long-term (ActivityNet-QA [35]) VQA datasets.

###  3.4 Multimodal Attention based Methods

The task of VQA demands representations from multiple modalities such as
videos and texts. Therefore, several approaches have worked upon the concept
of multimodal attention to fuse the extracted knowledge from both modalities
and obtain accurate answers.

####  3.4.1 MDAM (Kim et al., 2019)

A video story QA architecture is proposed by [12], in which authors improve on
[8] by proposing a Multimodal Dual Attention Model (MDAM). The main idea
behind [12] was to incorporate a dual attention method with late fusion. The
proposed model uses self attention mechanism to learn the latent concepts from
video frames and captions. For a question, the model uses the attention over
latent concepts. After performing late fusion, multimodal fusion is done. The
main objective of [12] was to avoid the risk of over-fitting by using
multimodal fusion methods such as concatenation or multimodal bilinear
pooling. Using this technique, MDAM [12] learns to infer joint level
representations from an abstraction of the video content. Experiments are
conducted on the PororoQA and MovieQA datasets.

####  3.4.2 Chao et al., 2019

One of the main challenges in VQA is that for a long video, the question may
correspond to only a small segment of the video. Thus, it becomes inefficent
to encode the whole video using an RNN. Therefore, authors of [14], propose a
Question-Guided Video Representation module which summarizes the video frame-
level features by employing an attention mechanism. Multimodal representations
are then generated by fusing video summary and question information. The
multimodal representations and dialogue-contexts are then passed with
attention as an input in seq2seq model for answer generation. The authors
conduct experiments using Audio-Visual Scene-Aware Dialog dataset (AVSD) [46]
for evaluating their approach.

####  3.4.3 AMN (Yuan et al., 2020)

In [23], authors present a framework Adversarial Multimodal Network (AMN) to
better understand movie story QA. Most movie story QA methods proposed by
[8],[12],[20] have 2 limitations in common. The models fail to learn coherent
representations for multimodal videos and corresponding texts. In addition to
that, they neglect the information that retains story cues. AMN [23] learns
multimodal features by finding a coherent subspace for videos and the texts
corresponding to those videos based on Generative Adversarial Networks (GANs).
The model consists of a generator and a discriminator which constitute a
couple of adversarial modules. The authors compare AMN [23] with other
extended Image-QA baseline models and previous state-of-the-art methods for
MovieQA [36] and TVQA [40] datasets.

####  3.4.4 VQA-HMAL (Zhao et al., 2020)

[24] proposes a method for open-ended long-form VQA which works upon the
limitations of the short-form VQA methods. Short-form VQA methods do not
consider the semantic representation in the long-form video contents hence
they are insufficient for modeling long-form videos. The authors of [24]
implement a hierarchical attentional encoder network using adaptive video
segmentation which jointly extracts the video representations with long-term
dependencies based on the questions. In order to generate the natural language
answer, a decoder network based on a Generative Adversarial Network (GAN) is
used on the multimodal text-generation task. Another major contribution of
this paper accounts for generating three large-scale datasets for VQA from the
ActivityNet [50], MSVD [49], TGIF [51] datasets.

####  3.4.5 Jin et al., 2019

[19] proposes a novel multi-interaction network that uses two different types
of interactions, i.e. multimodal and multi-level interaction. The interaction
between the visual and textual information is termed as multimodal
interaction. Multi-level interaction occurs inside the multimodal interaction.
The proposed new attention mechanism can simultaneously capture the element-
wise and the segment-wise sequence interactions. Moreover, fine-grained
information is extracted using object-level interactions that aid in fetching
object motions and their interactions. The model is evaluated over the TGIF-
QA, MSVD-QA and MSRVTT-QA datasets.

###  3.5 Miscellaneous Models

In addition to the spatio-temporal, memory-based, attention-based and
multimodal attention based models, the following methods use different novel
approaches to solve the problem of VQA.

####  3.5.1 MAR-VQA(a+m) (Zhuang et al., 2020)

[25] takes into consideration multiple channels of information present in a
video such as audio, motion and appearance. It differs from earlier works
which either represented the question as a single semantic vector or directly
fused the features of appearance and motion leading to loss of important
information. [25] also incorporates external knowledge from Wikipedia which
suggests attribute text related to the videos and aids the proposed model with
support information. The model is experimented on two datasets - TGIF-QA and
ActivityNet-QA.

####  3.5.2 MQL (Lei et al., 2020)

A majority of previous works in VQA consider video-question pairs separately
during training. However,there could be many questions addressed to a
particular video and most of them have abundant semantic relations. In order
to explore these semantic relations, [26] proposes Multi-Question Learning
(MQL) that learns multiple questions jointly with their candidate answers for
a particular video sequence. These joint learned representations of video and
question can then be used to learn new questions. [26] introduces an efficient
framework and training paradigm for MQL, where the relations between video and
questions are modeled using attention network. This framework enables the co-
training of multiple video-question pairs. MQL [26] is capable to perform
better understanding and reasoning even in the case of a single question.
Authors carry out extensive experiments on 2 popular datasets- TVQA and CPT-
QA.

####  3.5.3 Yang et al., 2020

In natural language processing, BERT has outperformed the LSTM in several
tasks. However, [27] presents one of the first works which explores BERT in
the domain of computer vision. [27] proposes to use BERT representations to
capture both the language and the visual information with the help of a pre-
trained language-based transformer. Firstly, Faster-RCNN is used to extract
the visual semantic information from video frames as visual concepts. Next,
two independent flows are developed to process the visual concepts and the
subtitles along with the questions and answers. In the individual flows, a
fine-tuned BERT network is present to predict the right answer. The final
prediction is obtained by jointly processing the output of the two independent
flows. The proposed model is evaluated on TVQA [40] and Pororo [8] datasets.

####  3.5.4 HCRN (Le et al., 2020)

[29] proposes a general-purpose reusable Conditional Relation Network (CRN)
that acts as a foundational block for constructing more complex structures for
reasoning and representation on video. CRN [29] takes an input array of
tensorial objects with a conditioning feature and computes an output array of
encoded objects. The model consists of a stack of these reusable units for
capturing contextual information and diverse modalities. This structural
design of the model bolsters multi-step and high-order relational reasoning.
CRN [29] hierarchy constitutes 3 main layers for capturing clip motion as
context, linguistic context and video motion as context. One important
advantage of CRN over [10],[16] and [13] is that it scales well on long-sized
video simply with the addition of more layers in the structure. Performance
evaluation methods demonstrate that CRN [29] achieves high performance
accuracy as compared to various state-of-the-art models. CRN [29] is evaluated
on 3 benchmark datasets- TGIF-QA [44], MSRVTT-QA [7] and MSVD-QA [7].

##  4 Discussion

Video Question Answering (VQA) is a complex and challenging task in the domain
of Artificial Intelligence. Extensive literature is available for VQA and to
the best of our knowledge no previous survey has been conducted for the same.
In this survey, we studied the prominent datasets and models that have been
researched in VQA. We also compare the results of all the models in tables 1,
2, 3 and 4 by grouping them according to the datasets. Significant
improvements can be witnessed in the past years in the field of VQA.
Therefore, there lies a huge potential for research in this field for future
tasks.

## References

  * [1] Tu, Kewei, et al. ”Joint video and text parsing for understanding events and answering queries.” IEEE MultiMedia 21.2 (2014): 42-70.
  * [2] Yu, Youngjae, et al. ”End-to-end concept word detection for video captioning, retrieval, and question answering.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.
  * [3] Zhu, Linchao, et al. ”Uncovering the temporal context for video question answering.” International Journal of Computer Vision 124.3 (2017): 409-421.
  * [4] Zeng, Kuo-Hao, et al. ”Leveraging video descriptions to learn video question answering.” Thirty-First AAAI Conference on Artificial Intelligence. 2017.
  * [5] Ye, Yunan, et al. ”Video question answering via attribute-augmented attention network learning.” Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval. 2017.
  * [6] Zhao, Zhou, et al. ”Video Question Answering via Hierarchical Spatio-Temporal Attention Networks.” IJCAI. 2017.
  * [7] Xu, Dejing, et al. ”Video question answering via gradually refined attention over appearance and motion.” Proceedings of the 25th ACM international conference on Multimedia. 2017.
  * [8] Kim, Kyung-Min, et al. ”Deepstory: Video story qa by deep embedded memory networks.” arXiv preprint arXiv:1707.00836 (2017).
  * [9] Zhao, Zhou, et al. ”Video question answering via hierarchical dual-level attention network learning.” Proceedings of the 25th ACM international conference on Multimedia. 2017.
  * [10] Gao, Jiyang, et al. ”Motion-appearance co-memory networks for video question answering.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
  * [11] Zhao, Zhou, et al. ”Open-Ended Long-form Video Question Answering via Adaptive Hierarchical Reinforced Networks.” IJCAI. 2018.
  * [12] Kim, Kyung-Min, et al. ”Multimodal dual attention memory for video story question answering.” Proceedings of the European Conference on Computer Vision (ECCV). 2018.
  * [13] Fan, Chenyou, et al. ”Heterogeneous memory enhanced multimodal attention model for video question answering.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.
  * [14] Chao, Guan-Lin, et al. ”Learning question-guided video representation for multi-turn video question answering.” arXiv:1907.13280 (2019).
  * [15] Zhang, Wenqiao, et al. ”Frame Augmented Alternating Attention Network for Video Question Answering.” IEEE Transactions on Multimedia 22.4 (2019): 1032-1041.
  * [16] Li, Xiangpeng, et al. ”Beyond rnns: Positional self-attention with co-attention for video question answering.” Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.
  * [17] Yang, Tianhao, et al. ”Question-aware tube-switch network for video question answering.” Proceedings of the 27th ACM International Conference on Multimedia. 2019.
  * [18] Gao, Lianli, et al. ”Structured two-stream attention network for video question answering.” Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.
  * [19] Jin, Weike, et al. ”Multi-interaction network with object relation for video question answering.” Proceedings of the 27th ACM International Conference on Multimedia. 2019.
  * [20] Kim, Junyeong, et al. ”Progressive attention memory network for movie story question answering.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.
  * [21] Huang, Deng, et al. ”Location-Aware Graph Convolutional Networks for Video Question Answering.” AAAI. 2020.
  * [22] Yu, Ting, et al. ”Compositional attention networks with two-stream fusion for video question answering.” IEEE Transactions on Image Processing 29 (2019): 1204-1218.
  * [23] Yuan, Zhaoquan, et al. ”Adversarial Multimodal Network for Movie Story Question Answering.” IEEE Transactions on Multimedia (2020).
  * [24] Zhao, Zhou, et al. ”Open-Ended Video Question Answering via Multi-Modal Conditional Adversarial Networks.” IEEE Transactions on Image Processing 29 (2020): 3859-3870.
  * [25] Zhuang, Yueting, et al. ”Multichannel Attention Refinement for Video Question Answering.” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 16.1s (2020): 1-23.
  * [26] Lei, Chenyi, et al. ”Multi-Question Learning for Visual Question Answering.” AAAI. 2020.
  * [27] Yang, Zekun, et al. ”BERT Representations for Video Question Answering.” The IEEE Winter Conference on Applications of Computer Vision. 2020.
  * [28] Jiang, Jianwen, et al. ”Divide and Conquer: Question-Guided Spatio-Temporal Contextual Attention for Video Question Answering.” AAAI. 2020.
  * [29] Le, Thao Minh, et al. ”Hierarchical Conditional Relation Networks for Video Question Answering.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
  * [30] Yu, Ting, et al. ”Long-Term Video Question Answering via Multimodal Hierarchical Memory Attentive Networks.” IEEE Transactions on Circuits and Systems for Video Technology (2020).
  * [31] Jiang, Pin, and Yahong Han. ”Reasoning with Heterogeneous Graph Alignment for Video Question Answering.” AAAI. 2020.
  * [32] Teney, Damien, et al.”Visual question answering: A tutorial.” IEEE Signal Processing Magazine 34.6 (2017): 63-75.
  * [33] Castro, Santiago, et al. ”LifeQA: A Real-life Dataset for Video Question Answering.” Proceedings of The 12th Language Resources and Evaluation Conference. 2020.
  * [34] Fan, Chenyou. ”EgoVQA-An Egocentric Video Question Answering Benchmark Dataset.” Proceedings of the IEEE International Conference on Computer Vision Workshops. 2019.
  * [35] Yu, Zhou, et al. ”Activitynet-QA: A dataset for understanding complex web videos via question answering.” Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.
  * [36] Tapaswi, Makarand, et al. ”Movieqa: Understanding stories in movies through question-answering.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
  * [37] Mun, Jonghwan, et al. ”Marioqa: Answering questions by watching gameplay videos.” Proceedings of the IEEE International Conference on Computer Vision. 2017.
  * [38] Zadeh, Amir, et al. ”Social-iq: A question answering benchmark for artificial social intelligence.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.
  * [39] Garcia, Noa, et al. ”KnowIT VQA: Answering knowledge-based questions about videos.” arXiv preprint arXiv:1910.10706 (2019).
  * [40] Lei, Jie, et al. ”Tvqa: Localized, compositional video question answering.” arXiv preprint arXiv:1809.01696 (2018).
  * [41] Lei, Jie, et al. ”Tvqa+: Spatio-temporal grounding for video question answering.” arXiv preprint arXiv:1904.11574 (2019).
  * [42] Maharaj, Tegan, et al. ”A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.
  * [43] Choi, Seongho, et al. ”DramaQA: Character-Centered Video Story Understanding with Hierarchical QA.” arXiv preprint arXiv:2005.03356 (2020).
  * [44] Jang, Yunseok, et al. ”Tgif-qa: Toward spatio-temporal reasoning in visual question answering.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.
  * [45] Xu, Jun, et al. ”Msr-vtt: A large video description dataset for bridging video and language.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
  * [46] Alamri, Huda, et al. ”Audio visual scene-aware dialog (avsd) challenge at dstc7.” arXiv preprint arXiv:1806.00525 (2018).
  * [47] Rohrbach, Anna, et al. ”A dataset for movie description.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
  * [48] Guadarrama, Sergio, et al. ”Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition.” Proceedings of the IEEE international conference on computer vision. 2013.
  * [49] Chen, David, and William B. Dolan. ”Collecting highly parallel data for paraphrase evaluation.” Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 2011.
  * [50] Caba Heilbron, Fabian, et al. ”Activitynet: A large-scale video benchmark for human activity understanding.” Proceedings of the ieee conference on computer vision and pattern recognition. 2015.
  * [51] Li, Yuncheng, et al. ”TGIF: A new dataset and benchmark on animated GIF description.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016.