# Cross-Data Knowledge Graph Construction for LLM-enabled Educational
Question-Answering System: A Case Study at HCMUT

Tuan Bui<EMAIL_ADDRESS>0000-0002-8587-182X Ho Chi Minh City
University of Technology (HCMUT)HoChiMinhCityVietNam ,  Oanh Tran
<EMAIL_ADDRESS>Ho Chi Minh City University of Technology
(HCMUT)HoChiMinhCityVietNam ,  Phuong Nguyen<EMAIL_ADDRESS>Ho Chi Minh City University of Technology (HCMUT)HoChiMinhCityVietNam ,  Bao
Ho<EMAIL_ADDRESS>Ho Chi Minh City University of Technology
(HCMUT)HoChiMinhCityVietNam ,  Long Nguyen<EMAIL_ADDRESS>Ho
Chi Minh City University of Technology (HCMUT)HoChiMinhCityVietNam ,  Thang
Bui<EMAIL_ADDRESS>Ho Chi Minh City University of Technology
(HCMUT)HoChiMinhCityVietNam and  Tho Quan<EMAIL_ADDRESS>0000−0003−0467−6254 Ho Chi Minh City University of Technology
(HCMUT)HoChiMinhCityVietNam

(2018)

###### Abstract.

In today’s rapidly evolving landscape of Artificial Intelligence, large
language models (LLMs) have emerged as a vibrant research topic. LLMs find
applications in various fields and contribute significantly. Despite their
powerful language capabilities, similar to pre-trained language models (PLMs),
LLMs still face challenges in remembering events, incorporating new
information, and addressing domain-specific issues or hallucinations. To
overcome these limitations, researchers have proposed Retrieval-Augmented
Generation (RAG) techniques, some others have proposed the integration of LLMs
with Knowledge Graphs (KGs) to provide factual context, thereby improving
performance and delivering more accurate feedback to user queries.

Education plays a crucial role in human development and progress. With the
technology transformation, traditional education is being replaced by digital
or blended education. Therefore, educational data in the digital environment
is increasing day by day. Data in higher education institutions are diverse,
comprising various sources such as unstructured/structured text, relational
databases, web/app-based API access, etc. Constructing a Knowledge Graph from
these cross-data sources is not a simple task. This article proposes a method
for automatically constructing a Knowledge Graph from multiple data sources
and discusses some initial applications (experimental trials) of KG in
conjunction with LLMs for question-answering tasks.

Open Intent Discovery, Knowledge Graph, Large language model, Education,
Question-Answering System

††copyright: acmlicensed††journalyear: 2018††doi: XXXXXXX.XXXXXXX††conference:
(AIQAM’24: The 1st ACM Workshop in AI-powered Question & Answering Systems;
June 10, 2024; Phuket, ThaiLand††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computing
methodologies Information extraction††ccs: Information systems
Ontologies††ccs: Information systems Question answering††ccs: Information
systems Clustering and classification††ccs: Computing methodologies Ontology
engineering

##  1\. Introduction

A Large Language Model (LLM) is a language model trained on a considerably
large size corpus. Nowadays, LLMs attract much attention from various
communities due to their capacity to accomplish versatile language generation
and comprehension. Notable LLM models reported to the public include OpenAI’s
GPT (Radford et al., 2018), Google’s BART (Lewis et al., 2019), LLaMa (Touvron
et al., 2023) . To achieve a broad spectrum of general knowledge (Petroni et
al., 2019) and refine their language ability (Wei et al., 2022), state-of-the-
art LLMs nowadays are trained on a massive corpus of textual data. However, an
LLM’s completion might contain ”hallucinations” (Ji et al., 2023) because of
limited access to information that is as up-to-date, proprietary, or domain-
specific as humans. To address this issue and other limitations, some hybrid
models have shown prominent results by combining parametric memory with non-
parametric memory (Karpukhin et al., 2020). Since their knowledge bases can be
directly revised and expanded, and retrieved knowledge can be inspected and
interpreted. This technique, which received a lot of traction after the
publication of the famous Facebook research paper (Lewis et al., 2021), is
well-known and coined as Retrieval-Augmented Generation (RAG).

However, in most real-world domains such as education, the data/information
often originate from various departments and faculties within an institution,
resulting in diverse data sources with different natures such as structured
text, unstructured text, databases, images, or access through API mechanisms
from existing web/app-based systems. For instance, at Ho Chi Minh City
University of Technology (HCMUT)111https://hcmut.edu.vn, unstructured data
originate from legal documents, frequently asked questions (FAQs) from student
support systems BKSI222https://mybk.hcmut.edu.vn/bksi/public/vi/ (Help Desk
System), news from the website, data from databases, and information retrieved
from API-enabled systems like teaching management systems
LMS333https://lms.hcmut.edu.vn/. Due to the cross-data nature, Knowledge
Graphs (KGs) (Hogan et al., 2021) serve as a suitable non-parametric memory
formalism for knowledge representation in the educational environment,
allowing the effective deployment of a RAG system in this context. However,
constructing a Knowledge Graph from multiple data sources, not specifically
designed to be interoperable, is not a trivial task, in particular when one
needs to deal with open intents [ref] commonly arising during casual
conversations between students and the university staffs. Hence, to the best
of our knowledge, there has not been a best-practice application of RAG from
KG for LLMs in practical scenarios.

In this paper, we aim to pioneer a KG-based RAG approach for an Educational
Question-Answering system. We propose a framework for the cross-data knowledge
graph construction in the educational domain, which is currently implemented
at HCMUT and leverages the Vietnamese language. This framework is applied as a
pilot in a Language Model (LLM)-based system. Our contributions are of three-
fold: (i) We introduce the technique of intental entity discovery for
unstructured text in Vietnamese FAQ conversations; (ii) we present the
embedding-based cross-data for relation discovery on education KG
construction; and (iii) we conduct real-world experiments, specifically, LLM-
enabled KG-based Question-Answering at HCMUT using RAG with the constructed
educational KG.

##  2\. Related Work

###  2.1. Open Intent Discovery

The task of Open Intent Discovery (Chen et al., 2022) presents several
challenges and difficulties in the field of Natural Language Understanding and
dialogue systems. One significant challenge is the inherent ambiguity and
variability in user expressions. Unlike traditional intent recognition, where
a predefined set of categories is used for classification, open intent
discovery involves identifying user intents that may not have been encountered
during the training phase. This introduces a level of uncertainty and
unpredictability, as users can express their intents in diverse and
unanticipated ways. Another challenge is the lack of sufficient labeled data
for training models in an open-world setting. Creating labeled datasets for
every potential intent becomes impractical, especially when dealing with a
wide range of domains and applications. This scarcity of annotated examples
for novel intents makes it challenging for models to generalize effectively
and accurately identify open intents. In recent years, there has been an
increased interest in determining user intent from both written and spoken
language, with a focus on modeling and comprehending interactions. Recent
studies, such as a method employing a bidirectional LSTM and CRF for detecting
user intent efficiently (Vedula et al., 2019), an unsupervised two-stage
approach aimed at discovering intents and generating meaningful intent labels
automatically from unlabeled utterances (Vedula et al., 2020), and a method
mining unlabeled utterance data to uncover common intents (Liu et al., 2021).
These studies collectively highlight the significance of robust approaches for
understanding user intent, paving the way for enhanced dialogue systems and
virtual assistants. Open intents has been studied in fields such as Customer
Care in businesses (Liu et al., 2021; Vedula et al., 2020) and Healthcare
facilities(Mullick et al., 2023); however, in the realm of education,
particularly in Student Care, it remains relatively unexplored.

###  2.2. Knowledge Graph in Education Domain

Knowledge Graphs (KGs) have evolved as an effective way to represent
knowledge. KGs present a structured and integrated representation of concepts,
relationships, and attributes within a domain. There have been many studies on
Educational KGs such as knowledge graph for mathematics (Chen et al., 2018),
ontology modeling university teaching programs and student profiles (EducOnto)
(Hubert et al., 2022), knowledge schema for university teaching (Rizun, 2019),
and knowledge graphs to identify labor market needs (education and employment)
(Fettach et al., 2022). Notably, research on KGs for the Vietnamese
educational area is sparse. Huong and Phuc (To and Do, 2020) introduced a
framework for extracting triples (subject-predicate-object relationships) from
Vietnamese. However, their technique has limitations in handling complex
sentences and is applied to the travel area.

###  2.3. KG-augmented LLMs

One major hurdle for LLMs is their occasional struggle with factual accuracy
and hallucinations. This is where KGs come in. KGs offer a structured
representation of knowledge, encoding entities and their relationships in a
machine-readable format. KG-augmented LLMs aim to bridge this gap by
leveraging the strengths of both approaches. If the generated information from
LLMs aligns with the ”ground truth” represented by the KGs, it can be
considered factually accurate and non-hallucinatory. This enhancing method
involves refining the inference process of LLMs (Baek et al., 2023),
optimizing learning mechanisms (Kim et al., 2023) , and establishing a result
validation mechanism (Kang et al., 2023). Significant progress has been made
through these studies, highlighting the importance of continuous innovation
and supporting the advancement of more advanced KG-augmented LLMs.

##  3\. Towards Cross-Data Knowledge Graph Construction for LLM-enabled
Educational Question-Answering System

In this section, we delve into the methodological approach employed in our
research to extract valuable insights from a multi-source educational data
environment for KG construction, served as the foundation of an LLM-enabled
educational QA system. We begin by discussing the nature of the data sources
in the university environment of HCMUT and how they contribute to a
comprehensive understanding of the educational ecosystem. We then introduce
the E-OED Framework (Educational Open Entity Discovery) , a robust methodology
designed to explore intents using unsupervised learning methods. This
framework is particularly adept at handling challenges such as overlapping
entities, open intents, and data fusion from multiple sources. Lastly, we
present our method for Relation Discovery, allowing us to explore
relationships among different types of entities and construct a KG
representing the education domain.

###  3.1. Multi-source educational data at HCMUT

In the dynamic landscape of a university environment like HCMUT, data streams
from a myriad of sources, forming a complex multi-source ecosystem. This
diverse and interconnected web of entities provides a comprehensive picture of
the university ecosystem, enabling a deeper understanding of user needs,
educational processes, and the overall university environment. includes the
following data sources.

Figure 1 presents a cross-data environment at HCMUT, which includes the
following data sources.

Figure 1. Multi-source educational data at HCMUT

  * •

FAQ and Help Desk System: In HCMUT, there is an online interactive system
allowing students to raise their concerns arising during their study and
feedback received from academic staff. Knowledge extractable from this source
includes open intents, events, student issues, university services, the
software system of the university.

  * •

Academic Portals/Websites: These are all the websites that HCMUT updates
frequently for academic news, university policies/regulations, and course
information. Knowledge extractable from this source includes guidance
documents, detailed curricula, and program descriptions offered by HCMUT,
along with other entities such as organizations (university, faculty,
department), locations (address number, city, district), and person names
(students, lecturers, staffs).

  * •

Learning Management System (LMS) and other supporting systems: These systems
help students keep track of their coursework throughout the school year.
Knowledge extractable from this source includes courses, keywords, discipline
terminology, discipline theory, books, documents, journals, and course
participants (students, lecturers, teaching assistants).

###  3.2. The E-OED Framework

Figure 2. Entity type in education domain

Figure 2 depicts the intricate task of extracting relationships within the
educational domain and exploring intents. For instance, a certain Student may
have his Student status, which is managed by Academic affairs office. This
student also belongs to a certain Faculty and is enrolling in various
instances of Class. The student can also leverage some academic service when
necessary such as Course drop or Transcript inquiry. The graph showcases the
complexity of mapping intents to academic entities, illustrating the nuanced
connections that exist within this domain. This complexity underscores the
necessity for robust methodologies to accurately decipher and analyze such
relationships. The most challenging task towards such a KG construction is
perhaps educational open entity discovery. These challenges include handling
overlapping entities, exploring open intents (which lack extensive research
compared to classified intents), and fusing data from multiple sources.
Moreover, the Vietnamese language presents additional hurdles due to its
status as a low-resource language. Furthermore, within educational
environments, the intent of conversations are dynamic rather than fixed, and
opened rather than limited. Consequently, the identification of open entities
is crucial in the educational sphere. The E-OED Framework (Educational Open
Entity Discovery), represented by Figure 3, aims to discover intents from FAQ
data. This framework includes three processing modules known as Data
Preprocessing, Semantic Clustering, and Automatic Cluster Labeling, each of
which plays a crucial role in the intent discovery process.

Figure 3. Educational Open Entity Discovery Framework

Data Preprocessing In this step, we split paragraphs to a list of single
sentences, filtering out the noise, etc. This initial filtering helps us focus
on the meat of the text.

Semantic Clustering To obtain sentence embedding, we employ SimCSE (Gao et
al., 2022) as our chosen embedding model. The model we use adheres to the
SimCSE framework and has been developed utilizing the renowned pre-trained
PhoBERT (Nguyen and Tuan Nguyen, 2020) base. Before applying the embeddings to
the clustering algorithm, it is essential to conduct dimension reduction
methods. This precautionary step is taken to address the ”curse of
dimensionality,” which can adversely impact the accuracy of clustering
algorithms by influencing distance metrics and introducing unintended noise.
We opt for UMAP over PCA and LDA. After implementing dimensionality reduction,
we opt for HDBSCAN (McInnes et al., 2017) as our clustering algorithm.

Automatic Cluster Labeling The final step in this process involves the
automatic labeling of the clusters. To accomplish this task, our initial
approach entails employing the word-segmentation tool for sentence
segmentation within each cluster. Subsequently, the segmented sentences
undergo analysis using the PhoNLP (Nguyen and Nguyen, 2021) to execute both
POS tagging and dependency tagging. Following the extraction of tags by the
PhoNLP, a rule-based approach is implemented. Further refinement is achieved
by identifying the most frequently occurring tokens within each subset, which
are then designated as our cluster labels. Figure 4 presents a compilation of
some of the discovered intents from the FAQ dataset as the results of Semantic
Clustering and Cluster Labelling processes. Involving courses and class
groups, there exists a multitude of inquiries ranging from course enrollment,
class transfers, and capacity expansions, to timetable matters. Investigating
the intents and associated entities can aid in providing good responses or
initiating subsequent actions.

Figure 4. List of some discovered intents

###  3.3. Embedding-based method for Relation Discovery

In the context of Relation Discovery between Intents and other entities, we
adopt an Embedding-based approach. Due to resource and time constraints, we
initially experimented with two major types of entities: intent and policy. To
accommodate a large amount of diverse data with varying lengths, we employ
two-stage retriever. We use Sentence-BERT to embed intent entities and policy
entities. Once the embeddings are generated, we map different types of
entities by measuring similarity or proximity between entity embeddings in
multidimensional space.

After getting mapping between entities using embedding, we use tf_idf to
rerank and give higher scores to policies that have similar keywords with the
intents. (Guo and Berkhahn, 2016), we identify implicit connections between
entities. This analysis enables us to uncover meaningful associations and
dependencies that might not be readily apparent from raw data alone. Figure 5
illustrates the approach of using Embedding-based methods for relation
discovery. Based on the discovered relationships, we construct a Knowledge
Graph (KG) representing the education domain. Each entity type is depicted as
a node in the graph, and relationships between entities are depicted as edges.
The KG offers a structured representation of the semantic landscape within the
education domain, facilitating efficient traversal, query processing, and
inference tasks.

Figure 5. Relation Discovery Module using Embedding-based method

##  4\. Experiments

###  4.1. Datasets

The experimental datasets consist of three different datasets Banking77_eng,
Banking77_vni, and FAQ_HCMUT_vni presented in Table 1. Banking77_en (Casanueva
et al., 2020) is an English dataset that contains 77 customer intents from
over 10,000 questions in the banking domain. Banking77_vni is a Vietnamese
dataset that is an auto-translation (using Google Translate) of the
Banking77_eng dataset. FAQ_HCMUT_vni is a Vietnamese dataset that contains
over 200,000 frequently asked questions (FAQs) collected from the HelpDesk
system of the HCMUT. The Banking77_eng and Banking77_vni datasets are used to
evaluate the performance of the our framework. While FAQ_HCMUT_vni dataset is
used to demonstrate the result of the OED framework. To investigate the impact
of different embeddings on clustering performance, our framework utilizes
Vietnamese SimCSE for Vietnamese datasets, while the original BERTopic employs
a sentence-transformers model (all-miniLM-L6 variant).

###  4.2. Discovered Educational Open Intents

Table 1 illustrates the outcomes of our experimentation in discovering intents
across the specified datasets. To validate the OED Framework, we executed the
BERTopic framework on two datasets, Banking77_eng and Banking77_vni. The
clustering results in Cases 1 and 2 demonstrate the superior performance of
the BERTopic framework with the English dataset, where 73 intents were
identified, closely aligning with the 77 predetermined categories. However, in
the Banking77_vni dataset, the count of identified intents notably decreased
to 65. In Case 3, the OED Framework exhibited better performance with the
Banking77_vni dataset compared to the BERTopic framework for the Vietnamese
language, yielding 76 extracted intents. Subsequently, we applied the OED
Framework to the HCMUT_FAQ_vni dataset (Case 4), resulting in a total of 284
clusters with significant noised clusters, and duplicated clusters. Despite
this, approximately 372 intents were derived from these clusters, although a
substantial number of intents remain undiscovered. As previously mentioned,
Figure 4 illustrates some remarkable open intents discovered by our approach.

Table 1. Open intent discovery result

Case Framework Dataset Cluster’s No Extracted intent 1 BERTopic Banking77_eng
157 73 2 BERTopic Banking77_vni 147 65 3 OED Framework Banking77_vni 257 76 4
OED Framework HCMUT_FAQ_vni 284 372

###  4.3. Embedding-based method for Relation Discovery Result

The embedding-based approach for relation discovery has yielded promising
results in exploring relationships between entities, particularly in the
education domain. Table 2 show the result of Embedding-based approach, in this
experiment, there are 243 intent entities and 237 policy entities. Following
our approach, and after several trials, we’ve settled on a threshold of 0.32.
A total of 613 relationships between entity pairs (intent and policy) have
been identified. Out of these, 53 intent entities do not have any associations
with policy entities. For instance, the intent ”download form” is broad and
lacks specific entity references. Additionally, among these 53 intent entities
without relationships, 22 intents are overlooked, indicating they indeed have
connections with policies but were not detected. For example, ”Cancel course”
and ”Withdraw course” are linked to the Withdrawal Policy.

Table 2. Relation discovery result

Entity pair [Intent, Policy] No of entities [243 , 237] Discovered
relationships 613 Non-associative intents 53 Overlooked intents 22

###  4.4. KG-augmented LLMs Approach in Educational Context

To address the ”KG-augmented LLMs” aspect of the research and its application
in the education domain at HCMUT, we conduct a general pipeline of the
question-answering system following the Figure 6 (a). The process of answering
user questions involves feeding the question into our institution’s LLM, URA,
which generates a structured query in the form of Neo4j’s Cypher language, as
our knowledge graph database is built on Neo4j. This query searches a
knowledge graph database handcrafted using intent discovery’s result to
retrieve matching sub-graphs and relevant triples. Next, the triples are
converted into a natural language format and combined with the original user
question to create a final prompt for the LLM. The prompt guides the LLM in
generating a coherent and accurate answer as demonstrated in Figure 6 (b).

(a) KG-augmented LLMs Approach (b) Demonstration of course withdrawal case

Figure 6. KG-augmented LLMs Approach and Demonstration of course withdrawal
case

###  4.5. Discussion

The section above outlines the outcomes of our research efforts focused on
three key areas: Discovering open-intents, Developing an embedding-based
technique for Relation Discovery, and Implementing KG-augmented LLMs approach.

First, discovering open-intents through unsupervised approach shows promising
results, but applying it to Vietnamese encounters various difficulties due to
the language’s specific characteristics. There is a considerable amount of
overlapping and repetitive clusters due to the inadequate performance of NLP
toolkits for Vietnamese. Named Entity (NE) within sentences further interferes
with the clustering process. Due to the subpar clustering step, automatic
labeling is also affected accordingly. Applying Hyponym-approach to these NERs
represents a potential approach for optimizing clustering.

Second, the Embedding-based method for Relation Discovery has shown promising
results in identifying relationships between entities. However, further
research is needed to explore the different types of relations that can be
discovered using this method. Additionally, there is a need to develop more
robust and accurate methods for relation extraction and classification.

Finally, the KG-augmented LLMs approach is initially implemented by using off-
the-shelf tools from Neo4J. The KG-retrieval engine extracts triplets. The
advantages of this method are that the information is condensed and accurate,
and the triplets also represent relations to help the LLM answer questions
better. However, the triplets retrieval engine only extracts relevant triples
and does not have a ranking mechanism to help eliminate redundant triples.

##  5\. Conclusion

In this article, we introduce an approach to open intent discovery using
unsupervised learning methods. To the best of our knowledge, this is the first
paper to use this approach for open intent discovery in Vietnamese. The
results are evaluated on three datasets: Bankinf77_eng, Banking77_vni, and
HCMUT_FAQ_vni. Additionally, we perform preliminary experiments on discovering
the relationships between intents and other entities to build a KG for the
education domain. We also conduct some experiments on applying the KG to LLM.

## References

  * (1)
  * Baek et al. (2023) Jinheon Baek, Alham Fikri Aji, and Amir Saffari. 2023.  Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering. In _Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE)_ , Bhavana Dalvi Mishra, Greg Durrett, Peter Jansen, Danilo Neves Ribeiro, and Jason Wei (Eds.). Association for Computational Linguistics, Toronto, Canada, 78–106.  https://doi.org/10.18653/v1/2023.nlrse-1.7
  * Casanueva et al. (2020) Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020.  Efficient Intent Detection with Dual Sentence Encoders.  arXiv:2003.04807 [cs.CL]
  * Chen et al. (2022) Minhua Chen, Badrinath Jayakumar, Michael Johnston, S. Eman Mahmoodi, and Daniel Pressel. 2022.  Intent Discovery for Enterprise Virtual Assistants: Applications of Utterance Embedding and Clustering to Intent Mining. In _North American Chapter of the Association for Computational Linguistics_.  https://api.semanticscholar.org/CorpusID:250390998
  * Chen et al. (2018) Penghe Chen, Yu Lu, Vincent W. Zheng, Xiyang Chen, and Boda Yang. 2018.  KnowEdu: A System to Construct Knowledge Graph for Education.  _IEEE Access_ 6 (2018), 31553–31563.  https://doi.org/10.1109/ACCESS.2018.2839607
  * Fettach et al. (2022) Yousra Fettach, Mounir Ghogho, and Boualem Benatallah. 2022.  Knowledge Graphs in Education and Employability: A Survey on Applications and Techniques.  _IEEE Access_ 10 (2022), 80174–80183.  https://doi.org/10.1109/ACCESS.2022.3194063
  * Gao et al. (2022) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2022.  SimCSE: Simple Contrastive Learning of Sentence Embeddings.  arXiv:2104.08821 [cs.CL]
  * Grootendorst (2022) Maarten Grootendorst. 2022.  BERTopic: Neural topic modeling with a class-based TF-IDF procedure.  _arXiv preprint arXiv:2203.05794_ (2022).
  * Guo and Berkhahn (2016) Cheng Guo and Felix Berkhahn. 2016.  Entity Embeddings of Categorical Variables.  arXiv:1604.06737 [cs.LG]
  * Hogan et al. (2021) Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia D’amato, Gerard De Melo, Claudio Gutierrez, Sabrina Kirrane, José Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, Axel-Cyrille Ngonga Ngomo, Axel Polleres, Sabbir M. Rashid, Anisa Rula, Lukas Schmelzeisen, Juan Sequeda, Steffen Staab, and Antoine Zimmermann. 2021.  Knowledge Graphs.  _Comput. Surveys_ 54, 4 (July 2021), 1–37.  https://doi.org/10.1145/3447772
  * Hubert et al. (2022) Nicolas Hubert, Armelle Brun, and Davy Monticolo. 2022.  New Ontology and Knowledge Graph for University Curriculum Recommendation.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of Hallucination in Natural Language Generation.  _Comput. Surveys_ 55, 12 (March 2023), 1–38.  https://doi.org/10.1145/3571730
  * Kang et al. (2023) Minki Kang, Jin Myung Kwak, Jinheon Baek, and Sung Ju Hwang. 2023.  Knowledge Graph-Augmented Language Models for Knowledge-Grounded Dialogue Generation.  arXiv:2305.18846 [cs.CL]
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020.  Dense Passage Retrieval for Open-Domain Question Answering.  arXiv:2004.04906 [cs.CL]
  * Kim et al. (2023) Seungone Kim, Se Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. 2023.  The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 12685–12708.  https://doi.org/10.18653/v1/2023.emnlp-main.782
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.  arXiv:1910.13461 [cs.CL]
  * Lewis et al. (2021) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021.  Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.  arXiv:2005.11401 [cs.CL]
  * Liu et al. (2021) Pengfei Liu, Youzhang Ning, King Keung Wu, Kun Li, and Helen Meng. 2021.  Open Intent Discovery through Unsupervised Semantic Clustering and Dependency Parsing.  arXiv:2104.12114 [cs.CL]
  * McInnes et al. (2017) Leland McInnes, John Healy, Steve Astels, et al. 2017\.  hdbscan: Hierarchical density based clustering.  _J. Open Source Softw._ 2, 11 (2017), 205.
  * Mullick et al. (2023) Ankan Mullick, Ishani Mondal, Sourjyadip Ray, R Raghav, G Sai Chaitanya, and Pawan Goyal. 2023.  Intent Identification and Entity Extraction for Healthcare Queries in Indic Languages.  arXiv:2302.09685 [cs.IR]
  * Nguyen and Tuan Nguyen (2020) Dat Quoc Nguyen and Anh Tuan Nguyen. 2020.  PhoBERT: Pre-trained language models for Vietnamese. In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 1037–1042.  https://doi.org/10.18653/v1/2020.findings-emnlp.92
  * Nguyen and Nguyen (2021) Linh The Nguyen and Dat Quoc Nguyen. 2021.  PhoNLP: A joint multi-task learning model for Vietnamese part-of-speech tagging, named entity recognition and dependency parsing. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations_. 1–7.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2019.  Language Models as Knowledge Bases?  arXiv:1909.01066 [cs.CL]
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018\.  Improving language understanding by generative pre-training.  (2018).
  * Rizun (2019) Mariia Rizun. 2019.  Knowledge Graph Application in Education: a Literature Review.  _Acta Universitatis Lodziensis. Folia Oeconomica_ 3 (08 2019), 7–19.  https://doi.org/10.18778/0208-6018.342.01
  * To and Do (2020) Huong Duong To and Phuc Do. 2020.  Extracting triples from Vietnamese text to create knowledge graph. In _12th International Conference on Knowledge and Systems Engineering, KSE 2020, Can Tho City, Vietnam, November 12-14, 2020_. IEEE, 219–223.  https://doi.org/10.1109/KSE50997.2020.9287471
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  arXiv:2307.09288 [cs.CL]
  * Vedula et al. (2019) Nikhita Vedula, Nedim Lipka, Pranav Maneriker, and Srinivasan Parthasarathy. 2019.  Towards Open Intent Discovery for Conversational Text.  arXiv:1904.08524 [cs.IR]
  * Vedula et al. (2020) Nikhita Vedula, Nedim Lipka, Pranav Maneriker, and Srinivasan Parthasarathy. 2020.  Open Intent Extraction from Natural Language Interactions. In _Proceedings of The Web Conference 2020_ (Taipei, Taiwan) _(WWW ’20)_. Association for Computing Machinery, New York, NY, USA, 2009–2020.  https://doi.org/10.1145/3366423.3380268
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers.  arXiv:2002.10957 [cs.CL]
  * Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022.  Emergent Abilities of Large Language Models.  arXiv:2206.07682 [cs.CL]

##  Appendix A Appendix

###  A.1. E-OED Framework Analysis

This appendix provides further details on the implementation of the E-OED
framework. We discuss key design choices, such as hyperparameter selection and
embedding models, along with the insights gained from related experiments.

This section will also discuss three core modules in E-OED frameworks known as
Data Preprocessing, Semantic Clustering and Automatic Cluster Labeling

The result achieved by different design choices is represented in detail at 4

####  A.1.1. Data Preprocessing

Due to Vietnamese being a predominantly analytic language with a large number
of monosyllabic morphemes (meaningful units), words are often formed by
combining these morphemes. This can present challenges for the original
BERTopic framework, which relies on cTF-IDF (class-based TF-IDF)(Grootendorst,
2022) for topic modeling. To address this, we implemented an additional
preprocessing step that leverages a word segmentation model to combine sub-
word units into complete words. These newly formed words are then joined using
an underscore character (”_”). (e.g., subject-”môn học” becomes ”môn_học”,
course-”khóa học” becomes ”khóa_học”).

Furthermore, to comply with dataset privacy policies, we implemented a text
anonymization tool to censor any personally identifiable information (PII)
within the sentences. Details regarding the specific objects censored and the
corresponding replacement terms are presented in Table 3

Table 3. Data censored tool

Objects Method Term Replace Person Name NER (BERT) [full_name] Student ID
Regex (7 number format) [student_id] Phone number Regex (10 number format)
[phone_number] Email Regex (ordinary email with @) [email]

####  A.1.2. Semantic Clustering

Our model consists of three primary components:

  * •

Sentence Embedding: extracts vector representations of sentences.

  * •

Dimension Reduction: addresses the ”curse of dimensionality” by compressing
these embeddings into a lower-dimensional space, improving computational
efficiency.

  * •

Sentence-density clustering: groups sentences based on their reduced-
dimensional representations.

Our model’s performance is highly sensitive to the chosen hyperparameters. We
explored various options for each module:

Sentence Embedding:

  * •

BERTopic Base Model (MiniLM-L6): A 6-layer Microsoft MiniLM (Wang et al.,
2020) fine-tuned on a 1 billion sentence pair dataset.

  * •

Supervised Vietnamese SimCSE: Fine-tuned from the PhoBERT model.

Dimension Reduction (UMAP):

  * •

Number of neighbors: 20, Number of components: 4

  * •

Number of neighbors: 15, Number of components: 9

Sentence-Density Clustering (HDBSCAN):

  * •

Minimum Cluster Size: 15

  * •

Minimum Cluster Size: 20

We evaluated different combinations of these hyperparameters to identify the
optimal configuration for our task.

####  A.1.3. Automatic Cluster Labeling

This section details the process for extracting descriptive labels for our
identified clusters. We first select the seven most representative sentences
for each cluster. These sentences are chosen based on their similarity to
keywords generated using cTF-IDF.

Within this approach, a key hyperparameter is the range of word lengths
considered for keyword extraction. We experiment with two settings: a range of
5-7 words and a range of 4-11 words. The first setting targets terms with
lengths between 5 and 7 words, while the second allows for terms between 4 and
11 words.

Following keyword extraction, the chosen sentences are processed using the
PhoNLP model to obtain Part-of-Speech (PoS) and dependency tags. We then
leverage these tags in conjunction with pre-defined rules (detailed in the
Algorithm 1) to generate descriptive labels for each cluster.

Algorithm 1 Tag extraction algorithms

1:function ExtractSentenceElements($category\\_docs$)

2: Initialize list of labels

3: for each sentence in $category\\_docs$ do

4: Clean the sentence by stripping whitespace

5: Get the annotation result from $PhoNLP$

6: Extract the part-of-speech (POS) and dependency (DEP) information

7: Initialize lists for verbs, direct objects, verb modifiers, and other list

8: Initialize variables for root position, direct object positions, verb
modifier position, and preposition position #Extract all related positions

9: Create a set to store the relevant positions

10: for each token in the sentence do

11: if the token is the root of the sentence then

12: Add its position to the set

13: while the length of the position set is changing do

14: for each token in the document do

15: if the token’s dependency is related to the root or other relevant
positions then

16: Add its position to the set

17: if the length of the set doesn’t change then

18: Break the loop #Extract the sentence elements

19: for each token in the document do

20: if the token’s dependency is ’root’ then

21: Add it to the verbs and consecutive lists

22: if the token’s dependency is ’prp’ and its position is in the set then

23: Add it to the direct objects lists

24: if the token’s dependency is ’dob’ then

25: Add it to the direct objects list

26: if the token’s dependency is ’vmod’ and its position is related to the
root, preposition, or verb modifier positions then

27: Add it to the verb modifiers list

28: if the token’s dependency is ’nmod’ and its position is related to a
direct object position then

29: Add it to the other list

30: if the token’s position or its dependency’s position is in the set and the
dependency is in a predefined list then

31: Add the token to the other list

32: Extract the most frequent tokens in each list.

33: Add the tokens into list of labels

34: return list of labels

In order to reduce noises on newly extracted tags, we then take the most
frequently occurring tokens within each subset, which are then designated as
our cluster labels. For better cluster labels, LLMs can be used.

####  A.1.4. Result and Insights

There are 5 experiments with different design selection in total:

  1. (1)

Experiment 1:

     * •

Dataset: Banking77_eng

     * •

Number of Neighbors: 20

     * •

Number of Components: 4

     * •

Minimum Cluster Size: 20

     * •

cTF-IDF Range: [5-7]

  2. (2)

Experiment 2:

     * •

Dataset: Banking77_vni

    1. (a)

Experiment 2.1:

       * •

Number of Neighbors: 20

       * •

Number of Components: 4

       * •

Minimum Cluster Size: 20

       * •

cTF-IDF Range: [5-7]

    2. (b)

Experiment 2.2:

       * •

Number of Neighbors: 15

       * •

Number of Components: 9

       * •

Minimum Cluster Size: 15

       * •

cTF-IDF Range: [4-11]

  3. (3)

Experiment 3:

     * •

Dataset: Banking77_vni

     * •

Language Model: SimCSE (instead of MiniLM)

    1. (a)

Experiment 3.1

       * •

Number of Neighbors: 20

       * •

Number of Topics: 4

       * •

Minimum Cluster Size: 20

       * •

cTF-IDF Range: [5-7]

    2. (b)

Experiment 3.2

       * •

Number of Neighbors: 15

       * •

Number of Components: 9

       * •

Minimum Cluster Size: 15

       * •

cTF-IDF Range: [4-11]

Table 4. Design selection result

Experiment Intentions Discovered Experiment 1 73 Experiment 2.1 65 Experiment
2.2 74 Experiment 3.1 71 Experiment 3.2 76

Our experiments revealed several key insights. Firstly, the original embedding
model, despite not being trained on Vietnamese data, achieved reasonable
performance on our banking77_vni dataset. This suggests potential
transferability due to structural similarities across languages. However,
fine-tuning a dedicated embedding model specifically for Vietnamese could
likely yield further improvements. Evaluating the quality of the embedding
model can be achieved by analyzing the representative sentences within each
cluster. Greater semantic similarity within clusters signifies a more
effective model.

Secondly, a higher number of components in UMAP increases focus on local
attributes, leading to greater separation and potentially clearer
visualizations. However, it’s crucial to be mindful of the ”curse of
dimensionality” when employing this approach.

Finally, the framework is sensitive to hyperparameter selection. Due to the
presence of numerous hyperparameters requiring tuning, careful consideration
should be given when applying this method to your specific domain or FAQ
dataset.


*mainfig/extern/iccvfinal

# On the hidden treasure of dialog in video question answering

Deniz Engin1,2 François Schnitzler2 Ngoc Q. K. Duong2 Yannis Avrithis1
1Inria, Univ Rennes, CNRS, IRISA 2InterDigital

###### Abstract

High-level understanding of stories in video such as movies and TV shows from
raw data is extremely challenging. Modern video question answering (VideoQA)
systems often use additional human-made sources like plot synopses, scripts,
video descriptions or knowledge bases. In this work, we present a new approach
to understand the whole story without such external sources. The secret lies
in the dialog: unlike any prior work, we treat dialog as a noisy source to be
converted into text description via _dialog summarization_ , much like recent
methods treat video. The input of each modality is encoded by transformers
independently, and a simple fusion method combines all modalities, using soft
temporal attention for localization over long inputs. Our model outperforms
the state of the art on the KnowIT VQA dataset by a large margin, without
using question-specific human annotation or human-made plot summaries. It even
outperforms human evaluators who have never watched any whole episode before.
Code is available at https://engindeniz.github.io/dialogsummary-videoqa

##  1 Introduction

Figure 1: In VideoQA, a question is associated with Scene B, but it can only
be answered by information from Scene A. We generate episode dialog summaries
from subtitles and give them as input to our VideoQA system, dispensing with
the need for external knowledge.

Deep learning has accelerated progress in vision and language tasks. _Visual-
semantic embeddings_ [18, 9] have allowed zero-shot learning, cross-modal
retrieval and generating new descriptions from embeddings. _Image captioning_
[33] and _visual question answering_(VQA) [2] have demonstrated generation of
realistic natural language description of images and a great extent of
multimodal semantic understanding. The extension to _video captioning_ [19,
32] and _video question answering_ (VideoQA) [29, 20] has enabled further
progress because video requires a higher level of reasoning to understand
complex events [37].

VideoQA systems typically have similar architecture focusing on multimodal
embeddings/description, temporal attention and localization, multimodal fusion
and reasoning. While it is often hard to isolate progress in individual
components, there are some clear trends. For instance, custom self-attention
and memory mechanisms for fusion and reasoning [24, 17, 7] are gradually being
streamlined by using _transformer_ architectures [30, 16, 36]; while visual
embeddings [29] are being replaced by semantic embeddings [20] and _text
descriptions_ by captioning [14, 3].

Datasets are essential for progress in the field, but often introduce bias.
For instance, questions from text summaries are less relevant to visual
information [29]; supervised temporal localization [20] biases system design
towards two-stage localization$\to$answering [21, 16]; fixed question
structure focusing on temporal localization [20] often results in mere
_alignment_ of questions with subtitles and _matching_ answers with the
discovered context [14], providing little progress on the main objective,
which is to study the level of understanding.

Bias can be removed by removing localization supervision and balancing
questions over different aspects of comprehension, for instance visual,
textual, or semantic [11]. However, the requirement of external knowledge,
which can be in the form of hints or even ground truth, does not leave much
progress in inferring such knowledge from raw data [11]. Even weakening this
requirement to plain text _human-generated summaries_ [10], still leaves a
system unusable in the absence of such data.

In many cases, as illustrated in Figure 1, a question on some part of a story
may require knowledge that can be recovered from dialog in other parts of the
story. However, despite being textual, raw dialog is often informal and
repetitive; searching over all available duration of such noisy source is
error-prone and impractical. Inspired by the trend of video captioning, we go
a step further and apply the same idea to _dialog_ : We _summarize_ raw
dialog, converting it into _text description_ for question answering.

Our finding is astounding: our dialog summary is not only a valid replacement
for human-generated summary in handling questions that require knowledge on a
whole story, but it outperforms them by a large margin.

Our contributions can be summarized as follows:

  1. 1.

We apply _dialog summarization_ to video question answering for the first time
(Subsection 5.1).

  2. 2.

Building on a modern VideoQA system, we convert all input sources into _plain
text description_.

  3. 3.

We introduce a weakly-supervised _soft temporal attention_ mechanism for
localization (Subsection 6.2).

  4. 4.

We devise a very simple _multimodal fusion_ mechanism that has no
hyperparameters (Section 7).

  5. 5.

We set a new state of the art on KnowIT VQA dataset [11] and we beat non-
expert humans for the first time, working only with raw data (Section 8).

##  2 Related Work

#### Video Question Answering

Progress on video question answering has been facilitated and driven by
several datasets and benchmarks. VideoQA by Tapaswi _et al_. [29] addresses
answering questions created from _plot synopses_ using a variety of input
sources, including video, subtitles, scene descriptions, scripts and the plot
synopses themselves. Methods experimenting on MovieQA focus on _memory
networks_ capturing information from the _whole movie_ by videos and subtitles
[24, 15], scene-based memory attention networks to learn joint representations
of frames and captions [17], and LSTM-based sequence encoders to learn visual-
text embeddings [23].

TVQA [20] and TVQA+ [21] address _scene-based_ questions containing _temporal
localization_ of the answer in TV shows, using video and subtitles. The
questions are structured in two parts: one specifying a temporal location in
the scene and the other requesting some information from that location. This
encourages working with more than one modalities. Methods experimenting on
these datasets focus on temporal localization and attention [21, 16],
_captioning_ [14, 3] and _transformer_ -based pipelines capturing visual-
semantic and language information [36, 30].

KnowIT VQA [11] is a _knowledge-based_ dataset, including questions related to
the scene, the episode or the entire story of a TV show, as well as _knowledge
annotation_ required to address certain questions, in the form of hints.
_Transformer_ -based methods are proposed to address this task by employing
knowledge annotation [11] or external human-generated _plot summaries_ [10].
Our method differs in substituting human-generated knowledge by summaries
automatically generated from raw dialog.

#### Dialog Summarization

Dial2Desc dataset [25] addresses generating _high-level short descriptions
from dialog_ using a transformer-based text generator. SAMSum corpus [12] is a
human-annotated dialog summarization dataset providing speaker information.
Methods experimenting on this dataset include existing _document
summarization_ methods [12], _graph neural networks_ integrating cross-
sentence information flow [39] and graph construction from utterance and
commonsense knowledge [8]. Since dialog differs from structured text and
requires extraction of the conversation structure, recent work focuses on
representing the dialog from different _views_ by sequence to sequence models
[4]. We follow this approach.

##  3 Overview

Figure 2: Our VideoQA system converts both video and dialog to text
descriptions/summaries, the latter at both scene and episode level. Converted
inputs are processed independently in streams, along with the question and
each answer, producing a score per answer. Finally, stream embeddings are
fused separately per answer and a prediction is made.

We address knowledge-based video question answering on TV shows. Each episode
is split in _scenes_. For each scene, we are given the _video_ (frames) and
_dialog_ (speaker names followed by subtitle text) and a number of _multiple-
choice questions_. Certain questions require high-level understanding of the
whole episode or show. Garcia _et al_. [10] rely on human-generated _plot
summaries_ (or _plot_ for short), which we use only for comparison. Our
objective is to extract the required knowledge from raw data.

As shown in Figure 2, we first convert inputs into _plain text description_ ,
including both video (by visual recognition) and dialog (by summarization)
(Section 5). A number of separate _streams_ then map text to embeddings, at
the level of both _scene_ (video and scene dialog summary) and _episode_
(episode dialog summary and plot). The question and answers are embedded
together with the input text of each stream. A _temporal attention_ mechanism
localizes relevant intervals from episode inputs. Finally, question answering
is addressed both in a _single-stream_ (Section 6) and a _multi-stream_
(Section 7) scenario. The latter amounts to _multi-modal fusion_. We begin our
discussion with _transformer_ networks (Section 4), which we use both for
dialog summarization and text embeddings in general.

##  4 Transformers

The _transformer_ [31] is a network architecture that allows for efficient
pairwise interaction between input elements. Its main component is an
_attention_ function, which acts as a form of associative memory.

_Multi-head attention_ is a fusion of several attention functions. The
architecture is a stack of multi-head attention, element-wise fully-connected
and normalization layers with residual connections. Originally developed for
machine translation, it includes an _encoder_ and a _decoder_ stack. The
decoder additionally attends over the output of the encoder stack and is
_auto-regressive_ , consuming previously generated symbols when generating the
next.

BERT [6] is a transformer bidirectional _encoder_ only, mapping a sequence of
tokens to a sequence of $d$-dimensional vectors. It is pre-trained on
unsupervised tasks including prediction of masked tokens and next sentence,
and can be also fine-tuned on supervised downstream tasks. It can take a
number of _sentences_ as in input, where a sentence is an arbitrary span of
contiguous text.

We use BERT as the backbone of our model architecture to represent text, using
two sentences at a time. Given strings $A$ and $B$, the input is given as

$\operatorname{tok}_{k}(\mathtt{[CLS]}+A+\mathtt{[SEP]}+B+\mathtt{[SEP]}),$
(1)

where $+$ is string concatenation and $\operatorname{tok}_{k}$ is tokenization
into $k$ tokens, with zero padding if the input length is less than $k$ and
truncation if it is greater. Tokens are represented by WordPiece embeddings
[28, 35], concatenated with _position embeddings_ representing their position
in the input sequence and _segment embeddings_ , where segments correspond to
sentences and are defined according to occurrences of the _separator_ token
$\mathtt{[SEP]}$. The output vector in d corresponding to token
$\mathtt{[CLS]}$ is an _aggregated representation_ of the entire input
sequence and we denote it as

$f(A,B).$ (2)

Sentence-BERT [26] takes a single sentence as input and is trained by _metric
learning_ objectives, _e.g_. in a siamese or triplet structure, facilitating
efficient sentence similarity search. It is learned by fine-tuning a pre-
trained BERT model on supervised semantic textual similarity.

BART [22] combines a bidirectional _encoder_ and an auto-regressive _decoder_.
It is pre-trained as an unsupervised denoising autoencoder, _i.e_., corrupting
input text and learning to reconstruct the original, and fine-tuned on
supervised classification, generation or translation tasks. It is particularly
effective on _text generation_ , including abstractive dialog, question
answering and summarization tasks.

Following [4], we use sentence-BERT and BART to _segment_ and _summarize
dialog_ respectively.

##  5 Input description

All input sources, _i.e_., _video_ , _dialog_ and _plot_ , are converted into
_plain text description_ before being used for question answering. Video is
first converted into a _scene graph_ by a visual recognition pipeline and then
to text description by a set of rules. Importantly, although already in
textual form, dialog is also converted into text description by _dialog
summarization_. The plot, already in text description form, is used as is, but
for comparison only: Our main contribution is to replace human-generated plots
by automatically generated descriptions.

###  5.1 Dialog

As the main form of human communication, dialog is an essential input source
for video understanding and question answering. We use dialog in three ways:
_raw dialog_ per scene, _dialog summary_ per scene and the collection of
dialog summary over a whole _episode_.

#### Raw scene dialog

As in all prior work, we use the raw dialog associated to the scene of the
question, _as is_. Although in textual form, it is _not_ a text description.
It may still contain more information than dialog summary, which is important
to investigate.

#### Scene dialog summary

Given the dialog associated to the scene of the question, we convert this
input source into text description by _dialog summarization_. Despite being of
textual form, dialog is very different from text _description_ : conversations
are often informal, verbose and repetitive, with few utterances being
informative; while a description is a narrative in _third-person_ point of
view with clear information flow structured in paragraphs [4]. Identifying the
speaking person is also substantial, especially with multiple people in a
conversation. Rather than generic document summarization [12], we follow a
dedicated dialog summarization method [4], which blends character names with
events in the generated summaries.

A dialog is a sequence of _utterances_ , each including a _speaker_
(character) name and a _sentence_ (sequence of tokens). Each utterance is
mapped to a vector embedding by Sentence-BERT [26]. The sequence of embeddings
over the entire dialog is segmented according to _topic_ , _e.g_. _greetings_
, _today’s plan_ , _etc_. by C99 [5], as well as _stage_ , _e.g_. _opening_ ,
_intention_ , _discussion_ , _conclusion_ by a _hidden Markov model_ (HMM)
[1]. As a result, for each _view_ (topic or stage), the dialog is represented
by a sequence of _blocks_ , each containing several utterances.

Given the above structure, the input is re-embedded and the summary is
generated using an extension of BART [22]. In particular, there is one
_encoder_ per view, mapping each block to an embedding. An LSTM [13] follows,
aggregating the entire view into one embedding, obtained as its last hidden
state. The _decoder_ attends over the output of each encoder using a _multi-
view attention_ layer to weight the contribution of each view. It is _auto-
regressive_ , using previous tokens from ground truth at training and
previously predicted tokens by the encoder at inference.

We train the HMM on the dialog sources of our video QA training set;
otherwise, we use Sentence-BERT and BART as used/trained by [4]. Once a scene
dialog summary is generated, it is re-embedded by BERT [6] like all other
input sources, as discussed in Section 6.

#### Episode dialog summary

We collect the scene dialog summaries for all scenes of an episode and we
concatenate them into an _episode dialog summary_. Assuming that the episode
of the scene of the question is known, we make available the associated
episode dialog summary for question answering. This is a long input source and
requires _temporal attention_ , as discussed in Subsection 6.2. Importantly,
episode dialog summary is our most important contribution in substituting plot
summary by an automatically generated description.

###  5.2 Plot summary

As part of our comparison to [10], we use publicly available plot
summaries111https://the-big-bang-theory.com/, already in text description
form. Assuming that the episode of the scene of the question is known, we make
available the associated plot _as is_ , to help answering _knowledge-based
questions_. A plot is shorter and higher-level than our episode dialog
summary, but it is still long enough to require _temporal attention_. It is
important to investigate whether we can dispense of such a human-generated
input and how much more information it contains relative to what we can
extract automatically.

###  5.3 Video

We use a visual recognition pipeline to convert raw input video into text
description. Following [10], this pipeline comprises four components:
_character recognition_ [27], _place recognition_ [40], _object relation
detection_ [38], and _action recognition_ [34]. The outputs of these
components are character, place, object, relation and action _nodes_. A
directed _video scene graph_ is generated by collecting all nodes along with
edges and then a textual _scene description_ is obtained according to a set of
predefined rules.

##  6 Single-stream QA

As shown in Figure 2, there is one stream per input source, using a
transformer to map inputs to embeddings. Following [10], we first attempt
question answering on each stream alone. In doing so, we learn a linear
classifier while fine-tuning the entire transformer representation per stream.
Unlike most existing works, this allows adapting to the data at hand, for
instance a particular TV show.

We differentiate _scene_ from _episode_ inputs, as discussed below. In both
cases, the given question and candidate answer strings are denoted as $q$ and
$a^{c}$ for $c=1,\dots,n_{c}$ respectively, where $n_{c}$ is the number of
candidate answers.

###  6.1 Scene input sources

Scene input sources refer to the scene of the question, _i.e_., _raw scene
dialog_ , _scene dialog summary_ or _video_. The input string is denoted by
$x$. For each $c=1,\dots,n_{c}$, we embed $x$, $q$ and $a^{c}$ jointly to
$d$-dimensional vector

$y^{c}\mathrel{:=}f(x+q,a^{c}),$ (3)

where $+$ is string concatenation and $f$ is BERT (2). A linear classifier
with parameters $\mathbf{w}\in^{d}$, $b\in$ yields a score per candidate
answer

$z^{c}\mathrel{:=}\mathbf{w}^{\top}\cdot y^{c}+b.$ (4)

The _score vector_ $z\mathrel{:=}(z^{1},...,z^{n_{c}})$ is followed by softmax
and cross-entropy loss. At training, we use $f$ as pre-trained and we fine-
tune it while optimizing $W,b$ on the correct answers of the QA training set.
At inference, we predict $\arg\max_{c}z^{c}$.

###  6.2 Episode input sources

Episode input sources refer to the entire episode of the scene of the
question, _i.e_., _episode dialog summary_ and _plot_. Because such input is
typically longer than the transformer’s maximum sequence length $k$ (1), we
split it into overlapping parts in a _sliding window_ fashion. Each part
contains the question and one answer, so the window length is
$w=k-|q|-|a^{c}|$. Given an input of length $\ell$ tokens, the number of parts
is $n\mathrel{:=}\left\lceil{\frac{\ell-w}{s}}\right\rceil+1$, where $s$ is
the _stride_. Because all inputs in a mini-batch must have the same number of
parts $n_{p}$ to be stacked in a tensor, certain parts are zero-padded if
$n<n_{p}$ and discarded if $n>n_{p}$.

#### Embedding

The input strings of the parts are denoted by $p_{j}$ for $j=1,\dots,n_{p}$.
Each part $p_{j}$ is combined with each candidate answer $a^{c}$ separately,
yielding the $d$-dimensional vectors

$y_{j}^{c}\mathrel{:=}f(p_{j}+q,a^{c})$ (5)

for $c=1,\dots,n_{c}$ and $j=1,\dots,n_{p}$. A classifier with parameters
$\mathbf{w}\in^{d}$, $b\in$ yields a score per candidate answer $c$ and part
$j$:

$z_{j}^{c}\mathrel{:=}\mathbf{w}^{\top}\cdot y_{j}^{c}+b.$ (6)

#### Temporal attention

At this point, unlike scene inputs (4), predictions from (6) are not
meaningful unless a part $j$ is known, which amounts to _temporal
localization_ of the part of the input sequence that contains the information
needed to answer a question. In TVQA [20] and related work [21, 14, 16],
localization ground truth is available, allowing a two-stage localize-then-
answer approach. Without such information, the problem is _weakly supervised_.

Previous work [10] simply chooses the part $j$ corresponding to the maximum
score $z_{j}^{c}$ over all answers $c$ and all parts $j$ in (6), which is
called _hard temporal attention_ in the following. Such hard decision may be
harmful when the chosen $j$ is incorrect, especially when the predicted answer
happens to be correct, because then the model may receive arbitrary gradient
signals at training. To alleviate this, we follow a _soft temporal attention_
approach.

In particular, let $S$ be the $n_{p}\times n_{c}$ matrix with elements
$z_{j}^{c}$ over all answers $c$ and all parts $j$ (6). For each part $j$, we
take the maximum score over answers

$s_{j}\mathrel{:=}\max_{c}z_{j}^{c},$ (7)

giving rise to a vector $s\mathrel{:=}(s_{1},\dots,s_{n_{p}})$, containing a
single best score per part. Then, by soft assignment over the rows of
$S$—corresponding to parts—we obtain a score for each answer $c$, represented
by _score vector_ $z\in^{c}$:

$z\mathrel{:=}\operatorname{softmax}(s/T)^{\top}\cdot S,$ (8)

where $T$ is a temperature parameter. With this definition of $z$, we have a
single score vector and we proceed as in (4).

##  7 Multi-stream QA

Once a separate transformer has been fine-tuned separately for each stream, we
combine all streams into a single question answering classifier, which amounts
to multi-modal fusion. Here, we introduce two new simple solutions.

In both cases, we freeze all transformers and obtain $d$-dimensional
embeddings $y^{c}$ for each candidate answer $c$ and for each stream. For
scene inputs, $y^{c}$ is obtained directly from (3). Episode input streams
produce $n_{p}$ embeddings per answer. Temporal localization is thus required
for part selection, similar to single stream training. Again, _hard temporal
attention_ amounts to choosing the part with the highest score according to
(6): $y^{c}\mathrel{:=}y_{j^{*}}^{c}$ where
$j^{*}\mathrel{:=}\arg\max_{j}(z_{j}^{c})$ and $y_{j}^{c}$ is given by (5).
Instead, similar to (8), we follow _soft temporal attention_ :

$y^{c}\mathrel{:=}\operatorname{softmax}(s/T)^{\top}\cdot Y_{c}^{emb},$ (9)

where $Y_{c}^{emb}$ is a $n_{p}\times d$ matrix collecting the embeddings
$y_{j}^{c}$ (5) of all parts $j$. Finally, for each answer $c$, the embeddings
$y^{c}$ of all streams are stacked into a $n_{s}\times d$ _embedding matrix_
$Y_{c}$, where $n_{s}$ is the number of streams.

#### Multi-stream attention

The columns of $Y_{c}$ are embeddings of different streams. We weight them
according to weights $w_{c}\in^{n_{s}}$ obtained from $Y_{c}$ itself, using a
_multi-stream attention_ block, consisting of two fully connected layers
followed by softmax:

$Y_{c}^{\text{att}}=\operatorname{diag}(w_{c})\cdot Y_{c}.$ (10)

For each answer $c$, a fully connected layer maps the $d\times n_{s}$ matrix
$Y_{c}^{\text{att}}$ to a scalar score. All $n_{c}$ scores are followed by
softmax and cross-entropy loss, whereby the parameters of all layers are
jointly optimized.

#### Self-attention

Alternatively, $Y_{c}$ is mapped to $Y_{c}^{\text{att}}\in^{d\times n_{s}}$ by
a single _multi-head self-attention_ block, as in transformers [31]:

$Y_{c}^{\text{att}}=\operatorname{MultiHeadAttention}(Y_{c},Y_{c},Y_{c}).$
(11)

The remaining pipeline is the same as in the previous case.

##  8 Experiments

###  8.1 Experimental setup

#### Datasets

The KnowIT VQA [11] dataset contains 24,282 human-generated questions
associated to 12,087 scenes, each of duration 20 seconds, from 207 episodes of
_The Big Bang Theory_ TV show. Questions are of four types: _visual_ (22%),
_textual_ (12%), _temporal_ (4%) and _knowledge_ (62%). Question types are
only known for the test set. Knowledge questions require reasoning based on
knowledge from the episode or the entire TV show, which differs from other
video question answering datasets. Questions are multiple-choice with
$n_{c}=4$ answers per question and performance is measured by _accuracy_ , per
question type and overall.

#### Implementation details

For scene dialog summary generation, we set the minimum sequence length to 30
tokens and the maximum to 100 in the BART [22] model. With this setting,
episode dialog summaries are 2078 tokens long on average, while plot summaries
are 659 tokens long.

We fine-tune the BERT${}_{\text{BASE}}$ [6] uncased model with $N=12$
transformer blocks, $h=12$ self-attention heads and embedding dimension
$d=768$ for single-stream models. The maximum token length $k$ is $512$ for
scene, $200$ for plot and $300$ for episode dialog summary inputs. The stride
$s$ is $100$ for plot and $200$ for episode dialog summary. The maximum number
of parts $n_{p}$ is $10$ for both. The batch size is $8$ for all single-stream
models and $32$ for multi-stream. We use SGD with momentum $0.9$ scheduled
with initial learning rate $10^{-4}$ for multi-stream fusions. We use $h=1$
attention head, and $N=2$ stacks for self-attention and multi-stream self-
attention methods. The number of streams $n_{s}$ varies per experiment.

Method | Knowledge | Vis. | Text. | Temp. | Know. | All
---|---|---|---|---|---|---
Rookies [11] | – | 0.936 | 0.932 | 0.624 | 0.655 | 0.748
Masters [11] | ✓ | 0.961 | 0.936 | 0.857 | 0.867 | 0.896
ROCK${}_{\text{GT}}$ [11] | question GT | 0.747 | 0.819 | 0.756 | 0.708 | 0.731
ROLL${}_{\text{human}}$ [10] | question GT | 0.708 | 0.754 | 0.570 | 0.567 | 0.620
TVQA [20] | – | 0.612 | 0.645 | 0.547 | 0.466 | 0.522
ROCK${}_{\text{facial}}$ [11] | dataset GT | 0.654 | 0.688 | 0.628 | 0.646 | 0.652
ROLL [10] | plot | 0.718 | 0.739 | 0.640 | 0.713 | 0.715
Ours | – | 0.755 | 0.783 | 0.779 | 0.789 | 0.781
Ours${}_{\text{plot}}$ | plot | 0.749 | 0.783 | 0.721 | 0.783 | 0.773
Table 1: _State-of-the-art accuracy_ on KnowIT VQA. Ours uses the video and
scene dialog summary as well as the episode dialog summary that we generate
from the dialog of the entire episode. Ours${}_{\text{plot}}$ also uses human-
generated plot summaries, like [10]. TVQA uses an LSTM based encoder; all
other methods use BERT. Rookies and Masters are humans.

###  8.2 Quantitative results

Table 1 compares of our method with the state of the art. Rookies and Masters
are human evaluators: Masters have watched most of the show, whereas Rookies
have never watched an episode before [11]. TVQA [20] encodes visual features
and subtitles without considering knowledge information; its results are as
reported in [11]. ROCK [11] uses four visual representations (image, concepts,
facial, caption); ROCK${}_{\text{facial}}$ is one of its best results.
ROCK${}_{\text{GT}}$ [11] and ROLL${}_{\text{human}}$ [10] use the human
knowledge annotation provided by the dataset [11], while ROLL [10] uses human-
written plot summaries instead. Our method uses scene video and scene dialog
summary as well as the episode dialog summary that it automatically generates,
without any human annotation. Ours${}_{\text{plot}}$ additionally uses the
same plot as [10]. TVQA uses LSTM; all other methods are based on BERT.

Our method outperforms the best state of the art method (ROLL [10]) by 6.6%,
without any human annotation. By using additional human-generated plots, the
gain decreases to 5.8%. This indicates that our episode dialog summary
captures the required knowledge and removes the requirement of human-generated
input; in fact, human-generated input is harmful. On temporal and knowledge
questions in particular, we gain 13.9% and 7.6%, respectively, without any
human annotation. This implies that our automatically generated episode dialog
summary increases the understanding of the episode and helps answering all
types of questions. Despite ROLL${}_{\text{human}}$ [10] and
ROCK${}_{\text{GT}}$ [11] using ground-truth knowledge, we outperform them by
16.1% and 5.0%, respectively, without any human annotation. We also outperform
Rookies, presumably by having access to the dialog of the entire episode.
Comparing to Masters, there is still room for improvement.

###  8.3 Qualitative analysis

|
---|---
(a) Knowledge QA | (b) Textual QA
|
(c) Temporal QA | (d) Visual QA
Figure 3:  _Multi-stream attention visualization_. We highlight in blue the
part of the source text that is relevant to answering the question. The most
attended stream is episode dialog summary for (a), (b), (c) and video
description for (d).

Figure 3 visualizes the correct predictions of our method with stream
attention scores for different question types. In all examples, the model
receives three input sources, question/answers and attention scores over
inputs. Figure 3(a) shows a _knowledge_ question, answered based on episode
dialog summary, which has the highest attention score. As shown in Figure
3(b), a _textual_ question can be answered by using scene dialog summary, but
also by episode dialog summary, since the latter includes the former.
_Temporal_ questions can be answered from scene inputs such as scene dialog
summary or video description. According to attention scores, the question in
Figure 3(c) is answered by episode dialog summary, which includes the correct
answer. Finally, Figure 3(d) shows a _visual_ question answered by video
description.

Method | Input | Vis. | Text. | Temp. | Know. | All
---|---|---|---|---|---|---
ROLL [10] | D | 0.656 | 0.772 | 0.570 | 0.525 | 0.584
V | 0.629 | 0.424 | 0.558 | 0.514 | 0.530
P | 0.624 | 0.620 | 0.570 | 0.725 | 0.685
ROLL [10]$\dagger$ | D | 0.649 | 0.801 | 0.581 | 0.543 | 0.598
V | 0.625 | 0.431 | 0.512 | 0.541 | 0.546
P | 0.647 | 0.554 | 0.674 | 0.694 | 0.667
Ours | P | 0.666 | 0.623 | 0.593 | 0.735 | 0.702
S | 0.631 | 0.746 | 0.605 | 0.537 | 0.585
E | 0.676 | 0.750 | 0.779 | 0.785 | 0.756
Table 2: _Single-stream QA accuracy_ on KnowIT VQA. ROLL [10]: as reported;
[10]$\dagger$: our reproduction. Our model incorporates the scene dialog and
video streams of the latter as well as the plot, scene dialog summary and
episode dialog summary streams. Plot differs between [10]$\dagger$ and our
model by our temporal attention and other improvements (Table 4). D: dialog;
V: video; P: plot; S: scene dialog summary; E: episode dialog summary.

###  8.4 Ablation studies

#### Single-stream results

Table 2 shows our single-stream QA results. We reproduce [10] for dialog,
video, and plot inputs. We replace the plot stream by one using our new
temporal attention (Subsection 6.2) and other improvements (Table 4) and we
add two new sources automatically generated from dialog: scene dialog summary
and episode dialog summary. Due to the dataset having a majority of knowledge
questions, episode dialog summary and plot inputs have higher accuracy than
other input sources since they span an entire episode. Our episode dialog
summary helps in answering questions better than the plot [10], bringing an
accuracy improvement of 5.4%.

Method | Vis. | Text. | Temp. | Know. | All
---|---|---|---|---|---
Product | 0.743 | 0.659 | 0.756 | 0.751 | 0.739
Modality weighting [10] | 0.708 | 0.786 | 0.767 | 0.787 | 0.769
Self-attention | 0.759 | 0.764 | 0.767 | 0.777 | 0.771
Multi-stream attention | 0.755 | 0.783 | 0.779 | 0.789 | 0.781
Multi-stream self-attn. | 0.755 | 0.768 | 0.756 | 0.777 | 0.770
Table 3: _Multi-stream QA accuracy_ on KnowIT VQA, fusing video, scene dialog
summary and episode dialog summary input sources. All fusion methods use soft
temporal attention for localization of episode input sources. Top:
baseline/competitors. Bottom: ours.

#### Multi-stream results

We evaluate our two multi-stream QA methods introduced in Section 7, namely
_multi-stream attention_ and _self-attention_ , comparing them with the
following combinations/baselines/competitors:

  1. 1.

_Multi-stream self-attention_ : combination of multi-stream attention and
self-attention: the output of the latter is weighted by the former. The
remaining pipeline is the same as in multi-stream attention.

  2. 2.

_Product_ : Hadamard product on embeddings of all streams per answer, followed
by a linear classifier per answer. The remaining pipeline is the same.

  3. 3.

_Modality weighting_ [10]: a linear classifier (4) and loss function is used
as in single-stream QA but with transformers frozen for each stream
separately. The obtained scores by single-stream classifiers are combined by a
multi-stream classifier and another loss function applies. The overall loss
all is a linear combination with weight $\beta_{\omega}$ on the multi-stream
loss and $1-\beta_{\omega}$ uniformly distributed over single-stream losses.

Table 3 shows results for fusion of video, scene dialog summary and episode
dialog summary. For modality weighting, we set $\beta_{\omega}=0.7$ according
to the validation set. Our multi-stream attention outperforms other fusion
methods. Besides, it does not require tuning of modality weight hyperparameter
$\beta_{\omega}$ or selecting the number of heads and blocks for self-
attention. Unless specified, we use multi-stream attention for fusion by
default.

Method | Vis. | Text. | Temp. | Know. | All
---|---|---|---|---|---
ROLL [10]$\dagger$ | 0.722 | 0.703 | 0.709 | 0.697 | 0.704
$+$ Multi-stream attention | 0.724 | 0.721 | 0.721 | 0.691 | 0.703
$+$ More parts for plot | 0.722 | 0.703 | 0.651 | 0.717 | 0.714
$+$ New order of plot inputs | 0.730 | 0.710 | 0.686 | 0.712 | 0.715
$+$ Temporal attention | 0.734 | 0.725 | 0.663 | 0.724 | 0.724
$\pm$ Replacing P $\rightarrow$ E | 0.753 | 0.815 | 0.814 | 0.773 | 0.775
$\pm$ Replacing D $\rightarrow$ S | 0.755 | 0.783 | 0.779 | 0.789 | 0.781
Table 4: _Accuracy improvements over_ ROLL [10]. $\dagger$: our reproduction.
Each row adds a new improvement except the last two, where we replace streams.
P: plot; E: episode dialog summary; D: dialog; S: scene dialog summary.

#### Improvements over [10]

We reproduce ROLL [10] using official code by the authors and default
parameters. This is our baseline, shown in the first row of Table 4. Then, we
evaluate our improvements, adding them one at a time. First, we replace
modality weighting with _multi-stream attention_. Despite its simplicity, its
performance is on par, losing only 0.1%, while requiring no hyperparameter
tuning. Then, we increase the _number of parts_ of plot summaries from 5 to
10, eliminating information loss by truncation and bringing an accuracy
improvement of 1.1%. We change the _order of arguments_ of BERT for episode
input sources from $f(q,a^{c}+p_{j})$ to $f(p_{j}+q,a^{c})$ (5), which is
consistent with (3) and improves only slightly by 0.1%. Our new _temporal
attention_ mechanism improves accuracy by 0.9%. Replacing plot with episode
dialog summary, which is our main contribution, brings an improvement of 5.1%.
Finally, the accuracy is improved by 0.6% by using _scene dialog summary_
instead of raw dialog. The overall gain over [10] is 7.7%.

Note that the relative improvement of each new idea depends on the order
chosen in Table 4. For instance, the order of BERT arguments brings
improvements of up to 2.3% in experiments including the episode dialog
summary.

##  9 Conclusion

KnowIT VQA is a challenging dataset where it was previously believed that some
form of external knowledge was needed to handle knowledge questions, as if
knowledge was yet another modality. Our results indicate that much of this
required knowledge was hiding in _dialog_ , waiting to be harnessed. It is
also interesting that our _soft temporal attention_ helps a lot more with our
episode dialog summary than human plot summary, which may be due to the
episode dialog summary being longer. This may also explain the astounding
performance of episode dialog summary, despite its low overall quality: plot
summaries are of much higher quality but may be missing a lot of information.

#### Acknowledgements

This work was supported by the European Commission under European Horizon 2020
Programme, grant number 951911 - AI4Media. This work was granted access to the
HPC resources of IDRIS under the allocation 2020-AD011012263 made by GENCI.

## References

  * [1] Tim Althoff, Kevin Clark, and Jure Leskovec.  Large-scale analysis of counseling conversations: An application of natural language processing to mental health.  Trans. ACL, 4:463–476, 2016.
  * [2] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.  VQA: Visual question answering.  In Proc. ICCV, 2015.
  * [3] Aman Chadha, Gurneet Arora, and Navpreet Kaloty.  iPerceive: Applying common-sense reasoning to multi-modal dense video captioning and video question answering.  In Proc. WACV, 2021.
  * [4] Jiaao Chen and Diyi Yang.  Multi-view sequence-to-sequence models with conversational structure for abstractive dialogue summarization.  In Proc. EMNLP, 2020.
  * [5] Freddy Y. Y. Choi.  Advances in domain independent linear text segmentation.  In Proc. NAACL, 2000.
  * [6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proc. NAACL, 2019.
  * [7] Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang.  Heterogeneous memory enhanced multimodal attention model for video question answering.  In Proc. CVPR, 2019.
  * [8] Xiachong Feng, Xiaocheng Feng, Bing Qin, and Ting Liu.  Incorporating commonsense knowledge into abstractive dialogue summarization via heterogeneous graph networks.  arXiv preprint arXiv:2010.10044, 2020.
  * [9] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc’Aurelio Ranzato, and Tomas Mikolov.  DeViSE: A deep visual-semantic embedding model.  In Proc. NIPS, 2013.
  * [10] Noa Garcia and Yuta Nakashima.  Knowledge-based video question answering with unsupervised scene descriptions.  In Proc. ECCV, 2020.
  * [11] Noa Garcia, Mayu Otani, Chenhui Chu, and Yuta Nakashima.  KnowIT VQA: Answering knowledge-based questions about videos.  In Proc. AAAI, 2020.
  * [12] Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer.  SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization.  In Proceedings of the 2nd Workshop on New Frontiers in Summarization. ACL, 2019.
  * [13] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural Computation, 9(8):1735–1780, 1997.
  * [14] Hyounghun Kim, Zineng Tang, and Mohit Bansal.  Dense-caption matching and frame-selection gating for temporal localization in VideoQA.  In Proc. ACL, 2020.
  * [15] Junyeong Kim, Minuk Ma, Kyungsu Kim, Sungjin Kim, and Chang D Yoo.  Progressive attention memory network for movie story question answering.  In Proc. CVPR, 2019.
  * [16] Junyeong Kim, Minuk Ma, Trung Pham, Kyungsu Kim, and Chang D. Yoo.  Modality shifting attention network for multi-modal video question answering.  In Proc. CVPR, 2020.
  * [17] Kyung-Min Kim, Seong-Ho Choi, Jin-Hwa Kim, and Byoung-Tak Zhang.  Multimodal dual attention memory for video story question answering.  In Proc. ECCV, 2018.
  * [18] Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel.  Unifying visual-semantic embeddings with multimodal neural language models.  arXiv preprint arXiv:1411.2539, 2014.
  * [19] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles.  Dense-captioning events in videos.  In Proc. ICCV, 2017.
  * [20] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg.  TVQA: Localized, compositional video question answering.  In Proc. EMNLP, 2018.
  * [21] Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal.  TVQA+: Spatio-temporal grounding for video question answering.  In Proc. ACL, 2019.
  * [22] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In Proc. ACL, 2020.
  * [23] Junwei Liang, Lu Jiang, Liangliang Cao, Li-Jia Li, and Alexander G Hauptmann.  Focal visual-text attention for visual question answering.  In Proc. CVPR, 2018.
  * [24] Seil Na, Sangho Lee, Jisung Kim, and Gunhee Kim.  A read-write memory network for movie story understanding.  In Proc. ICCV, 2017.
  * [25] Haojie Pan, Junpei Zhou, Zhou Zhao, Yan Liu, Deng Cai, and Min Yang.  Dial2desc: end-to-end dialogue description generation.  arXiv preprint arXiv:1811.00185, 2018.
  * [26] Nils Reimers and Iryna Gurevych.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In Proc. EMNLP-IJCNLP, 2019.
  * [27] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  FaceNet: A unified embedding for face recognition and clustering.  In Proc. CVPR, 2015.
  * [28] Mike Schuster and Kaisuke Nakajima.  Japanese and korean voice search.  In Proc. ICASSP, 2012.
  * [29] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler.  MovieQA: Understanding stories in movies through question-answering.  In Proc. CVPR, 2016.
  * [30] Aisha Urooj, Amir Mazaheri, Niels Da Vitoria Lobo, and Mubarak Shah.  MMFT-BERT: Multimodal fusion transformer with bert encodings for visual question answering.  In Proc. EMNLP, 2020.
  * [31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Proc. NIPS, 2017.
  * [32] Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko.  Sequence to sequence-video to text.  In Proc. ICCV, 2015.
  * [33] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.  Show and tell: A neural image caption generator.  In Proc. CVPR, 2015.
  * [34] Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick.  Long-term feature banks for detailed video understanding.  In Proc. CVPR, 2019.
  * [35] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  arXiv preprint arXiv:1609.08144, 2016.
  * [36] Zekun Yang, Noa Garcia, Chenhui Chu, Mayu Otani, Yuta Nakashima, and Haruo Takemura.  BERT representations for video question answering.  In Proc. WACV, 2020.
  * [37] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  From recognition to cognition: Visual commonsense reasoning.  In Proc. CVPR, 2019.
  * [38] Ji Zhang, Yannis Kalantidis, Marcus Rohrbach, Manohar Paluri, Ahmed Elgammal, and Mohamed Elhoseiny.  Large-scale visual relationship understanding.  In Proc. AAAI, 2019.
  * [39] Lulu Zhao, Weiran Xu, and Jun Guo.  Improving abstractive dialogue summarization with graph structures and topic words.  In Proc. COLING, 2020.
  * [40] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  IEEE Trans. PAMI, 40(6):1452–1464, 2017.

## Appendix

##  Appendix A Additional qualitative analysis

#### Dialog summarization

In the example of Figure 4, Howard says, “I invited _her_.” in scene B. Our
dialog summarization interprets this sentence by assigning the correct
character name: “Howard invited _Bernadette_ in.” Hence, we can answer the
question of scene A, “Who did Howard invite to join him and Raj in Raj’s lab?”
correctly. Thanks to the episode dialog summary spanning all scenes and the
use of character names instead of pronouns, our method can answer character-
related questions correctly.

Figure 4: Dialog summarization converts pronouns in dialog to character names
in episode dialog summary, supporting question answering. In particular, “I”
is substituted by “Howard” and “her” by “Bernadette”.

#### Plot _vs_. episode dialog summary

A comparison of plot summary and episode dialog summary is given in Figure 5.
There are three different topics in the story line, and each is highlighted
with the same color in both summaries. The first topic, highlighted in purple,
is “Sheldon’s forgotten flash drive.” The second, highlighted in yellow, is
“Sheldon’s grandmother.” The third, highlighted in blue, is “Asking Summer
out.” The plot summary is topic-centered, while the episode dialog summary is
following the narrative order. Hence, topics may be fragmented in the latter.
The episode dialog summary has more detail than the plot. In particular, it
contains enough information to answer the question _Why does Sheldon’s
grandmother call him Moon Pie?_ That is, _because he’s nummy-nummy_. This
information is missing from the plot summary, which focuses on the main
topics/events of an episode. Even though the episode dialog summary is noisy,
it contains details that help in question answering.

Figure 5: An example of plot summary and episode dialog summary, with each topic highlighted in the same color in both summaries. Phrases relevant to QA in blue. Only the episode dialog summary contains enough information to answer the question. |
---|---
(a) Knowledge QA (1) | (b) Textual QA (1)
|
(c) Knowledge QA (2) | (d) Textual QA (2)
|
(e) Temporal QA | (f) Visual QA
Figure 6: _Failed predictions of multi-stream attention_. We highlight in blue
the part of the source text that might be relevant to answering the question.
“Pred”/blue: model predictions. “GT”/green: ground truth.

#### Failure cases

Figure 6 shows examples of failed predictions of our model along with stream
attention scores for different question types. The model receives three input
sources, question/answers and attention scores over inputs.

Figure 6(a) refers to a knowledge question, which requires recurrent knowledge
of the whole TV show. In other words, the correct answer cannot be found in
episode dialog summary. The question is answered as ”a lasagna” found in
episode dialog summary, even though it is wrong.

Figure 6(b) refers to a textual question, which should have been answered by
scene dialog summary. However, scene dialog summary does not contain the
correct answer. Our model gives most attention to episode dialog summary. The
prediction is made according to the highlighted text, which is the same in
both sources. However, this prediction refers to the wrong person.

Figure 6(c) refers to another knowledge question, which could be answered by
the highlighted text in episode dialog summary. Even though episode dialog
summary has the most attention, the prediction is incorrect.

Figure 6(d) refers to another textual question, which should have been
answered by scene dialog summary. Although both scene dialog summary and
episode dialog summary include the correct answer, and episode dialog summary
has the most attention, the prediction indicates the wrong person.

Figure 6(e) refers to a temporal question. The scene dialog summary and
episode dialog summary imply that _Raj_ and _Howard_ might be changing the
tire. The video description is not helpful either. Hence, our model predicts
_Raj_ , while the correct answer is _Howard_.

Figure 6(f) is a visual question. However, the video description fails to
convey relevant information to answer the question. The other inputs do not
contain relevant information either. One of the character names appearing in
episode dialog summary is predicted, which is incorrect.

##  Appendix B Additional ablation studies

#### Hyperparameter validation

_Modality weighting_ [10] fusion method requires selection of hyperparameter
$\beta_{\omega}$. Figure 7 shows validation accuracy _vs_. $\beta_{\omega}$
for fusion of video, scene dialog summary and episode dialog summary. We
choose $\beta_{\omega}=0.7$ for both soft and hard temporal selection to
report results in Table 3 and Table 6. The remaining weight of
$1-\beta_{\omega}$ is evenly distributed over individual stream losses as
$0.1$ per stream.

0.20.40.60.810.7400.7450.7500.7550.760$\beta_{\omega}$validation accuracySoft
Attn.Hard Attn. Figure 7: Accuracy _vs_. $\beta_{\omega}$ for fusion of video,
scene dialog summary and episode dialog summary by modality weighting [10] on
KnowIT VQA validation set.

#### Effect of temporal attention on single-stream QA

We investigate the effect of our soft temporal attention (Subsection 6.2) on
single-stream QA for episode input sources. We also evaluate the effect of
single-stream training with soft or hard temporal attention on multi-stream
attention, where we use soft temporal attention. According to Table 5,
temporal attention improves the accuracy of plot and episode dialog summary by
1.9% and 3.3%, respectively. Accordingly, the accuracy of multi-stream QA on
the same episode sources as well as video and scene dialog summary increases
by 0.6% and 7.0%, respectively. The gain is higher when episode dialog summary
is used, since the episode dialog summary is longer than plot.

Stream | Inputs | Soft | Vis. | Text. | Temp. | Know. | All
---|---|---|---|---|---|---|---
Attn.
Single | P | - | 0.656 | 0.594 | 0.628 | 0.712 | 0.683
P | ✓ | 0.666 | 0.623 | 0.593 | 0.735 | 0.702
E | - | 0.604 | 0.721 | 0.733 | 0.765 | 0.723
E | ✓ | 0.676 | 0.750 | 0.779 | 0.785 | 0.756
Multi | V + S + P | - | 0.732 | 0.688 | 0.674 | 0.720 | 0.717
V + S + P | ✓ | 0.739 | 0.699 | 0.628 | 0.728 | 0.723
V + S + E | - | 0.707 | 0.772 | 0.721 | 0.700 | 0.711
V + S + E | ✓ | 0.755 | 0.783 | 0.779 | 0.789 | 0.781
Table 5:  _Effect of temporal attention on single-stream QA_ on KnowIT VQA.
Soft Attn.: soft temporal attention on single-stream training. We use soft
temporal attention for multi-stream QA, but this is still affected by the
temporal attention used in single-stream training. V: video; S: scene dialog
summary; P: plot; E: episode dialog summary.

#### Effect of temporal attention on multi-stream QA

Table 6 shows the effect of _soft temporal attention_ on multi-stream QA for
fusion of video, scene dialog summary and episode dialog summary input
sources. We use soft temporal attention for single-stream QA of episode dialog
summary. In all fusion methods, the overall accuracy is improved by using soft
temporal attention.

Method | Soft | Vis. | Text. | Temp. | Know. | All
---|---|---|---|---|---|---
Attn.
Product | - | 0.728 | 0.645 | 0.744 | 0.756 | 0.736
✓ | 0.743 | 0.659 | 0.756 | 0.751 | 0.739
Modality weighting | - | 0.716 | 0.815 | 0.791 | 0.776 | 0.768
[10] | ✓ | 0.708 | 0.786 | 0.767 | 0.787 | 0.769
Self-attention | - | 0.753 | 0.804 | 0.802 | 0.766 | 0.769
✓ | 0.759 | 0.764 | 0.767 | 0.777 | 0.771
Multi-steam attention | - | 0.743 | 0.790 | 0.779 | 0.785 | 0.776
✓ | 0.755 | 0.783 | 0.779 | 0.789 | 0.781
Multi-steam self attn. | - | 0.749 | 0.797 | 0.791 | 0.768 | 0.768
✓ | 0.755 | 0.768 | 0.756 | 0.777 | 0.770
Table 6: _Effect of temporal attention on multi-stream QA_ on KnowIT VQA for fusion of video, scene dialog summary, and episode dialog summary input sources. Soft Attn.: soft temporal attention on multi-stream training. We use soft temporal attention for single-stream QA of episode dialog summary. Analysed | Inputs | Vis. | Text. | Temp. | Know. | All
---|---|---|---|---|---|---
Inputs
D V P | D+V | 0.693 | 0.768 | 0.593 | 0.554 | 0.611
D+P | 0.732 | 0.721 | 0.674 | 0.723 | 0.723
D+V+P | 0.734 | 0.725 | 0.663 | 0.724 | 0.724
D V P S | D+S | 0.664 | 0.786 | 0.628 | 0.548 | 0.604
V+S | 0.689 | 0.721 | 0.581 | 0.549 | 0.601
P+S | 0.716 | 0.710 | 0.628 | 0.727 | 0.719
D+V+P+S | 0.734 | 0.732 | 0.663 | 0.725 | 0.726
D V P S E | D+E | 0.743 | 0.812 | 0.779 | 0.779 | 0.775
V+E | 0.732 | 0.761 | 0.767 | 0.788 | 0.772
P+E | 0.716 | 0.743 | 0.721 | 0.791 | 0.766
D+S+E | 0.743 | 0.822 | 0.802 | 0.771 | 0.772
V+S+E | 0.755 | 0.783 | 0.779 | 0.789 | 0.781
P+S+E | 0.739 | 0.779 | 0.733 | 0.783 | 0.771
D+V+P+S+E | 0.751 | 0.797 | 0.744 | 0.781 | 0.775
Table 7: _Multi-stream QA accuracy_ on KnowIT VQA: comparison of different
input combinations for multi-stream attention. D: dialog; V: video; P: plot;
S: scene dialog summary; E: episode dialog summary.

#### Different input combinations

Table 7 shows the accuracy of multi-stream QA for different input
combinations, where the number of input streams varies in $\\{2,3,4,5\\}$.
Scene dialog summaries improves the accuracy compared with single-stream QA
results in Table 2. Moreover, using the episode dialog summary always improves
the overall accuracy by a large margin. The best overall accuracy of 0.781 is
achieved by video, scene dialog summary, and episode dialog summary.

#### Question type $\leftrightarrow$ attention scores

We perform significance testing for the dependence between the question type
and attention scores. There are 2 independent variables in the scores of 3
streams, whose values we discretize into $10\times 10$ bins. We form a
$4\times 10\times 10$ joint histogram of question type ($X$) and scores ($Y$)
and compute the mutual information $I(X;Y)$. We perform a
$G$-test222https://en.wikipedia.org/wiki/G-test with $G=2N\cdot I(X;Y)$, where
$N=2361$ is the number of test questions. Finally, using a chi-square
distribution of $3\times 9\times 9$ DoF, we find a $p$-value of $1.52\times
10^{-25}$ for the null hypothesis. This indicates that attention scores depend
on question type.

#### Replacing attention scores with oracle scores determined by question type

Assuming that we know the question type for the test set, we perform an
_oracle_ experiment where attention scores are based on question type rather
than our fusion method. We only consider visual, textual, and knowledge types
of question. In particular, we assign visual questions to video input, textual
questions to scene dialog summary and knowledge questions to episode dialog
summary. We exclude temporal questions since they can be answerable by scene
dialog summary or video. Only 3.6% of questions are of temporal type in the
test set. We find that our multi-stream attention method (0.781%) is 3.6%
better than the oracle experiment (0.745%). This indicates that our fusion
mechanism is more effective than a naïve oracle that assumes more knowledge.