# Cross-Modal Reasoning with Event Correlation for Video Question Answering

Chengxiang Yin1, Zhengping Che2, Kun Wu1, Zhiyuan Xu2, Qinru Qiu1, Jian Tang2
1Syracuse University, 2Midea Group

###### Abstract

Video Question Answering (VideoQA) is an attractive and challenging research
direction aiming to understand complex semantics of heterogeneous data from
two domains, i.e., the spatio-temporal video content and the word sequence in
question. Although various attention mechanisms have been utilized to manage
contextualized representations by modeling intra- and inter-modal
relationships of the two modalities, one limitation of the predominant VideoQA
methods is the lack of reasoning with event correlation, that is, sensing and
analyzing relationships among abundant and informative events contained in the
video. In this paper, we introduce the dense caption modality as a new
auxiliary and distill event-correlated information from it to infer the
correct answer. To this end, we propose a novel end-to-end trainable model,
Event-Correlated Graph Neural Networks (EC-GNNs), to perform cross-modal
reasoning over information from the three modalities (i.e., caption, video,
and question). Besides the exploitation of a brand new modality, we employ
cross-modal reasoning modules for explicitly modeling inter-modal
relationships and aggregating relevant information across different
modalities, and we propose a question-guided self-adaptive multi-modal fusion
module to collect the question-oriented and event-correlated evidence through
multi-step reasoning. We evaluate our model on two widely-used benchmark
datasets and conduct an ablation study to justify the effectiveness of each
proposed component.

## Introduction

Video Question Answering (VideoQA) (Zhu et al. 2017; Zeng et al. 2017; Jang et
al. 2019; Fan et al. 2019; Jiang and Han 2020; Le et al. 2020) is a
challenging task. It can be seen as a proxy task for evaluating a vision-
language system’s capacity for a more profound understanding of heterogeneous
data from two domains, i.e., spatio-temporal video contents and word sequence
in question. A mature VideoQA agent is able to deliver the correct answer by
associating relevant visual contents in the video with the actual subject
queried in question. To achieve this, predominant VideoQA methods (Ye et al.
2017; Xu et al. 2017; Jang et al. 2017, 2019; Li et al. 2019) utilize various
attention mechanisms to find relevant details in spatial or/and temporal
dimensions. However, none of the existing approaches reasons with events and
their correlations for VideoQA. Temporally localized and occurring
concurrently or successively, events are numerous and ubiquitous in video and
imply massive knowledge of the video. For example, in Figure 1, a man in the
red shirt is playing the violin and then the man jumps off the ladder and
begins to walk on the side of the roof are two consecutive events depicting
the same person. Sensing such events and analyzing their relationships with
video contents and questions provides a promising way to deliver an accurate
and explainable answer for a given question.

Figure 1:  An illustration of how EC-GNNs handle VideoQA tasks. EC-GNNs aim to
associate semantic contents in the generated dense captions and relevant
visual contents in frame sequence with the actual subject queried in question.
For a complex question such as “What is the man playing while walking on the
roof?”, EC-GNNs first localize when the man is walking on the roof and then
focus on what the man is playing (i.e., the violin).

Events can be properly understood through modeling the complex video dynamics
into a natural language description, namely video captioning. Massive works
(Venugopalan et al. 2014, 2015; Yao et al. 2015; Pan et al. 2016) explain
video with a single sentence. For example, they would most likely to focus on
a man in the red shirt is playing the violin in Figure 1, which provides some
details about the man and his outfits but fails to articulate other events in
the video. Instead, incorporating dense video captions (Krishna et al. 2017;
Zhou et al. 2018; Wang, Zheng, and Yu 2020) as a new auxiliary modality helps
to identify and understand more events involved, which provides valuable clues
to infer correct answers.

With multiple modalities available in VideoQA, integrating intra- and inter-
modal relationships may further benefit the answer inference. Taking Figure 1
as an instance, the model needs to establish subject-predicate relationships
between the word man and the words playing and walking in the question. RNNs
are commonly adopted (Zhu et al. 2017; Zeng et al. 2017; Yin et al. 2019) but
often fail to model the true complexity of language structure especially for
semantic-complex questions. Alternatively, Graph Convolutional Networks (GCNs)
(Kipf and Welling 2016) provide a more flexible way to utilize context-aware
neural mechanisms to learn the complex intra-modal relationships, and the
requirement of intra-modal modeling and the efficacy of GCNs also hold for
videos and dense captions. One the other hand, the model needs to establish a
solid semantic relationship between the man playing in the question, the
caption the man in red shirt is playing the violin, and the few frames
highlighted in the middle of the video, to correctly answer the question in
Figure 1. Modeling only one-way interactions (Anderson et al. 2018; Teney et
al. 2018; Jang et al. 2017, 2019) is not enough, and methods aggregating
relevant information across different modalities and learning long-term
semantic dependencies is preferred. Self-attention (Vaswani et al. 2017) with
different cross-modal key-query pairs provides a promising way of modeling
inter-modal relationships.

At the final stage of VideoQA, to aggregate multi-modal features for answer
prediction, the predominant approaches (Zhu et al. 2017; Zeng et al. 2017; Yin
et al. 2019) generate a monolithic representation using simple vector
operations, including concatenation, element-wise addition and/or element-wise
multiplication. However, the monolithic representation cannot fully understand
cross-modal semantics and often lead to incorrect answers. As (Fan et al.
2019) found the sub-optimal performance may come from distraction from the
question, it is promising that multi-step reasoning with question guidance can
progressively gather the question-oriented and event-correlated evidences and
provide better answers.

In this paper, to deal with VideoQA, we propose Event-Correlated Graph Neural
Networks (EC-GNNs), a novel end-to-end trainable model, which performs
reasoning with event correlation, by exploiting the relationships among dense
events, video contents and question words. To the best of our knowledge, this
is the first work that explicitly incorporates dense video captions to perform
the reasoning of VideoQA. Specifically, we develop a dense caption modality
and handle the VideoQA task via three modality-aware graphs, where the dense
caption features, visual features, and question embedding features are kept in
the corresponding graphs. Three procedures are performed afterwards to infer
the final answer. First, graph reasoning modules capture the intra-relations
of each modality. Then, cross-modal reasoning modules aggregate relevant
information across different modalities by leveraging a cross-modal attention
mechanism to model the inter-modal relationships. Finally, after repeating the
above two procedures several times, a question-guided self-adaptive multi-
modal fusion module conducts multiple inference loops to collect the question-
oriented and event-related evidences and refines the final answer prediction.

The contributions are summarized as follows: 1) We propose a novel scheme to
perform cross-modal reasoning with event correlation over information from
three modalities (i.e., caption, video, and question). 2) We are the first to
introduce dense video captions for VideoQA and clarify how to incorporate the
event-correlated information into reasoning process. 3) We explicitly model
the inter-modal relationships through a cross-modal attention mechanism in
cross-modal reasoning. 4) We design a question-guided self-adaptive multi-
modal fusion module that adaptively collects question-oriented and event-
correlated evidence. 5) The proposed model outperforms most VideoQA methods
and performs on par with the state-of-the-art on two VideoQA benchmarks.

## Related Work

### Video Question Answering

Recently, significant progress has been made to video question answering
(VideoQA) (Zhu et al. 2017; Zeng et al. 2017; Jang et al. 2017; Ye et al.
2017; Xu et al. 2017; Fan et al. 2019; Jang et al. 2019; Jiang and Han 2020;
Le et al. 2020), which extends image question answering (ImageQA) (Antol et
al. 2015; Chen et al. 2015; Anderson et al. 2018; Teney et al. 2018; Ma et al.
2018; Yin et al. 2021) to the video domain and requires a more comprehensive
spatio-temporal understanding of heterogeneous data from multiple modalities.
As a common practice, most works on VideoQA utilized appearance features
and/or motion features to represent video frames and employ word embedding
(e.g., Word2vec (Mikolov et al. 2013) and Glove (Pennington, Socher, and
Manning 2014)) to represent question words. Representative methods (Zhu et al.
2017; Zeng et al. 2017; Yin et al. 2019) on VideoQA leveraged RNNs-based
encoders to encode frame features and word features into a single
representation. Additionally, a number of VideoQA works (Ye et al. 2017; Xu et
al. 2017; Jang et al. 2017, 2019; Li et al. 2019) employed temporal attention
(Bahdanau, Cho, and Bengio 2014) to attend to the most relevant frames in
video. More recently, some works (Gao et al. 2018; Kim et al. 2018; Yin et al.
2019; Fan et al. 2019; Kim et al. 2019) proposed the use of external memory
combined with a customized memory addressing mechanism to enhance the spatio-
temporal modeling of VideoQA. In this paper, we explore the VideoQA task
through new modality generation and cross-modal reasoning with event
correlation from three modalities.

### Video Captioning and Dense Video Captioning

Similar to VideoQA, the earliest video captioning works (Venugopalan et al.
2014, 2015; Cho, Courville, and Bengio 2015; Donahue et al. 2015; Pan et al.
2016) employed encoder-decoder architectures based on RNNs to model the
temporal dynamics of the frame sequences and word sequences. Later works (Yao
et al. 2015; Gao et al. 2017; Li et al. 2017) applied customized attention
mechanisms to selectively focus on salient frames in the video, resulting in
better performance. However, describing video semantics in one sentence often
fails to recognize or articulate the other events in the video, which brings
the demand of dense video captioning (Yu et al. 2016; Krishna et al. 2017;
Zhou et al. 2018; Wang et al. 2018, 2021). By extending video captions in the
event dimension, dense video captioning model first locates multiple events
occurring in the video and generates a caption for each of them. A seminal
work (Krishna et al. 2017) employed an existing proposal module (Escorcia et
al. 2016) to detect the event locations and caption each event by utilizing
the context from surrounding events. As dense events provide valuable clues to
infer the correct answer, we explicitly extract and incorporate dense video
captions as a new modality for VideoQA reasoning.

## Methodology

This section describes the architecture of the Event-Correlated Graph Neural
Networks (EC-GNNs) as shown in Figure 2. We first introduce the representation
(Sec. Contextual Representation with Generated Modality) and graph
construction (Sec. Graph Construction) of the three modalities (i.e., the
caption, the video, and the question). Then we demonstrate how the graph
reasoning modules (Sec. Graph Reasoning) and the cross-modal reasoning modules
(Sec. Cross-Modal Reasoning with Cross-Modal Attention Mechanism) can model
the intra- and inter-relationships. Finally, we elaborate on the design of the
question-guided self-adaptive multi-modal fusion module (Sec. Question-Guided
Self-Adaptive Multi-Modal Fusion) and the answer prediction (Sec. Answer
Prediction).

Figure 2: The architecture of Event-Correlated Graph Neural Networks (EC-
GNNs), which consists of three components: graph reasoning modules, cross-
modal reasoning modules, and a question-guided self-adaptive multi-modal
fusion module. Caption features, visual features, and word features are
forwarded to caption graph, video graph, and word graph respectively. Each of
the graphs contains three graph reasoning procedures where two cross-modal
reasoning procedures are interleaved. Finally, multi-modal fusion is performed
to infer the answer. Green, blue, and red arrows between modules corresponds
to the information flow of caption, video, and question modalities
respectively.

### Contextual Representation with Generated Modality

Generating caption modality. Dense video captions are a set of captions in
descriptive natural language, which describe multiple events temporally
localized in a video. To generate dense captions of a video, we first extract
its appearance features and optical flow features using ResNet-200 (He et al.
2016) and BN-Inception (Ioffe and Szegedy 2015) respectively. Given extracted
features, we employ the dense video captioning model (Zhou et al. 2018)
trained on both ActivityNet Captions (Krishna et al. 2017) and YouCookII
(Zhou, Xu, and Corso 2018) datasets to obtain the corresponding dense
captions. Each caption is first represented as a sequence of word embeddings
based on GloVe 300-D (Pennington, Socher, and Manning 2014), and then encoded
with a trainable GRU (Chung et al. 2014) to obtain its final hidden state as a
monolithic representation. Therefore, the dense captions are denoted as a
sequence of monolithic representations
${\textbf{F}}^{c}=\\{{\textbf{f}}_{i}^{c}:i\leq
N_{c},{\textbf{f}}_{i}^{c}\in{\mathcal{R}}^{d_{c}}\\}$, where $N_{c}$
represents the number of captions and $d_{c}$ is the dimensionality of the
caption features. In Figure 2, we highlight the caption features in green. In
order to match the words in captions with the words in questions, same
dictionary are used when dealing with dense captions and questions.

Representing video and question modalities. Besides the generated caption
modality, we also model directly the video and question modalities. We extract
frame-level appearance features
${\textbf{F}}^{a}=\\{{\textbf{f}}_{i}^{a}:i\leq
N_{v},{\textbf{f}}_{i}^{a}\in{\mathcal{R}}^{d_{a}}\\}$, shot-level motion
features ${\textbf{F}}^{m}=\\{{\textbf{f}}_{i}^{m}:i\leq
N_{v},{\textbf{f}}_{i}^{m}\in{\mathcal{R}}^{d_{m}}\\}$, and common visual
features ${\textbf{F}}^{v}=\\{{\textbf{f}}_{i}^{v}:i\leq
N_{v},{\textbf{f}}_{i}^{v}\in{\mathcal{R}}^{d_{v}}\\}$ from videos, as well as
a sequence of word embeddings ${\textbf{F}}^{q}=\\{{\textbf{f}}_{i}^{q}:i\leq
N_{q},{\textbf{f}}_{i}^{q}\in{\mathcal{R}}^{d_{q}}\\}$ from questions. Here,
$N_{v}$ is the number of frames in the video, $N_{q}$ is the number of words
in the question, and $d_{a}$, $d_{m}$, $d_{v}$, and $d_{q}$ are
dimensionalities of the feature views. In Figure 2, we highlight the
appearance/motion/visual features in light/normal/dark blue and the word
features in red. More details are provided in Appendix.

Obtaining contextualized representations. All the features of the three
modalities (i.e., caption features, visual features, and word features) are
time series. To exploit the dynamic temporal information and obtain contextual
representations for each modality, we leverage three independent GRUs to
encode these features separately.

$\displaystyle{\textbf{C}}^{1},{\textbf{c}}_{N_{c}}$
$\displaystyle=\mbox{GRU}({\textbf{F}}^{c};{\bm{\theta}}_{GRU}^{c})$ (1)
$\displaystyle{\textbf{V}}^{1},{\textbf{v}}_{N_{v}}$
$\displaystyle=\mbox{GRU}({\textbf{F}}^{v};{\bm{\theta}}_{GRU}^{v})$ (2)
$\displaystyle{\textbf{Q}}^{1},{\textbf{q}}_{N_{q}}$
$\displaystyle=\mbox{GRU}({\textbf{F}}^{q};{\bm{\theta}}_{GRU}^{q})$ (3)

The contextualized caption features are denoted as
${\textbf{C}}^{1}=\\{{\textbf{C}}^{1}_{i}:i\leq
N_{c},{\textbf{C}}^{1}_{i}\in{\mathcal{R}}^{d_{C}}\\}$, the contextualized
visual features are denoted as ${\textbf{V}}^{1}=\\{{\textbf{V}}^{1}_{i}:i\leq
N_{v},{\textbf{V}}^{1}_{i}\in{\mathcal{R}}^{d_{V}}\\}$, and the contextualized
word features are denoted as ${\textbf{Q}}^{1}=\\{{\textbf{Q}}^{1}_{i}:i\leq
N_{q},{\textbf{Q}}^{1}_{i}\in{\mathcal{R}}^{d_{Q}}\\}$, where ${d_{C}}$,
${d_{V}}$, and ${d_{Q}}$ are dimensionalities of the contextualized features;
${\textbf{c}}_{N_{c}}\in{\mathcal{R}}^{d_{C}}$,
${\textbf{v}}_{N_{v}}\in{\mathcal{R}}^{d_{V}}$ and
${\textbf{q}}_{N_{q}}\in{\mathcal{R}}^{d_{Q}}$ are the last hidden states,
which represent the global features of the three modalities.

### Graph Construction

Given the contextualized features of three modalities, we construct three
modality-aware graphs: caption graph, video graph, and question graph. As
shown in Figure 2, each modality-aware graph is a three-layer fully-connected
GNNs (i.e., $l=1,2,3$), where the caption features ${\textbf{C}}^{l}$, visual
features ${\textbf{V}}^{l}$, and word features ${\textbf{Q}}^{l}$ are kept in
the caption, video, question graphs, with green, blue and red nodes,
respectively. The numbers of nodes in each layer of the three graphs are
$N_{c}$, $N_{v}$, and $N_{q}$, respectively, which equals to the numbers of
dense captions, frames, and words in questions.

### Graph Reasoning

As each layer of the modality-aware GCNs contains modality-specific knowledge,
we first capture the intra-modal relationships from each graph by performing
graph reasoning independently. Given the node features ${\textbf{C}}^{l}$ (or
${\textbf{V}}^{l}$, ${\textbf{Q}}^{l}$), we measure the semantic correlations
among nodes (Wang and Gupta 2018) and obtain the normalized adjacency matrix
${\textbf{G}}_{C}^{l}\in{\mathcal{R}}^{N_{c}\times N_{c}}$ (or
${\textbf{G}}_{V}^{l}\in{\mathcal{R}}^{N_{v}\times N_{v}}$,
${\textbf{G}}_{Q}^{l}\in{\mathcal{R}}^{N_{q}\times N_{q}}$), and we apply GCNs
(Kipf and Welling 2016) to update each node based on its neighbors and the
corresponding weights. Detailed intra-modal graph reasoning procedures are
described in Appendix. By doing so, we obtain the updated node features
${\hat{\textbf{C}}}^{l}$, ${\hat{\textbf{V}}}^{l}$, and
${\hat{\textbf{Q}}}^{l}$ accordingly, which are forwarded to cross-modal
reasoning ($l=1,2$) and multi-modal fusion ($l=3$) respectively.

Figure 3: An illustration of the cross-modal reasoning module. For simplicity,
the cross-modal reasoning module in video graph between layer $l$ and $l+1$ is
presented, and the query node ${\textbf{V}}_{i}^{l}$ is taken as an example
for demonstration. Green, blue, and red arrows between modules corresponds to
the information flow of caption, video, and question modalities respectively.
Dotted arrows represent the transmission of node features. Different shades of
color indicate representations of different nodes. FF and LN denote feed-
forward network and layer normalization.

### Cross-Modal Reasoning with Cross-Modal Attention Mechanism

A video footage tends to focus on part of the question rather than the whole,
and tends to pay more attention on a few captions instead of assigning equal
weight to all captions. Same rule applies for a word in the question and a
caption in the set of dense captions. Therefore, to infer the correct answer,
we need to fully understand the dense interactions between factors of
different modalities.

As shown in Figure 2, between two intra-modal graph reasoning procedures, we
propose a cross-modal attention mechanism (CAM) in cross-module reasoning
procedure to explicitly exploit the inter-modal relationships with two other
modalities. For instance, in the cross-modal reasoning of video graph, CAM
allows a factor from the video modality as a clue to determine the weights of
factors from question and caption modalities to aggregate the relevant
information.

Given a query and a set of key-value pairs, CAM computes the weighted sum of
values based on the dot-product similarity of the query and keys. With query,
key and value denoted as a set of vectors (i.e., ${\mathcal{M}}_{Q}$,
${\mathcal{M}}_{K}$, and ${\mathcal{M}}_{V}$), we formulate the cross-modal
attention mechanism as

$\displaystyle\mbox{CAM}({\mathcal{M}}_{Q},{\mathcal{M}}_{K},{\mathcal{M}}_{V})$
$\displaystyle=\mbox{softmax}(\frac{{\mathcal{M}}_{Q}{{\mathcal{M}}_{K}}^{T}}{\sqrt{d}}){\mathcal{M}}_{V}$
(4)

in which ${\mathcal{M}}_{K}$ and ${\mathcal{M}}_{V}$ belong to the same
modality which is different from that of ${\mathcal{M}}_{Q}$, and $d$
indicates the dimensionality of the vectors.

With CAM, which is designed to better embed and capture information across
multiple modalities, as the core module, we build the cross-modal reasoning
procedure, which is illustrated in Figure 3 and elaborated in Appendix. In
brief, the cross-modal reasoning is performed on caption graph, video graph,
and question graph with the input node features ${\hat{\textbf{C}}}^{l}$,
${\hat{\textbf{V}}}^{l}$, and ${\hat{\textbf{Q}}}^{l}$ from $l$-th graph
reasoning module. Then, it produces a set of CAM-based features for the three
modalities and further obtains the updated node features
${\textbf{C}}^{l+1}=\\{{\textbf{C}}^{l+1}_{i}:i\leq
N_{c},{\textbf{C}}^{l+1}_{i}\in{\mathcal{R}}^{d_{C}}\\}$,
${\textbf{V}}^{l+1}=\\{{\textbf{V}}^{l+1}_{i}:i\leq
N_{v},{\textbf{V}}^{l+1}_{i}\in{\mathcal{R}}^{d_{V}}\\}$, and
${\textbf{Q}}^{l+1}=\\{{\textbf{Q}}^{l+1}_{i}:i\leq
N_{q},{\textbf{Q}}^{l+1}_{i}\in{\mathcal{R}}^{d_{Q}}\\}$, which are forwarded
to the next GNN layer for the next round of intra-modal graph reasoning.

Figure 4: An illustration of the question-guided self-adaptive multi-modal
fusion module at the $t$-th step. An LSTM controller with previous state
${\textbf{h}}_{t-1}$ attends to relevant caption features, visual features,
and word features with question guidance, and combines them to obtain current
state ${\textbf{h}}_{t}$.

### Question-Guided Self-Adaptive Multi-Modal Fusion

In multi-modal fusion procedure, to collect the question-oriented and event-
correlated evidence from the three modalities, we design a question-guided
self-adaptive multi-modal fusion module, which performs multiple cycles of
reasoning to selectively attend to multi-modal features through gradually
refining the soft attention weights with the guidance of question. Inspired by
the multi-modal fusion module proposed by HME (Fan et al. 2019), we adopts
multi-step reasoning with an LSTM controller.

The multi-modal fusion module takes the updated node features from the last
GNN layer (i.e., caption features ${\hat{\textbf{C}}}^{3}$, visual features
${\hat{\textbf{V}}}^{3}$, and word features ${\hat{\textbf{Q}}}^{3}$) and the
last hidden state of question (i.e., ${\textbf{q}}_{N_{q}}$) as the inputs.
The module deployed at each reasoning step is illustrated in Figure 4. At each
reasoning step $t$, the module employs temporal attention mechanism to attend
to different parts of the caption, visual, and word features independently,
and collects the question-oriented information from each modality. The
attended features (i.e., $\tilde{\textbf{C}}$, $\tilde{\textbf{V}}$, and
$\tilde{\textbf{Q}}$) are further incorporated together with learned modality
weights to gather the event-correlated information across the modalities and
produce representation ${\textbf{h}}_{t}$ of the current state, which is
forwarded to the reasoning of next step.

By iteratively performing the reasoning step $N_{r}$ times, we obtain the
final representation of the multi-modal fusion ${\textbf{h}}_{N_{r}}$.
Finally, we apply standard temporal attention (Jang et al. 2017) on the
contextualized caption features ${\textbf{C}}^{1}$ and visual features
${\textbf{V}}^{1}$, and concatenate with ${\textbf{h}}_{N_{r}}$ to obtain the
final representation ${\textbf{s}}_{a}$ for answer prediction. The computation
procedure and formulas in detail are available in Appendix.

### Answer Prediction

Based on the final representation ${\textbf{s}}_{a}$ from the fusion module,
we predict the final answer of the VideoQA tasks. We aim at solving three
different types of VideoQA, same as (Jang et al. 2017): open-ended words,
open-ended numbers and multiple-choice. Given the input video $v$ and the
corresponding question $q$, the open-ended tasks is to infer an answer
$\hat{a}$ from a pre-defined answer set $\mathcal{A}$ of size $C$ that matches
the ground-truth ${a}^{\star}\in{\mathcal{A}}$, while the multiple-choice task
is to select the correct answer $\hat{a}$ out of the candidate set
$\\{{a}_{i}\\}_{i=1}^{K}$. With the representation ${\textbf{s}}_{a}$, for
each of the three tasks, we train and evaluate a separate model with different
predicting functions and training losses. Detailed descriptions of the three
predictors with training and inference settings can be found in Appendix.

Table 1: Comparison on TGIF-QA in terms of MSE ($\downarrow$) for Count and
Accuracy (%, $\uparrow$) for others.

Methods |  |  FrameQA ($\uparrow$)
---
|  Count ($\downarrow$)
---
|  Action ($\uparrow$)
---
|  Trans ($\uparrow$)
---
ST-VQA-Sp (Jang et al. 2017) | 45.5 | 4.28 | 57.3 | 63.7
ST-VQA-Tp (Jang et al. 2017) | 49.3 | 4.40 | 60.8 | 67.1
ST-VQA-SpTp (Jang et al. 2017) | 47.8 | 4.56 | 57.0 | 59.6
ST-VQA$\star$ (Jang et al. 2019) | 52.0 | 4.22 | 73.5 | 79.7
Co-Mem (Gao et al. 2018) | 51.5 | 4.10 | 68.2 | 74.3
PSAC (Li et al. 2019) | 55.7 | 4.27 | 70.4 | 76.9
HME (Fan et al. 2019) | 53.8 | 4.10 | 73.9 | 77.8
HGA (Jiang and Han 2020) | 55.1 | 4.09 | 75.4 | 81.0
Ours | 55.3 | 4.18 | 75.8 | 81.2

## Performance Evaluation

In this section, we evaluate our proposed model on two widely used large-scale
VideoQA benchmarks and discuss the experimental results quantitatively and
qualitatively.

### Benchmark Datasets

TGIF-QA. TGIF-QA (Jang et al. 2017) is the most commonly used benchmark for
VideoQA with 165K QA pairs from 72 animated GIFs. It includes four different
VideoQA tasks: (1) FrameQA: An open-ended words task, in which the question
can be answered based on a single video frame. (2) Count: An open-ended
numbers task, asking the number of repetition of a given action. (3) Action: A
5-option multiple-choice task, which aims to recognize a repeated action given
its specific times. (4) Trans: A 5-option multiple-choice task, asking the
transition happened between two states.

MSVD-QA. MSVD-QA (Xu et al. 2017) is generated automatically based on
Microsoft Research Video Description Corpus (Chen and Dolan 2011). It contains
50K QA pairs associated with 1,970 video clips and consists of five different
types of questions, including What, Who, How, When and Where. All questions
are open-ended words tasks, where the answer is predicted from a pre-defined
answer set of size 1,000.

### Experimental Setup

Implementation details. We employed the pre-trained ResNet-152 and VGG to
obtain the appearance features for TGIF-QA and MSVD-QA respectively and the
pre-trained C3D to extract the motion features. $d_{c}$, $d_{a}$, $d_{m}$,
$d_{v}$, and $d_{q}$ are set to 512, 2,048, 4,096, 4,096, and 300,
respectively. ${d_{V}}$, ${d_{Q}}$, and ${d_{C}}$ are all set to 512. The
dimensions of ${\textbf{h}}_{t}$ and ${\textbf{s}}_{a}$ are set to 512 and
1,536 respectively. The proposed model was trained with Adam optimizer (Kingma
and Ba 2015) with an initial learning rate $10^{-4}$ and a batch size 64. Our
code was implemented in PyTorch (Paszke et al. 2017) and run with NVIDIA Tesla
P100 GPUs.

Evaluation metrics. For both open-ended words and multiple-choice tasks, we
adopted the classification accuracy (i.e., Accuracy) as the evaluation metric.
For open-ended number task, as all the answers belong to 11 possible integer
values ranging from 0 to 10, we evaluated the model in terms of the mean
square error (i.e., MSE) between the predicted integer and the ground-truth.
Note that the model delivers better performance with higher classification
accuracy while lower mean square error.

Baselines. Our model was compared with some typical and state-of-the-art
methods. For TGIF-QA dataset, we compared EC-GNNs with ST-VQAs (Jang et al.
2017), ST-VQA$\star$ (Jang et al. 2019), Co-Mem (Gao et al. 2018), HME (Fan et
al. 2019), PSAC (Li et al. 2019), HGA (Jiang and Han 2020) and HCRN (Le et al.
2020). For MSVD-QA dataset, in addition to ST-VQA, Co-Mem, HME and HGA, we
also compared against E-VQA (Zeng et al. 2017), E-MN (Zeng et al. 2017) and
AMU (Xu et al. 2017).

### Experimental Results

We compare our proposed model with the aforementioned baselines. The results
on TGIF-QA and MSVD-QA datasets are shown in Tables 1 and 2, and we can make
the following observations.

State-of-the art performance especially on event-related tasks. EC-GNNs
outperforms most baselines and performs comparably with state-of-the-art
methods (e.g., HGA (Jiang and Han 2020) and HCRN (Le et al. 2020)) for both
datasets. Notably, EC-GNNs delivers an accuracy of $75.8\%$ on the Action task
of TGIF-QA, excelling the second best by $0.4$ percentage points; On MSVD-QA,
it performs the best on What task and the union of all tasks (denoted as All
in Table 2). The results suggest that leveraging event-related information
(i.e., dense video captions) is beneficial for the cases where obvious event
information (e.g., actions) is involved and multiple events occur concurrently
or successively (e.g., state transitions), and clearly demonstrate the
effectiveness and feasibility of the proposed EC-GNNs.

Advantages of modality-aware graph networks. Comparing to HGA (Jiang and Han
2020), which also uses GCNs for the modeling of intra-modal relationships, EC-
GNNs performs better on most tasks and question types. The reason may be that
HGA uses one graph, ignoring the pattern differences between visual and
linguistic modalities and leading to semantic bias, while EC-GNNs employ
modality-aware graphs that model heterogeneous intra-modal relationships and
revealing multi-modal features.

Gains of question guidance. EC-GNNs consistently outperforms HME (Fan et al.
2019), which employs similar multi-modal fusion but lacks question guidance.
This suggests that gathering question-oriented information is crucial for a
comprehensive understanding of cross-modal semantics. An ablation study is
conducted to further justify the superiority of question guidance in multi-
modal fusion.

Superiority of GCNs to RNNs. EC-GNNs perform consistently better than E-VQA
(Zeng et al. 2017), a typical RNNs-based VideoQA method. It implies that for
VideoQA tasks, GCNs learn better long-term dependencies than RNNs, especially
when complex semantics are involved.

Table 2: Comparison on MSVD-QA in terms of Accuracy (%, $\uparrow$).

Methods |  |  What ($\uparrow$)
---
|  Who ($\uparrow$)
---
|  How ($\uparrow$)
---
|  When ($\uparrow$)
---
|  Where ($\uparrow$)
---
|  All ($\uparrow$)
---
E-VQA (Zeng et al. 2017) | 9.7 | 42.2 | 83.8 | 72.4 | 53.6 | 23.3
E-MN (Zeng et al. 2017) | 12.9 | 46.5 | 80.3 | 70.7 | 50.0 | 26.7
AMU (Xu et al. 2017) | 20.6 | 47.5 | 83.5 | 72.4 | 53.6 | 32.0
ST-VQA (Jang et al. 2017) | 18.1 | 50.0 | 83.8 | 72.4 | 28.6 | 31.3
Co-Mem (Gao et al. 2018) | 19.6 | 48.7 | 81.6 | 74.1 | 31.7 | 31.7
HME (Fan et al. 2019) | 22.4 | 50.1 | 73.0 | 70.7 | 42.9 | 33.7
HGA (Jiang and Han 2020) | 23.5 | 50.4 | 83.0 | 72.4 | 46.4 | 34.7
Ours | 24.0 | 49.9 | 79.7 | 70.7 | 50.0 | 34.8

Figure 5: Visualization of learned attention weights over the caption (in
green), visual (in blue), and word (in red) features. Dark/light color denotes
higher/lower values.

### Visualization

To demonstrate the efficacy of the proposed model, we use Figure 5 to
visualize the learned attention weights222Please refer to Appendix for the
definitions and equations of the learned attention weights. over caption,
visual, and word features in the question-guided self-adaptive multi-modal
fusion.

The first sample shows that the proposed model can effectively focus on the
most relevant frames, words and captions for a given question. This example is
from the MSVD-QA dataset. From the point view of the video as a whole, it is
difficult to know who is in the middle of the stage, especially when close-up
shots take up most of the video. Furthermore, the model needs to find out the
most important ones from a large number of dense video captions, which helps
to infer the correct answer. Our model subtly pays more attention to a few
overall shots in the middle of the video, where a woman leads the dance in the
center of the stage while others dance around. Correspondingly, in the caption
modality, the model assigns a higher weight to the caption the women in the
dress continue dancing and one of the women in the middle of the stage while
reducing other weights.

In the latter two samples, the model is able to directly infer the correct
answer by referring to some correlated captions. In addition to providing some
visually intuitive information, such as the presence of two women in the video
of the second sample, the captions can also incorporate some commonsense
knowledge that may not be represented by an end-to-end model for VideoQA
datasets with limited sample diversity (Khademi 2020). For instance, in the
third sample, the caption a man is riding a horse in a field can naturally
associate the horse riding with field, which provides exactly where the horse
and rider trot across. Nonetheless, without assistance from dense video
captions, most other models have to recognize the environment based on visual
features, which is quite challenging due to the ambiguities contained in video
frames.

Table 3: Ablation study on TGIF-QA. MSE ($\downarrow$) for Count and Accuracy
(%, $\uparrow$) for others. (Vid: Video Modality; Cap: Caption Modality; CMR:
Cross-Modal Reasoning Module; MMF: Multi-Modal Fusion module from (Fan et al.
2019); Q-MMF: Question-guided self-adaptive Mutli-Modal Fusion module.)

Case # | Vid | Cap | CMR | MMF | Q-MMF |  |  FrameQA ($\uparrow$)
---
|  Count ($\downarrow$)
---
|  Action ($\uparrow$)
---
|  Trans ($\uparrow$)
---
1 | ✓ |  | ✓ |  | ✓ | 52.1 | 4.33 | 73.4 | 77.5
2 | ✓ | ✓ |  |  | ✓ | 53.3 | 4.28 | 74.0 | 76.8
3 | ✓ | ✓ | ✓ |  |  | 52.6 | 4.32 | 74.7 | 78.5
4 | ✓ | ✓ | ✓ | ✓ |  | 54.1 | 4.15 | 75.3 | 80.8
ALL | ✓ | ✓ | ✓ |  | ✓ | 55.3 | 4.18 | 75.8 | 81.2

### Ablation Study

To validate the contribution of each component in the proposed EC-GNNs,
several ablation experiments were conducted on TGIF-QA. The following
observations are made based on Table 3.

Benifits of generating an event-correlated modality. EC-GNNs incorporates
dense video captions as a new modality to perform reasoning of VideoQA, which
leads to better performance. Compared with Case-ALL, which learns the full
model, the accuracy of the Action task decreases by $2.4\%$ in Case-1, where
the caption modality is not modeled. This suggests that dense video captions
can provide valuable evidence for answer reasoning, especially when the videos
and questions contain complex semantics. Similar trends can be observed in
FrameQA, Count and Trans tasks.

Effectiveness of cross-modal reasoning. The model in Case-2, without the
cross-modal reasoning module, gets $1.8\%$ accuracy degradation of the Action
task compared with Case-ALL. Degraded performance of this ablated model is
also observed on the other three tasks. This demonstrates the benefits of
cross-modal reasoning in the modeling inter-modal relationships, which can
aggregate relevant information across different modalities.

Gains of question guidance in multi-modal fusion. The leveraging of multi-
modal fusion delivers a performance gain over Case-3, where only the self-
attention pooling and bi-linear fusion are applied for feature fusion (Jiang
and Han 2020). For instance, for the Action task, Case-4 brings up
improvements of $0.6\%$ on accuracy, and further adopting question-guided
self-adaptive multi-modal fusion in Case-ALL achieves an additional gain of
$0.5\%$. Similar trends can be observed in FrameQA and Trans tasks. In brief,
collecting the question-oriented and event-correlated evidence facilitates a
comprehensive understanding of cross-modal semantics and outperforms simple
multi-modal fusion.

## Conclusions

By introducing Event-Correlated Graph Neural Networks (EC-GNNs), we propose a
unified end-to-end trainable model for Video Question Answering (VideoQA).
Notably, this paper clarifies how event-correlated information can be
extracted as a new modality and integrated into VideoQA inference.
Furthermore, we solve the VideoQA task through a multi-modal graph consisting
of three modality-aware graph convolution networks that perform cross-modal
reasoning over three modalities (i.e., the caption, the video, and the
question). Comprehensive experiments have been conducted on two widely-used
VideoQA benchmarks, and extensive experimental results show the superiority of
introducing the event-based caption modality and the effectiveness of the
proposed model equipped with the cross-modal attention mechanism and the
question guidance.

## References

  * Anderson et al. (2018) Anderson, P.; He, X.; Buehler, C.; Teney, D.; Johnson, M.; Gould, S.; and Zhang, L. 2018.  Bottom-up and top-down attention for image captioning and visual question answering.  In _Proceedings of IEEE CVPR_ , 6077–6086.
  * Antol et al. (2015) Antol, S.; Agrawal, A.; Lu, J.; Mitchell, M.; Batra, D.; Zitnick, C. L.; and Parikh, D. 2015.  Vqa: Visual question answering.  In _Proceedings of IEEE ICCV_ , 2425–2433.
  * Ba, Kiros, and Hinton (2016) Ba, J. L.; Kiros, J. R.; and Hinton, G. E. 2016.  Layer normalization.  _arXiv preprint arXiv:1607.06450_.
  * Bahdanau, Cho, and Bengio (2014) Bahdanau, D.; Cho, K.; and Bengio, Y. 2014.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_.
  * Chen and Dolan (2011) Chen, D.; and Dolan, W. B. 2011.  Collecting highly parallel data for paraphrase evaluation.  In _ACL_ , 190–200.
  * Chen et al. (2015) Chen, K.; Wang, J.; Chen, L.-C.; Gao, H.; Xu, W.; and Nevatia, R. 2015.  Abc-cnn: An attention based convolutional neural network for visual question answering.  _arXiv preprint arXiv:1511.05960_.
  * Cho, Courville, and Bengio (2015) Cho, K.; Courville, A.; and Bengio, Y. 2015.  Describing multimedia content using attention-based encoder-decoder networks.  _IEEE TMM_ , 17(11): 1875–1886.
  * Chung et al. (2014) Chung, J.; Gulcehre, C.; Cho, K.; and Bengio, Y. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_.
  * Donahue et al. (2015) Donahue, J.; Anne Hendricks, L.; Guadarrama, S.; Rohrbach, M.; Venugopalan, S.; Saenko, K.; and Darrell, T. 2015.  Long-term recurrent convolutional networks for visual recognition and description.  In _Proceedings of IEEE CVPR_ , 2625–2634.
  * Escorcia et al. (2016) Escorcia, V.; Caba Heilbron, F.; Niebles, J. C.; and Ghanem, B. 2016.  Daps: Deep action proposals for action understanding.  In _Proceedings of ECCV_ , 768–784. Springer.
  * Fan et al. (2019) Fan, C.; Zhang, X.; Zhang, S.; Wang, W.; Zhang, C.; and Huang, H. 2019.  Heterogeneous memory enhanced multimodal attention model for video question answering.  In _Proceedings of IEEE CVPR_ , 1999–2007.
  * Gao et al. (2018) Gao, J.; Ge, R.; Chen, K.; and Nevatia, R. 2018.  Motion-appearance co-memory networks for video question answering.  In _Proceedings of IEEE CVPR_ , 6576–6585.
  * Gao et al. (2017) Gao, L.; Guo, Z.; Zhang, H.; Xu, X.; and Shen, H. T. 2017.  Video captioning with attention-based LSTM and semantic consistency.  _IEEE TMM_ , 19(9): 2045–2055.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of IEEE CVPR_ , 770–778.
  * Ioffe and Szegedy (2015) Ioffe, S.; and Szegedy, C. 2015.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _Proceedings of ICML_ , 448–456. PMLR.
  * Jang et al. (2019) Jang, Y.; Song, Y.; Kim, C. D.; Yu, Y.; Kim, Y.; and Kim, G. 2019.  Video question answering with spatio-temporal reasoning.  _IJCV_ , 127(10): 1385–1412.
  * Jang et al. (2017) Jang, Y.; Song, Y.; Yu, Y.; Kim, Y.; and Kim, G. 2017.  Tgif-qa: Toward spatio-temporal reasoning in visual question answering.  In _Proceedings of IEEE CVPR_ , 2758–2766.
  * Jiang and Han (2020) Jiang, P.; and Han, Y. 2020.  Reasoning with heterogeneous graph alignment for video question answering.  In _Proceedings of AAAI_ , volume 34, 11109–11116.
  * Karpathy et al. (2014) Karpathy, A.; Toderici, G.; Shetty, S.; Leung, T.; Sukthankar, R.; and Fei-Fei, L. 2014.  Large-scale video classification with convolutional neural networks.  In _Proceedings of IEEE CVPR_ , 1725–1732.
  * Khademi (2020) Khademi, M. 2020.  Multimodal neural graph memory networks for visual question answering.  In _ACL_ , 7177–7188.
  * Kim et al. (2019) Kim, J.; Ma, M.; Kim, K.; Kim, S.; and Yoo, C. D. 2019.  Progressive attention memory network for movie story question answering.  In _Proceedings of IEEE CVPR_ , 8337–8346.
  * Kim et al. (2018) Kim, K.-M.; Choi, S.-H.; Kim, J.-H.; and Zhang, B.-T. 2018.  Multimodal dual attention memory for video story question answering.  In _Proceedings of ECCV_ , 673–688.
  * Kingma and Ba (2015) Kingma, D. P.; and Ba, J. 2015.  Adam: A method for stochastic optimization.  In _ICLR_.
  * Kipf and Welling (2016) Kipf, T. N.; and Welling, M. 2016.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_.
  * Krishna et al. (2017) Krishna, R.; Hata, K.; Ren, F.; Fei-Fei, L.; and Carlos Niebles, J. 2017.  Dense-captioning events in videos.  In _Proceedings of IEEE ICCV_ , 706–715.
  * Le et al. (2020) Le, T. M.; Le, V.; Venkatesh, S.; and Tran, T. 2020.  Hierarchical conditional relation networks for video question answering.  In _Proceedings of IEEE CVPR_ , 9972–9981.
  * Li et al. (2017) Li, L.; Tang, S.; Zhang, Y.; Deng, L.; and Tian, Q. 2017.  GLA: Global–local attention for image description.  _IEEE TMM_ , 20(3): 726–737.
  * Li et al. (2019) Li, X.; Song, J.; Gao, L.; Liu, X.; Huang, W.; He, X.; and Gan, C. 2019.  Beyond rnns: Positional self-attention with co-attention for video question answering.  In _Proceedings of AAAI_ , volume 33, 8658–8665.
  * Ma et al. (2018) Ma, C.; Shen, C.; Dick, A.; Wu, Q.; Wang, P.; van den Hengel, A.; and Reid, I. 2018\.  Visual question answering with memory-augmented networks.  In _Proceedings of IEEE CVPR_ , 6975–6984.
  * Mikolov et al. (2013) Mikolov, T.; Chen, K.; Corrado, G.; and Dean, J. 2013.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Pan et al. (2016) Pan, Y.; Mei, T.; Yao, T.; Li, H.; and Rui, Y. 2016.  Jointly modeling embedding and translation to bridge video and language.  In _Proceedings of IEEE CVPR_ , 4594–4602.
  * Paszke et al. (2017) Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; DeVito, Z.; Lin, Z.; Desmaison, A.; Antiga, L.; and Lerer, A. 2017.  Automatic differentiation in pytorch.
  * Pennington, Socher, and Manning (2014) Pennington, J.; Socher, R.; and Manning, C. D. 2014.  Glove: Global vectors for word representation.  In _Proceedings of EMNLP_ , 1532–1543.
  * Russakovsky et al. (2015) Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al. 2015.  Imagenet large scale visual recognition challenge.  _IJCV_ , 115(3): 211–252.
  * Simonyan and Zisserman (2014) Simonyan, K.; and Zisserman, A. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_.
  * Teney et al. (2018) Teney, D.; Anderson, P.; He, X.; and Van Den Hengel, A. 2018.  Tips and tricks for visual question answering: Learnings from the 2017 challenge.  In _Proceedings of IEEE CVPR_ , 4223–4232.
  * Tran et al. (2015) Tran, D.; Bourdev, L.; Fergus, R.; Torresani, L.; and Paluri, M. 2015.  Learning spatiotemporal features with 3d convolutional networks.  In _Proceedings of IEEE ICCV_ , 4489–4497.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  _NeurIPS_ , 30.
  * Venugopalan et al. (2015) Venugopalan, S.; Rohrbach, M.; Donahue, J.; Mooney, R.; Darrell, T.; and Saenko, K. 2015.  Sequence to sequence-video to text.  In _Proceedings of IEEE ICCV_ , 4534–4542.
  * Venugopalan et al. (2014) Venugopalan, S.; Xu, H.; Donahue, J.; Rohrbach, M.; Mooney, R.; and Saenko, K. 2014\.  Translating videos to natural language using deep recurrent neural networks.  _arXiv preprint arXiv:1412.4729_.
  * Wang et al. (2018) Wang, J.; Jiang, W.; Ma, L.; Liu, W.; and Xu, Y. 2018.  Bidirectional attentive fusion with context gating for dense video captioning.  In _Proceedings of IEEE CVPR_ , 7190–7198.
  * Wang et al. (2021) Wang, T.; Zhang, R.; Lu, Z.; Zheng, F.; Cheng, R.; and Luo, P. 2021.  End-to-End Dense Video Captioning with Parallel Decoding.  In _Proceedings of IEEE ICCV_ , 6847–6857.
  * Wang, Zheng, and Yu (2020) Wang, T.; Zheng, H.; and Yu, M. 2020.  Dense-captioning events in videos: Sysu submission to activitynet challenge 2020.  _arXiv preprint arXiv:2006.11693_.
  * Wang and Gupta (2018) Wang, X.; and Gupta, A. 2018.  Videos as space-time region graphs.  In _Proceedings of ECCV_ , 399–417.
  * Xu et al. (2017) Xu, D.; Zhao, Z.; Xiao, J.; Wu, F.; Zhang, H.; He, X.; and Zhuang, Y. 2017.  Video question answering via gradually refined attention over appearance and motion.  In _Proceedings of ACMMM_ , 1645–1653.
  * Yao et al. (2015) Yao, L.; Torabi, A.; Cho, K.; Ballas, N.; Pal, C.; Larochelle, H.; and Courville, A. 2015.  Describing videos by exploiting temporal structure.  In _Proceedings of IEEE ICCV_ , 4507–4515.
  * Ye et al. (2017) Ye, Y.; Zhao, Z.; Li, Y.; Chen, L.; Xiao, J.; and Zhuang, Y. 2017.  Video question answering via attribute-augmented attention network learning.  In _Proceedings of International ACM SIGIR_ , 829–832.
  * Yin et al. (2019) Yin, C.; Tang, J.; Xu, Z.; and Wang, Y. 2019.  Memory augmented deep recurrent neural network for video question answering.  _IEEE TNNLS_ , 31(9): 3159–3167.
  * Yin et al. (2021) Yin, C.; Wu, K.; Che, Z.; Jiang, B.; Xu, Z.; and Tang, J. 2021.  Hierarchical graph attention network for few-shot visual-semantic learning.  In _Proceedings of the IEEE ICCV_ , 2177–2186.
  * Yu et al. (2016) Yu, H.; Wang, J.; Huang, Z.; Yang, Y.; and Xu, W. 2016.  Video paragraph captioning using hierarchical recurrent neural networks.  In _Proceedings of IEEE CVPR_ , 4584–4593.
  * Zeng et al. (2017) Zeng, K.-H.; Chen, T.-H.; Chuang, C.-Y.; Liao, Y.-H.; Niebles, J. C.; and Sun, M. 2017.  Leveraging video descriptions to learn video question answering.  In _Proceedings of AAAI_.
  * Zhou, Xu, and Corso (2018) Zhou, L.; Xu, C.; and Corso, J. J. 2018.  Towards automatic learning of procedures from web instructional videos.  In _Proceedings of AAAI_.
  * Zhou et al. (2018) Zhou, L.; Zhou, Y.; Corso, J. J.; Socher, R.; and Xiong, C. 2018.  End-to-end dense video captioning with masked transformer.  In _Proceedings of IEEE CVPR_ , 8739–8748.
  * Zhu et al. (2017) Zhu, L.; Xu, Z.; Yang, Y.; and Hauptmann, A. G. 2017.  Uncovering the temporal context for video question answering.  _IJCV_ , 124(3): 409–421.

##  Appendix A Appendix

### Methodological Details

#### Representing Video and Question Modalities

In addition to the generated caption modality described in the paper, the
video modality and question modality are also properly represented and
utilized in the proposed model.

Representing video modality. In order to take into account the appearance and
motion information involved in video contents, we represent a video at both
frame-level and shot-level. Specifically, to obtain appearance features, we
use the 2D ConvNets (i.e., ResNet-152 (He et al. 2016) and VGG (Simonyan and
Zisserman 2014)) pre-trained on the ImageNet 2012 classification dataset
(Russakovsky et al. 2015). For shot-level motion features, we leverage 3D
ConvNets (i.e., C3D (Tran et al. 2015)) pre-trained on the Sport1M dataset
(Karpathy et al. 2014). Then, the video is represented as two feature views,
appearance features ${\textbf{F}}^{a}=\\{{\textbf{f}}_{i}^{a}:i\leq
N_{v},{\textbf{f}}_{i}^{a}\in{\mathcal{R}}^{d_{a}}\\}$, and motion features
${\textbf{F}}^{m}=\\{{\textbf{f}}_{i}^{m}:i\leq
N_{v},{\textbf{f}}_{i}^{m}\in{\mathcal{R}}^{d_{m}}\\}$, where $N_{v}$ is
number of frames and $d_{a}$, $d_{m}$ are dimensionalities of the two feature
views. We project the concatenation of the two features into a common visual
space by two fully-connected layers, to obtain a joint representation of the
video ${\textbf{F}}^{v}=\\{{\textbf{f}}_{i}^{v}:i\leq
N_{v},{\textbf{f}}_{i}^{v}\in{\mathcal{R}}^{d_{v}}\\}$, where $d_{v}$
represents the dimensionality of visual features.

Representing question modality. For a given question, we represent each word
as a fixed-length vector initialized with the GloVe $300D$ word embedding
(Pennington, Socher, and Manning 2014) pre-trained on the Common Crawl
dataset. The question is then denoted as a sequence of word embeddings
${\textbf{F}}^{q}=\\{{\textbf{f}}_{i}^{q}:i\leq
N_{q},{\textbf{f}}_{i}^{q}\in{\mathcal{R}}^{d_{q}}\\}$, in which $N_{q}$
represents the number of words and $d_{q}$ equals 300.

#### Intra-Modal Graph Reasoning

The intra-modal graph reasoning procedures in these three graphs share the
common operations but differ in their node representations. Thus we take the
video graph as an instance to illustrate the operations of graph reasoning.

We first project the node features ${\textbf{V}}^{l}$ into an interaction
space by a non-linear transformation operation $\phi(.)$. Then we calculate
the dot-product similarity (Wang and Gupta 2018), which measures the semantic
correlations among nodes, and employ softmax function on each row of the
matrix to obtain the normalized adjacency matrix.

$\displaystyle{\textbf{G}}_{V}^{l}$
$\displaystyle=\mbox{softmax}(\phi({\textbf{V}}^{l})\phi({\textbf{V}}^{l})^{T})$
(5)

where ${\textbf{G}}_{V}^{l}\in{\mathcal{R}}^{N_{v}\times N{v}}$ represents the
adjacency matrix and ${{\textbf{G}}_{V}}_{i,j}^{l}$ indicates the weight
between ${\textbf{V}}_{i}^{l}$ and ${\textbf{V}}_{j}^{l}$.

To perform intra-modal reasoning on the graph, we apply the Graph
Convolutional Networks (GCNs) proposed in (Kipf and Welling 2016), which
allows us to update each node based on its neighbors and the corresponding
weights. Given the node features ${\textbf{V}}^{l}$ and weights specified by
the adjacency matrix ${\textbf{G}}_{V}^{l}$, the nodes are updated by a linear
transformation of aggregated excitation of its neighbors and itself. Formally,
we represent the graph convolution on layer $l$ of video graph as

$\displaystyle{\hat{\textbf{V}}}^{l}$
$\displaystyle=\mbox{Relu}(\mbox{LayerNorm}({\textbf{G}}_{V}^{l}{\textbf{V}}^{l}{\textbf{W}}))$
(6)

where ${\textbf{W}}\in{\mathcal{R}}^{d_{V}\times d_{V}}$ is a learnable weight
matrix and ${\hat{\textbf{V}}}^{l}=\\{{\hat{\textbf{V}}}^{l}_{i}:i\leq
N_{v},{\hat{\textbf{V}}}^{l}_{i}\in{\mathcal{R}}^{d_{V}}\\}$ represents the
updated node features. Two non-linear functions including Layer Normalization
(Ba, Kiros, and Hinton 2016) and ReLU are applied after the graph convolution.

We conduct the above intra-modal graph reasoning on caption graph, video
graph, and question graph independently and obtain the updated node features
${\hat{\textbf{C}}}^{l}$, ${\hat{\textbf{V}}}^{l}$ and
${\hat{\textbf{Q}}}^{l}$ accordingly, which are forwarded to cross-modal
reasoning ($l=1,2$) and multi-modal fusion ($l=3$) respectively.

Figure 6: An illustration of the cross-modal reasoning module. For simplicity,
the cross-modal reasoning module in video graph between layer $l$ and $l+1$ is
presented, and the query node ${\textbf{V}}_{i}^{l}$ is taken as an example
for demonstration. Green, blue, and red arrows between modules corresponds to
the information flow of caption, video, and question modalities respectively.
Dotted arrows represent the transmission of node features. Different shades of
color indicate representations of different nodes. FF and LN denote feed-
forward network and layer normalization.

#### Operations of Cross-Modal Graph Reasoning

The cross-modal attention mechanism (CAM) is used to explicitly exploit the
inter-modal relationships of one modality with two other modalities. As same
operations shared across the cross-modal reasoning procedures in three graphs,
in Figure 6, we give a description of cross-modal reasoning using video graph
as an example, in which the node features of video graph are employed as query
vectors, and node features of question graph and caption graph serve as key-
value pairs.

We first linearly project the updated node features ${\hat{\textbf{V}}}^{l}$,
${\hat{\textbf{Q}}}^{l}$ and ${\hat{\textbf{C}}}^{l}$ into a transformed
space.

$\displaystyle{{\textbf{M}}_{Q}^{v}}^{l}$
$\displaystyle={\textbf{W}}_{Q}^{v}{\hat{\textbf{V}}}^{l}$ (7)
$\displaystyle{{\textbf{M}}_{K}^{q}}^{l},{{\textbf{M}}_{V}^{q}}^{l}$
$\displaystyle={\textbf{W}}_{K}^{q}{\hat{\textbf{Q}}}^{l},{\textbf{W}}_{V}^{q}{\hat{\textbf{Q}}}^{l}$
(8) $\displaystyle{{\textbf{M}}_{K}^{c}}^{l},{{\textbf{M}}_{V}^{c}}^{l}$
$\displaystyle={\textbf{W}}_{K}^{c}{\hat{\textbf{C}}}^{l},{\textbf{W}}_{V}^{c}{\hat{\textbf{C}}}^{l}$
(9)

where ${\textbf{W}}_{Q}^{v}$, ${\textbf{W}}_{K}^{q}$, ${\textbf{W}}_{V}^{q}$,
${\textbf{W}}_{K}^{c}$ and ${\textbf{W}}_{V}^{c}$ are the learnable weight
matrices.
${{\textbf{M}}_{Q}^{v}}^{l}=\\{{{\textbf{M}}_{{Q}_{i}}^{v}}^{l}:i\leq
N_{v}\\}$ represents the transformed visual features in the query space.
${{\textbf{M}}_{K}^{q}}^{l}=\\{{{\textbf{M}}_{{K}_{i}}^{q}}^{l}:i\leq
N_{q}\\}$ and
${{\textbf{M}}_{V}^{q}}^{l}=\\{{{\textbf{M}}_{{V}_{i}}^{q}}^{l}:i\leq
N_{q}\\}$ represent the transformed word features in the key and value spaces.
${{\textbf{M}}_{K}^{c}}^{l}=\\{{{\textbf{M}}_{{K}_{i}}^{c}}^{l}:i\leq
N_{c}\\}$ and
${{\textbf{M}}_{V}^{c}}^{l}=\\{{{\textbf{M}}_{{V}_{i}}^{c}}^{l}:i\leq
N_{c}\\}$ represent the transformed caption features in the key and value
spaces. Then we apply the CAM to embed the visual information into the feature
spaces of question words and dense captions respectively to collect the
relevant information from question and caption modalities.

$\displaystyle{{\textbf{V}}^{vq}}^{l}$
$\displaystyle=\mbox{CAM}({{\textbf{M}}_{Q}^{v}}^{l},{{\textbf{M}}_{K}^{q}}^{l},{{\textbf{M}}_{V}^{q}}^{l})$
(10) $\displaystyle{{\textbf{V}}^{vc}}^{l}$
$\displaystyle=\mbox{CAM}({{\textbf{M}}_{Q}^{v}}^{l},{{\textbf{M}}_{K}^{c}}^{l},{{\textbf{M}}_{V}^{c}}^{l})$
(11)

where ${{\textbf{V}}^{vq}}^{l}=\\{{{\textbf{V}}_{i}^{vq}}^{l}:i\leq N_{v}\\}$
and ${{\textbf{V}}^{vc}}^{l}=\\{{{\textbf{V}}_{i}^{vc}}^{l}:i\leq N_{v}\\}$
indicate the word features and caption features attended by the visual
features (i.e., CAM-based features), which are combined with the node features
${\hat{\textbf{V}}}^{l}$ to obtain the updated visual features
${\textbf{V}}^{l+1}=\\{{\textbf{V}}^{l+1}_{i}:i\leq
N_{v},{\textbf{V}}^{l+1}_{i}\\}$ after cross-modal reasoning.

$\displaystyle{\textbf{V}}^{l+1}$
$\displaystyle=\mbox{LayerNorm}(\mbox{FF}([{{\textbf{V}}^{vq}}^{l}||{{\textbf{V}}^{vc}}^{l}||{\hat{\textbf{V}}}^{l}])+{\hat{\textbf{V}}}^{l})$
(12)

where $||$ denotes the concatenation operation and FF represents a feed-
forward network. To achieve a better performance, a residual connection (He et
al. 2016) is adopted for multiple layers stacking.

The above cross-modal reasoning is performed on video graph, question graph
and caption graph and obtains the updated node features ${\textbf{V}}^{l+1}$,
${\textbf{Q}}^{l+1}$ and ${\textbf{C}}^{l+1}$, which are forwarded to the next
GNN layer for the next round of intra-modal graph reasoning.

Figure 7: An illustration of the question-guided self-adaptive multi-modal
fusion module at the $t$-th step. An LSTM controller with previous state
${\textbf{h}}_{t-1}$ attends to relevant visual features, word features and
caption features with question guidance, and combines them to obtain current
state ${\textbf{h}}_{t}$.

#### Calculating Question-Guided Self-Adaptive Multi-Modal Fusion

The multi-modal fusion module takes the updated node features from the last
GNN layer (i.e., caption features ${\hat{\textbf{C}}}^{3}$, visual features
${\hat{\textbf{V}}}^{3}$, and word features ${\hat{\textbf{Q}}}^{3}$) and the
last hidden state of question (i.e., ${\textbf{q}}_{N_{q}}$) as the inputs.
For the multi-step reasoning, inspired by the multi-modal fusion module in HME
(Fan et al. 2019), we adopt an LSTM controller with its previous state denoted
as ${\textbf{h}}_{t-1}$.

At $t$-th step of reasoning, as shown in Figure 7, by interacting with the
last hidden state of question as well as the previous state of LSTM
controller, the module first employs temporal attention mechanism to attend to
different parts of the caption features, visual features, and word features,
independently, and collects the question-oriented information from each
modality.

$\displaystyle{\textbf{att}}^{c}$
$\displaystyle=\mbox{softmax}({\textbf{W}}^{c}\mbox{tanh}({\textbf{W}}_{q}^{c}{\textbf{q}}_{N_{q}}+{\textbf{W}}_{h}^{c}{\textbf{h}}_{t-1}+{\textbf{W}}_{c}^{c}{\hat{\textbf{C}}}^{3}+{\textbf{b}}^{c}))$
(13) $\displaystyle{\textbf{att}}^{v}$
$\displaystyle=\mbox{softmax}({\textbf{W}}^{v}\mbox{tanh}({\textbf{W}}_{q}^{v}{\textbf{q}}_{N_{q}}+{\textbf{W}}_{h}^{v}{\textbf{h}}_{t-1}+{\textbf{W}}_{v}^{v}{\hat{\textbf{V}}}^{3}+{\textbf{b}}^{v}))$
(14) $\displaystyle{\textbf{att}}^{q}$
$\displaystyle=\mbox{softmax}({\textbf{W}}^{q}\mbox{tanh}({\textbf{W}}_{q}^{q}{\textbf{q}}_{N_{q}}+{\textbf{W}}_{h}^{q}{\textbf{h}}_{t-1}+{\textbf{W}}_{q}^{q}{\hat{\textbf{Q}}}^{3}+{\textbf{b}}^{q}))$
(15) $\displaystyle\tilde{\textbf{C}}=$
$\displaystyle\sum_{i=1}^{N_{c}}{\textbf{att}}_{i}^{c}{\hat{\textbf{C}}}_{i}^{3}\quad\tilde{\textbf{V}}=\sum_{i=1}^{N_{v}}{\textbf{att}}_{i}^{v}{\hat{\textbf{V}}}_{i}^{3}\quad\tilde{\textbf{Q}}=\sum_{i=1}^{N_{q}}{\textbf{att}}_{i}^{q}{\hat{\textbf{Q}}}_{i}^{3}$
(16)

where ${\textbf{att}}^{c}\in{\mathcal{R}}^{N_{c}}$,
${\textbf{att}}^{v}\in{\mathcal{R}}^{N_{v}}$, and
${\textbf{att}}^{q}\in{\mathcal{R}}^{N_{q}}$ indicate the normalized attention
weights over the caption features ${\hat{\textbf{C}}}^{3}$, visual features
${\hat{\textbf{V}}}^{3}$, and word features ${\hat{\textbf{Q}}}^{3}$
respectively. ${\textbf{W}}^{c}$, ${\textbf{W}}^{v}$, ${\textbf{W}}^{q}$,
${\textbf{W}}_{q}^{c}$, ${\textbf{W}}_{q}^{v}$, ${\textbf{W}}_{q}^{q}$,
${\textbf{W}}_{h}^{c}$, ${\textbf{W}}_{h}^{v}$, ${\textbf{W}}_{h}^{q}$,
${\textbf{W}}_{c}^{c}$, ${\textbf{W}}_{v}^{v}$, ${\textbf{W}}_{q}^{q}$,
${\textbf{b}}^{c}$, ${\textbf{b}}^{v}$ and ${\textbf{b}}^{q}$ are the
learnable weights. $\tilde{\textbf{C}}$, $\tilde{\textbf{V}}$, and
$\tilde{\textbf{Q}}$ represent the attended caption features, attended visual
features, and attended word features respectively, which are then incorporated
together with the learned modality weights
${\bm{\alpha}}=\\{{\alpha}^{c},{\alpha}^{v},{\alpha}^{q}\\}$ to gather the
event-correlated information across the three modalities.

$\displaystyle{\bm{\alpha}}$
$\displaystyle=\mbox{softmax}({\textbf{W}}^{\alpha}\mbox{tanh}([{\textbf{W}}_{c}^{\alpha}\tilde{\textbf{C}}||{\textbf{W}}_{v}^{\alpha}\tilde{\textbf{V}}||{\textbf{W}}_{q}^{\alpha}\tilde{\textbf{Q}}]+{\textbf{W}}_{h}^{\alpha}{\textbf{h}}_{t-1}+{\textbf{b}}^{\alpha}))$
(17) $\displaystyle{{\textbf{x}}_{t}}$
$\displaystyle=\mbox{tanh}({\alpha}^{c}{\textbf{W}}_{c}^{x}\tilde{\textbf{C}}+{\alpha}^{v}{\textbf{W}}_{v}^{x}\tilde{\textbf{V}}+{\alpha}^{q}{\textbf{W}}_{q}^{x}\tilde{\textbf{Q}}+{\textbf{W}}_{h}^{x}{\textbf{h}}_{t-1}+{\textbf{b}}^{x})$
(18)

where ${\textbf{W}}^{\alpha}$, ${\textbf{W}}_{c}^{\alpha}$,
${\textbf{W}}_{v}^{\alpha}$, ${\textbf{W}}_{q}^{\alpha}$,
${\textbf{W}}_{h}^{\alpha}$, ${\textbf{W}}_{c}^{x}$, ${\textbf{W}}_{v}^{x}$,
${\textbf{W}}_{q}^{x}$, ${\textbf{W}}_{h}^{x}$, ${\textbf{b}}^{\alpha}$ and
${\textbf{b}}^{x}$ are the learnable weights. The output feature
${{\textbf{x}}_{t}}$ is forwarded to obtain the current state
${\textbf{h}}_{t}$ of the LSTM controller by
${\textbf{h}}_{t}=\mbox{LSTM}({{\textbf{x}}_{t}},{\textbf{h}}_{t-1})$.

We iteratively perform the above reasoning step to obtain the final
representation of the multi-modal fusion. After $N_{r}$ steps, the state LSTM
controller ${\textbf{h}}_{N_{r}}$ captures the question-oriented and event-
correlated information across the three modalities. We set the value of
$N_{r}$ to 3, referring to that in HME (Fan et al. 2019). Following (Jang et
al. 2017), we apply the standard temporal attention on the contextualized
visual features ${\textbf{V}}^{1}$ and caption features ${\textbf{C}}^{1}$,
and concatenate with ${\textbf{h}}_{N_{r}}$ to obtain the final representation
${\textbf{s}}_{a}$.

#### Answer Prediction from Final Representation

In this section, we introduce how to predict the correct answer based on the
final representation ${\textbf{s}}_{a}$. Same as (Jang et al. 2017), we are
dealing with the VideoQA tasks in three different types, open-ended words,
open-ended numbers and multiple-choice. Given the input video $v$ and the
corresponding question $q$, the open-ended tasks is to infer an answer
$\hat{a}$ from a pre-defined answer set $\mathcal{A}$ of size $C$ that matches
the ground-truth ${a}^{\star}\in{\mathcal{A}}$, while the multiple-choice task
aims to select the correct answer $\hat{a}$ out of the candidate set
$\\{{a}_{i}\\}_{i=1}^{K}$. Note that a separate model is trained and evaluated
for each type of class.

Open-ended word task. The open-ended words is treated as a classification
problem. We apply a linear layer and a softmax function upon the
representation ${\textbf{s}}_{a}$ to generate the probabilities p for all
answers in the pre-defined answer set $\mathcal{A}$.

p
$\displaystyle=\mbox{softmax}({\textbf{W}}_{a}^{w}{\textbf{s}}_{a}+{\textbf{b}}^{w})$
(19)

in which ${\textbf{W}}_{a}^{w}$ and ${\textbf{b}}^{w}$ are the learnable
weights. The cross-entropy loss is adopted to train the model, and the
predicted answer is represented as
${\hat{a}}=\mbox{argmax}_{{\mathcal{A}}}({\textbf{p}})$ in testing phase.

Open-ended number task. We consider the open-ended numbers as a linear
regression problem, which takes the representation ${\textbf{s}}_{a}$ and
outputs a rounded integer value as the predicted answer.

$\displaystyle{\hat{a}}$
$\displaystyle=\mbox{round}({\textbf{W}}_{a}^{n}{\textbf{s}}_{a}+{\textbf{b}}^{n})$
(20)

where ${\textbf{W}}_{a}^{n}$ and ${\textbf{b}}^{n}$ are the learnable weights.
The ${l}_{2}$ loss is leveraged to minimize the gap between the predicted
integer and the ground-truth. Note that $\mathcal{A}$ is an integer-valued
answer set (i.e., 0-10) for the open-ended numbers.

Multiple choice task. For multiple-choice, we concatenate the question $q$
with each answer in the candidate set $\\{{a}_{i}\\}_{i=1}^{K}$ to obtain $K$
word sequences, which are forward to the model individually to get $K$ final
representations $\\{{\textbf{s}}_{a}\\}_{i=1}^{K}$. Then we use a linear
function to transfer the $K$ representations to $K$ scores
${\textbf{s}}=\\{s^{p},s_{1}^{n},s_{2}^{n},...,s_{K-1}^{n}\\}$. The answer
with the highest score is selected as the predicted one. We optimize the model
by maximizing the hinge loss ${L_{hinge}}$ between the score for the correct
answer $s^{p}$ and the scores for the incorrect answers
$\\{s_{i}^{n}\\}_{i=1}^{K-1}$.

$\displaystyle{L_{hinge}}$
$\displaystyle=\sum_{i=1}^{K-1}\mbox{max}(0,1+s_{i}^{n}-s^{p})$ (21)


Further author information: (Send correspondence to Thuong D. Hoang)
Thuong D. Hoang: E-mail<EMAIL_ADDRESS>
# Testbed preparation of a small prototype polarization modulator for LiteBIRD
low-frequency telescope

Thuong D. Hoang  Kavli IPMU (WPI), UTIAS, The University of Tokyo, Kashiwa,
Chiba 277-8583, Japan  Tomotake Matsumura  Kavli IPMU (WPI), UTIAS, The
University of Tokyo, Kashiwa, Chiba 277-8583, Japan  Ryota Takaku  Department
of Physics, The University of Tokyo, Tokyo, Japan  Takashi Hasebe  Kavli IPMU
(WPI), UTIAS, The University of Tokyo, Kashiwa, Chiba 277-8583, Japan  Tommaso
Ghigna  Kavli IPMU (WPI), UTIAS, The University of Tokyo, Kashiwa, Chiba
277-8583, Japan  Nobuhiko Katayama  Kavli IPMU (WPI), UTIAS, The University of
Tokyo, Kashiwa, Chiba 277-8583, Japan  Yuki Sakurai  Okayama University,
3-1-1, Tsushimanaka, Kita-ku, Okayama, Okayama, 700-8530, Japan  Kunimoto
Komatsu  Okayama University, 3-1-1, Tsushimanaka, Kita-ku, Okayama, Okayama,
700-8530, Japan  Teruhito Iida  ispace inc., Tokyo Japan  Yurika Hoshino
Saitama University, 255 Shimookubo, Sakura-ku, Saitama, 338-8570, Japan
Shinya Sugiyama  Saitama University, 255 Shimookubo, Sakura-ku, Saitama,
338-8570, Japan  Hirokazu Ishino  Okayama University, 3-1-1, Tsushimanaka,
Kita-ku, Okayama, Okayama, 700-8530, Japan  for the LiteBIRD collaboration

###### Abstract

LiteBIRD is the Cosmic Microwave Background (CMB) radiation polarization
satellite mission led by ISAS/JAXA. The main scientific goal is to search for
primordial gravitational wave signals generated from the inflation epoch of
the Universe. LiteBIRD telescopes employ polarization modulation units (PMU)
using continuously rotating half-wave plates (HWP). The PMU is a crucial
component to reach unprecedented sensitivity by mitigating systematic effects,
including 1/f noise. We have developed a 1/10 scale prototype PMU of the
LiteBIRD LFT, which has a 5-layer achromatic HWP and a diameter of 50 mm,
spanning the observational frequency range of 34-161 GHz. The HWP is mounted
on a superconducting magnetic bearing (SMB) as a rotor and levitated by a
high-temperature superconductor as a stator. In this study, the entire PMU
system is cooled down to 10 K in the cryostat chamber by a 4-K Gifford-McMahon
(GM) cooler. We propagate an incident coherent millimeter-wave polarized
signal throughout the rotating HWP and detect the modulated signal. We study
the modulated optical signal and any rotational synchronous signals from the
rotation mechanism. We describe the testbed system and the preliminary data
acquired from this setup. This testbed is built to integrate the broadband HWP
PMU and evaluate the potential systematic effects in the optical data. This
way, we can plan with a full-scale model, which takes a long time for
preparation and testing.

###### keywords:

CMB polarization, polarization modulation unit, half-wave plate

##  1 INTRODUCTION

LiteBIRD is a satellite mission to measure the cosmic microwave background
(CMB) over the full sky at a large angular scale, searching for the
inflationary B-mode signal. The LiteBIRD’s main scientific goal is to achieve
the sensitivity in the tensor-to-scalar ratio, $r<0.001$ [1]. The LiteBIRD’s
focal plane accommodates $\sim 5000$ multi-chroic polarized Transition Edge-
sensor (TES) bolometers covering a wide range of frequencies from 34 to 448
GHz. LiteBIRD payload module (PLM) consists of Low-Frequency Telescope (LFT),
Mid-Frequency Telescope (MHT), and High-Frequency Telescope (HFT). These
telescopes operate at a cryogenic temperature of 5 K. Each telescope deploys a
polarization modulation unit (PMU) which is the first optical element of the
satellite. The PMU contains a continuously rotating broadband half-wave plate
(HWP) to modulate the incident CMB polarization signal.

Measurement of the B-mode polarization signal at large angular scales can be
contaminated by $\rm 1/f$ noise[2, 3]. The temperature-to-polarization leakage
also contaminates it by the instrumentally induced systematic effects, e.g.
beam shapes, band-passes filter mismatch [4], and different gains among
detectors. A HWP is the key instrumental element in mitigating systematic
effects. The precise measurement of the CMB polarization signal requires
accurate characterizations and models of all satellite subsystems, including
the first optical element, PMU.

This study is a part of the development program of the LiteBIRD LFT PMU [5, 6,
7, 8, 9]. We made a $\rm 1/10$ scale prototype PMU, which contains a five-
layer achromatic HWP (AHWP). A single wave plate is a sapphire disc with a
diameter of 50 mm. The first and fifth layers include anti-reflective sub-
wavelength structures. This small prototype PMU contains all the sub-
components, which are to be scaled to a full-scale LFT PMU model, e.g. a
cryogenic holder mechanism, rotation mechanism, superconducting magnetic
bearing (SMB), encoder, readout monitoring, and drive electronics. Therefore,
this prototype PMU system is a valuable development model for understanding
the potential malfunctions, systematic effects, and unexpected features we
anticipate in an upcoming full-scale model, reducing the risk involved in the
full-scale development program. Examples are the HWP rotor vibrations, the HWP
position angle reconstruction, the heat dissipation, and the realistic
modulated signal from the HWP rotation.

In this paper, we describe a small prototype PMU, which is placed inside a 4-K
Gifford-McMahon (GM) cryostat operating at 10 K. A part of the development
status and the components of the small prototype PMU have been previously
reported in Komatsu et al.[7]. We describe the progress and some of the sub-
components, which are not detailed in the previous report.

The paper is structured as follows: Section 2 presents the experimental setup
of the small prototype PMU inside a cryostat. Section 3 shows the preliminary
results of the angle reconstruction, the heat dissipation estimation, and the
modulated signal of the continuously rotating AHWP. Section 4 discusses future
improvements, the misalignment of the optical setup, and the impact of the
inhomogeneous magnetic fields on the focal plane. Finally, we summarize the
proceeding of this paper in Section 5.

##  2 EXPERIMENT SETUP

In this section, we describe the sub-components of the PMU in detail: the
rotational mechanism, the gripper, the encoder, and the AHWP. Figure 1 shows
the photograph of the sub-components. Then we describe an assembled setup.

---
Figure 1:  Photographs of the YBCO ring, gripper arms, the Hall sensor
location, the installation of the rotor system, the encoder subsystem and the
AHWP.

###  2.1 Cryogenic rotation mechanism

We aim to maintain a stable rotation of the AHWP at 5 K. The cryogenic
rotation mechanism employs a superconducting magnetic bearing (SMB) [5]. The
SMB consists of a high-temperature superconductor (HTS) YBCO and a NdFeB
permanent magnet ring. The ten segments of YBCO bulk superconductors are
arranged in a ring shape with an outer diameter, an inner diameter, and a
height of 95 mm, 55 mm, and 20 mm, respectively. The NdFeB permanent magnet
ring has an outer diameter of 85 mm, an inner diameter of 65 mm, and a height
of 10 mm. At temperatures below 20 K, the frozen magnetic fields of the HTS
will levitate the permanent magnet ring as a rotor. Therefore, the SMB with no
mechanical contact can avoid the heat dissipation from the physical friction.
However, we anticipate the energy loss due to the magnetic interaction, e.g.
eddy current and hysteresis loss in the rotor and stator components [6]. One
of the magnetic field sources is the rotor magnet. Any inhomogeneity of the
ring magnet can produce the ac time-varying magnetic field for the stator
components. We installed a Hall sensor (BHT921) and monitored its magnetic
field inhomogeneity. The SMB can levitate the rotor magnet and thus the HWP.
We must, however, drive the rotor and maintain its rotation at a constant
rotational frequency. We employ a custom motor and drive electronics developed
by Tamagawa Seiki. This AC driver motor outputs a three-phase signal with
changing frequency depending on the feedback of the rotor speed.

###  2.2 Cryogenic holder mechanism

We use three cryogenic holder mechanisms separated by 120 degrees around the
rotor. Each of them is controlled by a cryogenic stepping motor with a
variable resistant slide volume, which implements the linear movement
triggered by the stepping motor. This holder mechanism holds the rotor until
the YBCO cools down below its critical temperature and functions as an SMB.
Also, this holder serves as a conductive thermal path to cool the rotor
because the thermal path is only through radiative heat exchange once the
rotor levitates.

###  2.3 Encoder system

We measure the HWP position angle by an optical encoder. It consists of an
encoder chopper disk on the rotor and a set of LED and silicon photodiode. We
prepare three sets of LED (L9337-01) as emitters and SiPD (S2386-18L) as
receivers. Each pair is placed face-to-face. The encoder disk has 64 slots for
the relative angle and one for the absolute angle within one revolution. Two
pairs of the LED-SiPD have represented phases A and B for the relative angle,
while the other pair is used for the absolute angle named phase Z. Figure 1
shows the photograph of the encoder jigs and the encoder disk.

###  2.4 Achromatic Half-Wave Plate

The AHWP is a five-layer sapphire stack with a diameter of 50 mm in diameter.
The detailed description of this AHWP can be found in [7, 9]. The thickness of
the single a-cut HWP is chosen based on the center frequency of the LFT at
97.5 GHz. Sapphire HWP can reflect about half of the incident signal due to
its high refractive index. Therefore, the first and fifth layers include a
moth-eye anti-reflective sub-wavelength structure (SWS) by laser machining.
This method performs $>90\%$ of transmittance over the frequency range from 43
to 161 GHz. The details can be found in [8].

###  2.5 PMU System in the Cryostat

Figure 2 shows the diagram of the experimental setup. A cryostat has two open
windows with a diameter of 100 mm. A UHMWPE window is used to transmit the
millimeter-wave source from the outside of the cryostat. We use a 2 mm thick
Acrylonitrile butadiene styrene (ABS) plate as an infrared filter. The
attenuated signal goes through AHWP mounted on the rotor of the rotational
mechanism. Then, a 45 degree plane mirror reflects the signal inside the
cryostat. Finally, a diode detector receives the output signal outside of the
cryostat. At room temperature, a mechanical chopper chops the input
millimeter-wave signal. In order to obtain the polarized signal, two-wire
grids are placed to align the polarization angle.

The PMU system is cooled down to below 10 K using a 4-K GM cryocooler. After
the cryogenic holder mechanisms are fully opened, the rotor levitates by the
SMB mechanism. The motor applies the driver torque to spin the rotor at a
constant speed. We then provide the incident signal through the rotating AHWP
into the cryostat and detect the output modulated signal. Finally, we monitor
the temperature of the PMU using thermometers.

---
Figure 2:  The set up for the optical measurements. We use a GM cryocooler
cryostat, off-axis parabolic mirrors, wire grids, and the chopper. The vacuum
windows are UHMWPE plates. ABS stands for Acrylonitrile Butadiene Styrene.

As the description of the AHWP formalism [10, 7, 9], the output modulated
signal can be fitted using the equation

$I_{out}(\nu,t)=A_{0}(\nu)+\sum_{n=1}^{8}A_{n}(\nu)\cos\left(n\omega_{hwp}t+n\phi_{n}\right),$
(1)

where $A_{0}$ is a constant amplitude, $A_{n}$ are the amplitudes,
$\omega_{hwp}$ and $\phi_{n}$ are the HWP modulated angle and the phase,
respectively. We limit the harmonic frequency up to the $8^{th}$ in Equation 1
to study the synchronous signal.

##  3 RESULTS

In this section, we present the preliminary results obtained from the
experiment. Once the PMU system cooled down below 10 K, we spin the rotor
using the electromagnetic drive mechanism and measure its rotation by the
encoder signal. Then, we present the result of the angle reconstruction and
the spin-down measurement. We also show the modulated signal, including the
rotating AHWP synchronous signal.

###  3.1 Angle reconstruction

The left panel of Figure 3 shows the raw encoder data when the PMU spins at 1
Hz. On the right panel, the power spectral density (PSD) of the encoder Z
signal shows the dominant peak at 1 Hz and its harmonic oscillations of the
PMU rotational frequency. The PSD of encoder A and B signals show the dominant
peaks at the expected frequency of 64 Hz, corresponding to the 64 slots of the
encoder disk.

---
Figure 3:  The encoder signal in time domain (left) and frequency domain
(right) when spinning the rotor at 1Hz. Figure 4:  The period $\Delta t_{i}$
(left) and the HWP angle position reconstruction $\rho_{t_{i}}$ (right) for
the encoder A signal in case of $f_{hwp}=1\ Hz$. The time in the horizontal
axis is the $t_{i}$ in the time stream.

Figure 4 shows the period $\Delta t_{i}$ and the HWP angle position
reconstruction $\rho_{t_{i}}$ from the encoder-A signal. We rotated the HWP at
1 Hz, the encoder disk has 64 slots as mentioned above, thus each slot is
equivalent to an angle of $360/64\simeq 5.6$ degrees. We estimated the
uncertainty of the HWP angle position reconstruction $\sigma_{\rho}\sim 0.1$
degrees.

###  3.2 Spin-down measurement

We conducted spin-down measurements to estimate the heat dissipation due to
the rotational loss. Spin-down measurements help to estimate the energy loss
from the contactless rotational mechanism. We set the speed of the rotor at a
constant frequency and then let it freely decelerate. The angular deceleration
can be expressed as a function of the HWP rotational frequency $f_{hwp}$. [11]

$\alpha=2\pi\dfrac{df_{hwp}}{dt}=a_{0}+2\pi a_{1}f_{hwp}.$ (2)

The coefficient $a_{0}$ represents the contribution of the hysteresis loss due
to the deceleration of the rotor. The coefficient $a_{1}$ determines the
amount of the eddy current loss. The solution of the differential equation 2
has an exponential form

$f_{hwp}\sim\dfrac{-a_{0}}{2\pi a_{1}}+\dfrac{1}{2\pi a_{1}}e^{-a_{1}(t+c)},$
(3)

where $c$ is a constant represents the starting point of the fitting time.

---
Figure 5:  Left: The spin-down measurements. The solid lines are experimental
data, while the dashed lines are models. Right: The heat dissipation powers
obtain from different rotational frequencies. Hysteresis losses are estimated
by considering only the parameter $a_{0}$, i.e. setting $a_{1}=0$. A similar
approach is used for eddy current losses by accounting only the contribution
of $a_{1}$ term.

The left panel of Figure 5 shows the spin-down measurements for a range of
rotational frequencies from 0.2 to 1.1 Hz. We fit the rotational frequency
data with the model Equation 3 to extract the parameters $a_{0}$ and $a_{1}$.
The performance of the fit is shown in Figure 5 (left). The typical values for
$a_{0}$ and $a_{1}$ are $5.8\times 10^{-3}$ $1/$s and $7.5\times 10^{-4}$
1/s2. It is clear that the system is dominated by the hysteresis.

We assume the energy loss during the spin-down is dissipated as heat energy.
The heat dissipation power $P$ is estimated as

$P=I\tau=I\dfrac{d\omega}{dt}\omega\\\ =I\left(a_{0}+2\pi
a_{1}f_{hwp}\right)2\pi f_{hwp},$ (4)

where $\tau$ is the torque of the rotor. The moment of inertia of the rotor is
assumed to be $3.6\times 10^{-3}$ kgm2, and the corresponding heat dissipation
is $0.23$ mW.

The right panel of Figure 5 shows the expected heat dissipation as a function
frequency. The two contributions, hysteresis and the eddy current losses have
a different frequency dependence. The hysteresis loss is higher than the eddy
current loss at this rotational frequency rage.

###  3.3 Modulated signal

Figure 6 shows the modulated signal as a function of the HWP angle for one
revolution. We fitted the data with the model Equation 1. We first extract the
modulation efficiency $A_{4}/A_{0}\sim 0.94$ and the phase $\phi_{4}=-17.6$
degrees.

---
Figure 6:  The modulated signal at 90 GHz versus the HWP rotational angle. The
black dotted curve is the experimental data. The red curve is the fitted model
Equation 1 compromised of $1f_{hwp}$, $2f_{hwp}$, $3f_{hwp}$ and $4f_{hwp}$
cosine waves. The seventeen free parameters for the fitting model are the
amplitude and phase of each cosine wave and an overall offset.
---
Figure 7:  (left) The power spectral density of the modulated signal and
(right) the Hall sensor signal for three different configurations.

Figure 7 shows the PSD of the optical signal (left) from the lock-in amplifier
and the PSD of the Hall sensor signal (right) for three configurations, no
rotation with levitation and no input signal, no rotation with levitation and
input signal, and rotation with levitation and input signal. We identified the
peaks at the modulated frequency, $4f_{hwp}$, and rotational synchronous
frequencies. The rich data set contains various subjects to be addressed. The
examples are to identify the origin of all the peaks, determine the width and
stability of the peaks, and demonstrate the demodulated PSD with the encoder
data.

##  4 DISCUSSIONS

We have presented the current status of the prototype PMU system and the
evaluation of the millimeter-wave polarimetric performance. We are aware that
this system is much smaller than the prospective flight size. Thus, the
results may not fully represent the effects we will encounter with the flight
size. However, we plan to explore broad parameter spaces using this setup with
the following motivations.

Our measurements in this paper are limited to a single electromagnetic
frequency at 90 GHz. We can expand this to a broader frequency range and study
the spectroscopic properties of the AHWP. We also plan to study the coupling
between the AHWP and the neighbor geometry, e.g. baffle and aperture. Any
rotational synchronous signal may originate from the AHWP itself or the
rotational parts that hold the AHWP. In addition to relying on a standard
optics simulation, it is beneficial to study this effect experimentally with
fast turn-round design modifications. Also, it is of great interest to
experimentally investigate the stability of the rotational synchronous signal.

This setup also helps prepare to combine with a TES detector and readout
system. When the PMU operates with the TES and its readout electronics, there
may be many additional effects, e.g. magnetic and EMI interferences and
temperature of the AHWP itself, that we cannot investigate in this setup[12,
13]. Therefore, the current setup is helpful in the hardware preparation and
should also help disentangle some of the effects in future data with the TES
and its readout system.

Last but not least, the small setup is easy to handle regarding the hardware
preparation time and cryostat run time. Any measurement from this setup
provides the recipe of the measurement methods and its sequence for the flight
scale model.

##  5 CONCLUSIONS

We have demonstrated a 1/10 scale prototype of the PMU for the LiteBIRD LFT.
The prototype PMU AHWP consists of a five-layer AHWP with the anti-reflection
sub-wavelength structure on the first and fifth layers. The rotational
mechanism contains the SMB, the cryogenic holder mechanism, and the encoder.
We cooled down the prototype PMU to 10 K with a 4-K GM cryocooler. Then, we
successfully levitated the rotor, and the AHWP rotated at several rotational
frequencies. We carried out the spin-down measurement to estimate the heat
dissipation due to the hysteresis loss and the eddy current loss. The coherent
millimeter-wave source at 90 GHz is provided from outside of the cryostat to
the inside, and the continuously rotating AHWP modulates the polarized signal.
Given this setup presented in this paper, we are ready for more extensive
tests, which are the preparation for the forthcoming full-scale model.

###### Acknowledgements.

We would like to thank Fabio Columbro and Peter Hargrave for fruitful
comments. We thank the World Premier International Research Center Initiative
(WPI), MEXT, Japan for support through Kavli IPMU. This work was also
supported by JSPS KAKENHI Grant Numbers 18KK0083, and JSPS Core-to-Core
Program number JPJSCCA20200003, A. Advanced Research Networks. LiteBIRD (phase
A) activities are supported by the following funding sources: ISAS/JAXA, MEXT,
JSPS, KEK (Japan); CSA (Canada); CNES, CNRS, CEA (France); DFG (Germany); ASI,
INFN, INAF (Italy); RCN (Norway); AEI (Spain); SNSA, SRC (Sweden); NASA, DOE
(USA).

## References

  * [1] LiteBIRD Collaboration, “Probing cosmic inflation with the litebird cosmic microwave background polarization survey,” (2022).
  * [2] Kusaka, A., Essinger-Hileman, T., Appel, J. W., Gallardo, P., Irwin, K. D., Jarosik, N., Nolta, M. R., Page, L. A., Parker, L. P., Raghunathan, S., Sievers, J. L., Simon, S. M., Staggs, S. T., and Visnjic, K., “Modulation of cosmic microwave background polarization with a warm rapidly rotating half-wave plate on the atacama b-mode search instrument,” Review of Scientific Instruments 85(2), 024501 (2014).
  * [3] Hill, C. A., Kusaka, A., Ashton, P., Barton, P., Adkins, T., Arnold, K., Bixler, B., Ganjam, S., Lee, A. T., Matsuda, F., Matsumura, T., Sakurai, Y., Tat, R., and Zhou, Y., “A cryogenic continuously rotating half-wave plate mechanism for the polarbear-2b cosmic microwave background receiver,” Review of Scientific Instruments 91(12), 124503 (2020).
  * [4] Hoang, D. T., Patanchon, G., Bucher, M., Matsumura, T., Banerji, R., Ishino, H., Hazumi, M., and Delabrouille, J., “Bandpass mismatch error for satellite CMB experiments i: estimating the spurious signal,” Journal of Cosmology and Astroparticle Physics 2017, 015–015 (dec 2017).
  * [5] Matsumura, T., Kataza, H., Utsunomiya, S., Yamamoto, R., Hazumi, M., and Katayama, N., “Design and performance of a prototype polarization modulator rotational system for use in space using a superconducting magnetic bearing,” IEEE Transactions on Applied Superconductivity 26(3), 1–4 (2016).
  * [6] Sakurai, Y., Matsumura, T., Katayama, N., Komatsu, K., Takaku, R., Sugiyama, S., Nomura, Y., Toda, T., Ghigna, T., Iida, T., Sugai, H., Imada, H., Hazumi, M., Ishino, H., Ohsaki, H., Terao, Y., Enokida, H., Ishida, Y., Iwata, Y., Ahmad, D., Konishi, K., Sakurai, H., Yumoto, J., Kuwata-Gonokami, M., Kusaka, A., and Hill, C., “Breadboard model of the polarization modulator unit based on a continuously rotating half-wave plate for the low-frequency telescope of the LiteBIRD space mission,” in [Millimeter, Submillimeter, and Far-Infrared Detectors and Instrumentation for Astronomy X ], Zmuidzinas, J. and Gao, J.-R., eds., 11453, 743 – 762, International Society for Optics and Photonics, SPIE (2020).
  * [7] Komatsu, K., Ishino, H., Kataza, H., Konishi, K., Kuwata-Gonokami, M., Katayama, N., Sugiyama, S., Matsumura, T., Sakurai, H., Sakurai, Y., Takaku, R., and Yumoto, J., “Demonstration of five-layer phase-flat achromatic half-wave plate with anti-reflective structures and superconducting magnetic bearing for CMB polarization experiments,” in [Millimeter, Submillimeter, and Far-Infrared Detectors and Instrumentation for Astronomy X ], Zmuidzinas, J. and Gao, J.-R., eds., 11453, 779 – 794, International Society for Optics and Photonics, SPIE (2020).
  * [8] Takaku, R., Hanany, S., Imada, H., Ishino, H., Katayama, N., Komatsu, K., Konishi, K., Kuwata-Gonokami, M., Matsumura, T., Mitsuda, K., Sakurai, H., Sakurai, Y., Wen, Q., Yamasaki, N. Y., Young, K., and Yumoto, J., “Broadband, millimeter-wave anti-reflective structures on sapphire ablated with femto-second laser,” Journal of Applied Physics 128(22), 225302 (2020).
  * [9] Komatsu, K., Ishino, H., Katayama, N., Matsumura, T., and Sakurai, Y., “Design of a frequency-independent optic axis Pancharatnam-based achromatic half-wave plate,” Journal of Astronomical Telescopes, Instruments, and Systems 7(3), 1 – 13 (2021).
  * [10] Matsumura, T., A cosmic microwave background radiation polarimeter using superconducting magnetic bearings, PhD thesis, University of Minnesota, Twin Cities (Jan. 2006).
  * [11] Hanany, S., Matsumura, T., Johnson, B., Jones, T., Hull, J., and Ma, K., “A cosmic microwave background radiation polarimeter using superconducting bearings,” IEEE Transactions on Applied Superconductivity 13(2), 2128–2133 (2003).
  * [12] Ghigna, T., Matsumura, T., Hazumi, M., Stever, S. L., Sakurai, Y., Katayama, N., Suzuki, A., Westbrook, B., and Lee, A. T., “Design of a testbed for the study of system interference in space cmb polarimetry,” Journal of Low Temperature Physics 199(3), 622–630 (2020).
  * [13] Sugiyama, S. e. a., “Vibration characteristics of a continuously rotating hwp using superconducting magnetic bearing and its potential impact on tes detectors and squids for cmb polarimetry,” in preparation .